This discussion is locked
ClientServices
Level 1 (0 points)

Q: Problem: Standby MDC keeps failing to mount volume...

Hello XSAN experts, I'm new to XSAN and I hope I can find some help here, the reason for using XSAN is we'd like to setup a file server for podcasts/audio files using the existing Xserves.

Currently we have a very simple Xsan 2.2 setup: two MDC's connected to a Brocade FC switch to our SAN storage raid units. And a private metadata network is on 192.168.1.0/255.255.255.0 with no DNS no Router configured (for metadata).  I can ping both servers by name and their public IPs no problem.  I can also ping both private IP addresses as well.  However I couldn't nslookup the private IPs, not sure if this is required?

Anyway, my first MDC is called itunesutm0, and my second MDC is called itunesutm1.  I have created two LUNs, LUN1 goes to Data, and LUN2 goes to metadata and journal.  The volume name is called iTunesUTM and it is hosted by itunesutm0 server right now.

So when I launch Xsan Admin, I see that the itunesutm0 mounts the volume no problem, I can see it in Finder and browse through it.  But when I tried to mount it on my second controller, it keeps giving me failed to mount message.

Here is what I found in the system log of my second MDC when mounting fails.


Nov 24 18:32:25 itunesutm1 fsmpm[140]: PortMapper: Initiating activation vote for FSS 'iTunesUTM'.
Nov 24 18:32:30 itunesutm1 fsmpm[140]: PortMapper: FSS 'iTunesUTM' disconnected.
Nov 24 18:32:30 itunesutm1 fsmpm[140]: PortMapper: kicking diskscan_thread 4330123264.
Nov 24 18:32:30 itunesutm1 KernelEventAgent[73]: tid 00000000 received event(s) VQ_NOTRESP (1) Nov 24 18:32:38 itunesutm1 com.apple.xsan[58]: mount_acfs: No such process Nov 24 18:32:38 itunesutm1 xsand[58]: mount of volume 'iTunesUTM' failed (exit code = 22) Nov 24 18:32:38 itunesutm1 servermgrd[67]: xsan: [67/680B3C0] ERROR: -[SANFilesystem mountVolumeNamed:writable:withOptions:]: mount of 'iTunesUTM' failed: Unable to mount volume `iTunesUTM'
Nov 24 18:32:38 itunesutm1 Xsan Admin[70103]: ERROR: Error mounting volume…: The operation couldn’t be completed. (SANTransactionErrorDomain error 100007.) (100007) Nov 24 18:32:38 itunesutm1 servermgrd[67]: xsan: [67/2112A0] ERROR: get_fsm_process_stats(iTunesUTM): Unable to find pid of fsm Nov 24 18:32:25 itunesutm1 fsmpm[140]: PortMapper: Initiating activation vote for FSS 'iTunesUTM'.
Nov 24 18:32:30 itunesutm1 fsmpm[140]: PortMapper: FSS 'iTunesUTM' disconnected.
Nov 24 18:32:30 itunesutm1 fsmpm[140]: PortMapper: kicking diskscan_thread 4330123264.
Nov 24 18:32:30 itunesutm1 KernelEventAgent[73]: tid 00000000 received event(s) VQ_NOTRESP (1) Nov 24 18:32:38 itunesutm1 com.apple.xsan[58]: mount_acfs: No such process Nov 24 18:32:38 itunesutm1 xsand[58]: mount of volume 'iTunesUTM' failed (exit code = 22) Nov 24 18:32:38 itunesutm1 servermgrd[67]: xsan: [67/680B3C0] ERROR: -[SANFilesystem mountVolumeNamed:writable:withOptions:]: mount of 'iTunesUTM' failed: Unable to mount volume `iTunesUTM'
Nov 24 18:32:38 itunesutm1 Xsan Admin[70103]: ERROR: Error mounting volume…: The operation couldn’t be completed. (SANTransactionErrorDomain error 100007.) (100007) Nov 24 18:32:38 itunesutm1 servermgrd[67]: xsan: [67/2112A0] ERROR: get_fsm_process_stats(iTunesUTM): Unable to find pid of fsm Nov 24 18:32:25 itunesutm1 fsmpm[140]: PortMapper: Initiating activation vote for FSS 'iTunesUTM'.
Nov 24 18:32:30 itunesutm1 fsmpm[140]: PortMapper: FSS 'iTunesUTM' disconnected.
Nov 24 18:32:30 itunesutm1 fsmpm[140]: PortMapper: kicking diskscan_thread 4330123264.
Nov 24 18:32:30 itunesutm1 KernelEventAgent[73]: tid 00000000 received event(s) VQ_NOTRESP (1) Nov 24 18:32:38 itunesutm1 com.apple.xsan[58]: mount_acfs: No such process Nov 24 18:32:38 itunesutm1 xsand[58]: mount of volume 'iTunesUTM' failed (exit code = 22) Nov 24 18:32:38 itunesutm1 servermgrd[67]: xsan: [67/680B3C0] ERROR: -[SANFilesystem mountVolumeNamed:writable:withOptions:]: mount of 'iTunesUTM' failed: Unable to mount volume `iTunesUTM'
Nov 24 18:32:38 itunesutm1 Xsan Admin[70103]: ERROR: Error mounting volume…: The operation couldn’t be completed. (SANTransactionErrorDomain error 100007.) (100007) Nov 24 18:32:38 itunesutm1 servermgrd[67]: xsan: [67/2112A0] ERROR: get_fsm_process_stats(iTunesUTM): Unable to find pid of fsm Nov 24 18:32:25 itunesutm1 fsmpm[140]: PortMapper: Initiating activation vote for FSS 'iTunesUTM'.
Nov 24 18:32:30 itunesutm1 fsmpm[140]: PortMapper: FSS 'iTunesUTM' disconnected.
Nov 24 18:32:30 itunesutm1 fsmpm[140]: PortMapper: kicking diskscan_thread 4330123264.
Nov 24 18:32:30 itunesutm1 KernelEventAgent[73]: tid 00000000 received event(s) VQ_NOTRESP (1) Nov 24 18:32:38 itunesutm1 com.apple.xsan[58]: mount_acfs: No such process Nov 24 18:32:38 itunesutm1 xsand[58]: mount of volume 'iTunesUTM' failed (exit code = 22) Nov 24 18:32:38 itunesutm1 servermgrd[67]: xsan: [67/680B3C0] ERROR: -[SANFilesystem mountVolumeNamed:writable:withOptions:]: mount of 'iTunesUTM' failed: Unable to mount volume `iTunesUTM'
Nov 24 18:32:38 itunesutm1 Xsan Admin[70103]: ERROR: Error mounting volume…: The operation couldn’t be completed. (SANTransactionErrorDomain error 100007.) (100007) Nov 24 18:32:38 itunesutm1 servermgrd[67]: xsan: [67/2112A0] ERROR: get_fsm_process_stats(iTunesUTM): Unable to find pid of fsm Nov 24 18:32:40 itunesutm1 fsmpm[140]: PortMapper: RESTART FSS service 'iTunesUTM[1]' on host itunesutm1.domain_name.ca.
Nov 24 18:32:40 itunesutm1 fsmpm[140]: PortMapper: Starting FSS service 'iTunesUTM[1]' on itunesutm1.domain_name.ca
Nov 24 18:32:40 itunesutm1 fsmpm[140]: PortMapper: FSS 'iTunesUTM'[1] (pid 72901) at port 53562 is registered.
Nov 24 18:32:25 itunesutm1 fsmpm[140]: PortMapper: Initiating activation vote for FSS 'iTunesUTM'.
Nov 24 18:32:30 itunesutm1 fsmpm[140]: PortMapper: FSS 'iTunesUTM' disconnected.
Nov 24 18:32:30 itunesutm1 fsmpm[140]: PortMapper: kicking diskscan_thread 4330123264.
Nov 24 18:32:30 itunesutm1 KernelEventAgent[73]: tid 00000000 received event(s) VQ_NOTRESP (1) Nov 24 18:32:38 itunesutm1 com.apple.xsan[58]: mount_acfs: No such process Nov 24 18:32:38 itunesutm1 xsand[58]: mount of volume 'iTunesUTM' failed (exit code = 22) Nov 24 18:32:38 itunesutm1 servermgrd[67]: xsan: [67/680B3C0] ERROR: -[SANFilesystem mountVolumeNamed:writable:withOptions:]: mount of 'iTunesUTM' failed: Unable to mount volume `iTunesUTM'
Nov 24 18:32:38 itunesutm1 Xsan Admin[70103]: ERROR: Error mounting volume…: The operation couldn’t be completed. (SANTransactionErrorDomain error 100007.) (100007) Nov 24 18:32:38 itunesutm1 servermgrd[67]: xsan: [67/2112A0] ERROR: get_fsm_process_stats(iTunesUTM): Unable to find pid of fsm



I'd very much appreciate if anyone can provide some insights or tips on how to fix this problem. 

Thanks guys,

Posted on Nov 24, 2011 4:40 PM

Close
ClientServices

Q: Problem: Standby MDC keeps failing to mount volume...

  • All replies
  • Helpful answers

  • by ClientServices,

    ClientServices ClientServices Nov 24, 2011 4:40 PM in response to ClientServices
    Level 1 (0 points)
    Nov 24, 2011 4:40 PM in response to ClientServices

    BTW I am also getting repeated messages like this in the system log in my 2nd MDC.


    Nov 24 18:37:30 itunesutm1 servermgrd[67]: xsan: [67/2112A0] ERROR: get_fsmvol_at_index: Could not connect to FSM because File System Manager "iTunesUTM" on 192.168.1.101 is on standby.
    Nov 24 18:37:30: --- last message repeated 1 time ---
    Nov 24 18:37:30 itunesutm1 servermgrd[67]: xsan: [67/2112A0] ERROR: get_quotas_for_fsmvol_named(iTunesUTM): Could not connect to FSM because File System Manager "iTunesUTM" on 192.168.1.101 is on standby.


    is this normal or ...?

  • by receng,

    receng receng Nov 30, 2011 11:10 AM in response to ClientServices
    Level 1 (20 points)
    Nov 30, 2011 11:10 AM in response to ClientServices

    It does not look normal. It looks like the FSM daemon on mdc2 is not working correctly.

    You can try a few things:

     

    1. Demote mdc2 to a client and try mounting and unmounting the volume. If it works, promote it back to controller

    2. If you can't make mdc2 a client, try an uninstall and re-install of the Xsan software (if this is not Lion)

    3. Re-image MDC2 completely

  • by ClientServices,

    ClientServices ClientServices Dec 1, 2011 4:02 PM in response to receng
    Level 1 (0 points)
    Dec 1, 2011 4:02 PM in response to receng

    Thanks for your reply receng,

     

    1. i tried to demote mdc2 to a client, but it failed at the end with an error message "could not write to config file?" something like that... I didn't get the correct message at that time...  so something is clearly not right i guess?

     

    2. i've tried the uninstaller and reinstall XAN a few times, the problem still persist.  MDCs are running 10.6.6

     

    3. re-install MDC2 completely from scratch.... not really the path I want to go in but if all else fails, i guess that will be my last attempt.

     

     

    Are there any commands that I can try in cvadmin mode to check if FSM is working properly?

  • by receng,

    receng receng Dec 8, 2011 8:46 AM in response to ClientServices
    Level 1 (20 points)
    Dec 8, 2011 8:46 AM in response to ClientServices

    One thing worth trying, if you have not tried already is stopping the volume(s) and shutting everything down (if you can't do the fiber switches, don't worry) and powering up in sequence: switches, storage, mdc1, mdc2, clients.

     

    I just noticed on your logs something about "iTunesUTM". I would try completely removing iTunes from MDC2:

    http://support.apple.com/kb/ht1925

     

    I would try uninstalling the Xsan from MDC2 again and make sure you remove this folder: Library/Filesystems/Xsan. Be sure to have your serial number (or keep a copy of that folder just in case)

     

    If this does not work, I would think of re-imaging.

     

    cvadmin would be usefull only to confirm that the controllers are out of sync.

    The typical output is bellow.

    The "*" shows that the volume is online, "located on xx.xx.xx.xx" tells you the IP of the MDC hosting the volume (throught the private network), "pid" tells you the process ID of the FSM process running on the MDC that you can confirm looking at the activity monitor (or "top" on terminal).

     

    If you run cvadmin on both MDC1 and MDC2 and see a difference, you can be sure the controllers are out of sync.

     

    Typical output:

     

    xserve-mdc2:~ root# cvadmin

    Xsan Administrator

     

     

    Enter command(s)

    For command help, enter "help" or "?".

     

     

    List FSS

     

     

    File System Services (* indicates service is in control of FS):

    1>*XSAN_Vol_1[0]    located on 10.3.xxx.xxx:49966 (pid 701)

    2>*XSAN_Vol_2[0]    located on 10.3.xxx.xxx:49985 (pid 704)

     

     

    Select FSM "none"

     

     

     

     

    Xsanadmin> list

    syntax error. Enter "help" or "?" for help.

  • by Blaidd Drwg,

    Blaidd Drwg Blaidd Drwg Dec 22, 2011 2:08 AM in response to ClientServices
    Level 1 (109 points)
    Dec 22, 2011 2:08 AM in response to ClientServices

    In addition to being unable to mount the volume, the logs showed the secondary MDC cannot keep an fsm process active for the volume. These problems together suggest that the secondary MDC can't communicate reliably (or at all) with the primary over the metadata network. A simple way to verify this would be to temporarily shut down the primary MDC and see if the problems persist on the secondary.