XSAN Volume keeps unmounting due to /dev/disk18 not responding

Question

Hi Support and everybody,I would like to ask some technical advice from you.We have a setup for XSAN Server.3 -MDC: Mac OS X Server 10.6.6, XSAN 2.2 (xserves)1-OD: Mac OS X Server 10.6.6, XSAN 2.2 (xserves)60-Clients: Mac OS X 10.6.6, XSAN 2.2 (macpros)3 x XSAN Volumes9 x promise StoragesISSUE: Recently, One of the XSAN Volume keeps unmounting.           The only way to mount it back is to reboot the whole Xserves and MDC including the Promise storages.According to the MDC Logs: /dev/disk18 is not respondingMy question is which LUN from the XSAN corresponds to /dev/disk18?How to isolate the issue?Which LUN does /dev/disk18 corresponds?What does the /dev/disk18 not responding means?Please advise.

Blaidd Drwg · Answer

That device number corresponds to the Xsan volume itself. It doesn&#x27;t correspond to any LUN. The fsm process that controls the volume is crashing--see the fsm panic messages in the log. Consider stopping the volume then running "cvfsck -nv VolumeName". This will do a read only check for problems with the metadata.

monching16 · Answer

Hi Sir Blaidd Drwg,Thanks for your response.So kind of you.Now it&#x27;s clear to me that the issue is not about the Physical storage or the LUN.It&#x27;s the File System as you adviced.So sorry but I have a followup question:1. cvfsck -nv VolumeName - Is this command will just check if there&#x27;s a problem?2. What is the command to fix the issue?Thnaks and have a nice day.

Blaidd Drwg · Answer

1. Yes, the -n means read-only.2. Replace the -n with -w, e.g:cvfsck -wv VolumeName

monching16 · Answer

Hi Sir Blaidd Drwg,Here is the Command Line Output after I ran the following ations:1. cvfsck -nvThe asterisk (*) denotes the file system is active.Choose a file system by number (1-3) or zero (0) to exit -&gt; 3Re-creating File System cvfsck Directory &#x27;/Library/Filesystems/Xsan/data/NAFA_DM3/trace&#x27;.Created directory /tmp/cvfsck1070a for temporary files.Creating NAFA_DM3_Metadata allocation check file.Creating NAFA_DM3_Pool1 allocation check file.Creating NAFA_DM3_Pool2 allocation check file.** NOTE ** Read Only Check.File system journal will not be recovered.The results may be inconsistent and mis-leading.Super Block information.  FS Created On               : Tue Aug  2 15:06:01 2011  Inode Version               : &#x27;2.5&#x27;  File System Status          : Clean  Allocated Inodes            : 1052672  Free Inodes                 : 53124  FL Blocks                   : 79  Next Inode Chunk            : 0x2907a  Metadump Seqno              : 0  Restore Journal Seqno       : 0  Windows Security Indx Inode : 0x6  Windows Security Data Inode : 0x7  Quota Database Inode        : 0x8  ID Database Inode           : 0xc  Client Write Opens Inode    : 0x9Stripe Group NAFA_DM3_Metadata              (  0) 0x745e9e0 blocks.Stripe Group NAFA_DM3_Pool1                 (  1) 0xcbba0740 blocks.Stripe Group NAFA_DM3_Pool2                 (  2) 0xcbba0740 blocks.Building Inode Index Database 1052672 (100%).          Verifying NT Security DescriptorsFound 18227 NT Security Descriptors: all are goodVerifying Free List Extents.Scanning inodes 307200 ( 29%).        *Error*: XATTR ERROR detected for inode 0x529ca63 (0x80000529ca64)*Error*: *** RESETTING RPL DATA FOR FILE SYSTEM ***Scanning inodes 819200 ( 77%).        *Error*: XATTR ERROR detected for inode 0x8b84e71 (0x8b84e73)Scanning inodes 1052672 (100%).          Sorting extent list for NAFA_DM3_Metadata pass 1/1Updating bitmap for NAFA_DM3_Metadata extents 59225 (  3%).                    Sorting extent list for NAFA_DM3_Pool1 pass 1/1Updating bitmap for NAFA_DM3_Pool1 extents 817735 ( 51%).                    Sorting extent list for NAFA_DM3_Pool2 pass 1/1Updating bitmap for NAFA_DM3_Pool2 extents 1584394 (100%).                    Checking for dead inodes 307200 ( 29%).        Clearing  inode 0x80000529ca64.Checking for dead inodes 614400 ( 58%).        Clearing  inode 0x8000077ee888.Checking for dead inodes 819200 ( 77%).        Clearing  inode 0x8b84e72.Clearing  inode 0x8b84e74.Checking for dead inodes 1052672 (100%).          Checking directories 58747 (100%).         Scanning for orphaned inodes 1052672 (100%).        Verifying link &amp; subdir counts 1052672 (100%).          Super Block -  Free Inodes of 0xcf84 is wrong! Should be 0xcf8a.Repairing free list.Checking pending free list.                        Checking Arbitration Control Block.Checking NAFA_DM3_Metadata allocation bit maps (100%).         Checking NAFA_DM3_Pool1 allocation bit maps (  9%).         Stripe Group &#x27;NAFA_DM3_Pool1&#x27; block 0x1ca9ee10 differs - ondisk/Allocated 0x00 incore/Free 0x80 byteoffs/60112322 .Map file &#x27;NAFA_DM3_Pool1&#x27; is incorrect. Updating.Checking NAFA_DM3_Pool2 allocation bit maps ( 39%).         Stripe Group &#x27;NAFA_DM3_Pool2&#x27; block 0x600e6427 differs - ondisk/Allocated 0xfe incore/Free 0xff byteoffs/201444484 .Map file &#x27;NAFA_DM3_Pool2&#x27; is incorrect. Updating.File system &#x27;NAFA_DM3&#x27; was modified.File system &#x27;NAFA_DM3&#x27;. Blocks-6835932800 free-2939148886 Inodes-1052672 free-53130.File System Check finished - file system modified.mdcthird:~ root# ________________________________________________________________________________ ____________________________________________________________________________________________ ____________2. cvfsck -wv** WARNING ** This file system check may modify the meta-data of the              --- [NAFA_DM3] ---              file system. This procedure cannot be un-done!Do you want to proceed? (Y/N) -&gt; YCreated directory /tmp/cvfsck1231a for temporary files.Attempting to acquire arbitration block... successful.Creating NAFA_DM3_Metadata allocation check file.Creating NAFA_DM3_Pool1 allocation check file.Creating NAFA_DM3_Pool2 allocation check file.Recovering Journal Log.Super Block information.  FS Created On               : Tue Aug  2 15:06:01 2011  Inode Version               : &#x27;2.5&#x27;  File System Status          : Clean  Allocated Inodes            : 1052672  Free Inodes                 : 53124  FL Blocks                   : 79  Next Inode Chunk            : 0x2907a  Metadump Seqno              : 0  Restore Journal Seqno       : 0  Windows Security Indx Inode : 0x6  Windows Security Data Inode : 0x7  Quota Database Inode        : 0x8  ID Database Inode           : 0xc  Client Write Opens Inode    : 0x9Stripe Group NAFA_DM3_Metadata              (  0) 0x745e9e0 blocks.Stripe Group NAFA_DM3_Pool1                 (  1) 0xcbba0740 blocks.Stripe Group NAFA_DM3_Pool2                 (  2) 0xcbba0740 blocks.Building Inode Index Database 1052672 (100%).          Verifying NT Security DescriptorsFound 18227 NT Security Descriptors: all are goodVerifying Free List Extents.Scanning inodes 307200 ( 29%).        *Error*: XATTR ERROR detected for inode 0x529ca63 (0x80000529ca64)*Error*: *** RESETTING RPL DATA FOR FILE SYSTEM ***Scanning inodes 819200 ( 77%).        *Error*: XATTR ERROR detected for inode 0x8b84e71 (0x8b84e73)Scanning inodes 1052672 (100%).          Sorting extent list for NAFA_DM3_Metadata pass 1/1Updating bitmap for NAFA_DM3_Metadata extents 10240 (  0%).          Updating bitmap for NAFA_DM3_Metadata extents 20480 (  1%).          Updating bitmap for NAFA_DM3_Metadata extents 30720 (  1%).          Updating bitmap for NAFA_DM3_Metadata extents 40960 (  2%).          Updating bitmap for NAFA_DM3_Metadata extents 51200 (  3%).          Updating bitmap for NAFA_DM3_Metadata extents 59225 (  3%).                    Sorting extent list for NAFA_DM3_Pool1 pass 1/1Updating bitmap for NAFA_DM3_Pool1 extents 61440 (  3%).             Updating bitmap for NAFA_DM3_Pool1 extents 71680 (  4%).             Updating bitmap for NAFA_DM3_Pool1 extents 81920 (  5%).             Updating bitmap for NAFA_DM3_Pool1 extents 92160 (  5%).             Updating bitmap for NAFA_DM3_Pool1 extents 102400 (  6%).            Updating bitmap for NAFA_DM3_Pool1 extents 112640 (  7%).            Updating bitmap for NAFA_DM3_Pool1 extents 122880 (  7%).            Updating bitmap for NAFA_DM3_Pool1 extents 133120 (  8%).            Updating bitmap for NAFA_DM3_Pool1 extents 143360 (  9%).            Updating bitmap for NAFA_DM3_Pool1 extents 153600 (  9%).            Updating bitmap for NAFA_DM3_Pool1 extents 163840 ( 10%).            Updating bitmap for NAFA_DM3_Pool1 extents 174080 ( 10%).            Updating bitmap for NAFA_DM3_Pool1 extents 184320 ( 11%).            Updating bitmap for NAFA_DM3_Pool1 extents 194560 ( 12%).            Updating bitmap for NAFA_DM3_Pool1 extents 204800 ( 12%).            Updating bitmap for NAFA_DM3_Pool1 extents 215040 ( 13%).            Updating bitmap for NAFA_DM3_Pool1 extents 225280 ( 14%).            Updating bitmap for NAFA_DM3_Pool1 extents 235520 ( 14%).            Updating bitmap for NAFA_DM3_Pool1 extents 245760 ( 15%).            Updating bitmap for NAFA_DM3_Pool1 extents 256000 ( 16%).            Updating bitmap for NAFA_DM3_Pool1 extents 266240 ( 16%).            Updating bitmap for NAFA_DM3_Pool1 extents 276480 ( 17%).            Updating bitmap for NAFA_DM3_Pool1 extents 286720 ( 18%).            Updating bitmap for NAFA_DM3_Pool1 extents 296960 ( 18%).            Updating bitmap for NAFA_DM3_Pool1 extents 307200 ( 19%).            Updating bitmap for NAFA_DM3_Pool1 extents 317440 ( 20%).            Updating bitmap for NAFA_DM3_Pool1 extents 327680 ( 20%).            Updating bitmap for NAFA_DM3_Pool1 extents 337920 ( 21%).            Updating bitmap for NAFA_DM3_Pool1 extents 348160 ( 21%).            Updating bitmap for NAFA_DM3_Pool1 extents 358400 ( 22%).            Updating bitmap for NAFA_DM3_Pool1 extents 368640 ( 23%).            Updating bitmap for NAFA_DM3_Pool1 extents 378880 ( 23%).            Updating bitmap for NAFA_DM3_Pool1 extents 389120 ( 24%).            Updating bitmap for NAFA_DM3_Pool1 extents 399360 ( 25%).            Updating bitmap for NAFA_DM3_Pool1 extents 409600 ( 25%).            Updating bitmap for NAFA_DM3_Pool1 extents 419840 ( 26%).            Updating bitmap for NAFA_DM3_Pool1 extents 430080 ( 27%).            Updating bitmap for NAFA_DM3_Pool1 extents 440320 ( 27%).            Updating bitmap for NAFA_DM3_Pool1 extents 450560 ( 28%).            Updating bitmap for NAFA_DM3_Pool1 extents 460800 ( 29%).            Updating bitmap for NAFA_DM3_Pool1 extents 471040 ( 29%).            Updating bitmap for NAFA_DM3_Pool1 extents 481280 ( 30%).            Updating bitmap for NAFA_DM3_Pool1 extents 491520 ( 31%).            Updating bitmap for NAFA_DM3_Pool1 extents 501760 ( 31%).            Updating bitmap for NAFA_DM3_Pool1 extents 512000 ( 32%).            Updating bitmap for NAFA_DM3_Pool1 extents 522240 ( 32%).            Updating bitmap for NAFA_DM3_Pool1 extents 532480 ( 33%).            Updating bitmap for NAFA_DM3_Pool1 extents 542720 ( 34%).            Updating bitmap for NAFA_DM3_Pool1 extents 552960 ( 34%).            Updating bitmap for NAFA_DM3_Pool1 extents 563200 ( 35%).            Updating bitmap for NAFA_DM3_Pool1 extents 573440 ( 36%).            Updating bitmap for NAFA_DM3_Pool1 extents 583680 ( 36%).            Updating bitmap for NAFA_DM3_Pool1 extents 593920 ( 37%).            Updating bitmap for NAFA_DM3_Pool1 extents 604160 ( 38%).            Updating bitmap for NAFA_DM3_Pool1 extents 614400 ( 38%).            Updating bitmap for NAFA_DM3_Pool1 extents 624640 ( 39%).            Updating bitmap for NAFA_DM3_Pool1 extents 634880 ( 40%).            Updating bitmap for NAFA_DM3_Pool1 extents 645120 ( 40%).            Updating bitmap for NAFA_DM3_Pool1 extents 655360 ( 41%).            Updating bitmap for NAFA_DM3_Pool1 extents 665600 ( 42%).            Updating bitmap for NAFA_DM3_Pool1 extents 675840 ( 42%).            Updating bitmap for NAFA_DM3_Pool1 extents 686080 ( 43%).            Updating bitmap for NAFA_DM3_Pool1 extents 696320 ( 43%).            Updating bitmap for NAFA_DM3_Pool1 extents 706560 ( 44%).            Updating bitmap for NAFA_DM3_Pool1 extents 716800 ( 45%).            Updating bitmap for NAFA_DM3_Pool1 extents 727040 ( 45%).            Updating bitmap for NAFA_DM3_Pool1 extents 737280 ( 46%).            Updating bitmap for NAFA_DM3_Pool1 extents 747520 ( 47%).            Updating bitmap for NAFA_DM3_Pool1 extents 757760 ( 47%).            Updating bitmap for NAFA_DM3_Pool1 extents 768000 ( 48%).            Updating bitmap for NAFA_DM3_Pool1 extents 778240 ( 49%).            Updating bitmap for NAFA_DM3_Pool1 extents 788480 ( 49%).            Updating bitmap for NAFA_DM3_Pool1 extents 798720 ( 50%).            Updating bitmap for NAFA_DM3_Pool1 extents 808960 ( 51%).            Updating bitmap for NAFA_DM3_Pool1 extents 817735 ( 51%).                    Sorting extent list for NAFA_DM3_Pool2 pass 1/1Updating bitmap for NAFA_DM3_Pool2 extents 819200 ( 51%).            Updating bitmap for NAFA_DM3_Pool2 extents 829440 ( 52%).            Updating bitmap for NAFA_DM3_Pool2 extents 839680 ( 52%).            Updating bitmap for NAFA_DM3_Pool2 extents 849920 ( 53%).            Updating bitmap for NAFA_DM3_Pool2 extents 860160 ( 54%).            Updating bitmap for NAFA_DM3_Pool2 extents 870400 ( 54%).            Updating bitmap for NAFA_DM3_Pool2 extents 880640 ( 55%).            Updating bitmap for NAFA_DM3_Pool2 extents 890880 ( 56%).            Updating bitmap for NAFA_DM3_Pool2 extents 901120 ( 56%).            Updating bitmap for NAFA_DM3_Pool2 extents 911360 ( 57%).            Updating bitmap for NAFA_DM3_Pool2 extents 921600 ( 58%).            Updating bitmap for NAFA_DM3_Pool2 extents 931840 ( 58%).            Updating bitmap for NAFA_DM3_Pool2 extents 942080 ( 59%).            Updating bitmap for NAFA_DM3_Pool2 extents 952320 ( 60%).            Updating bitmap for NAFA_DM3_Pool2 extents 962560 ( 60%).            Updating bitmap for NAFA_DM3_Pool2 extents 972800 ( 61%).            Updating bitmap for NAFA_DM3_Pool2 extents 983040 ( 62%).            Updating bitmap for NAFA_DM3_Pool2 extents 993280 ( 62%).            Updating bitmap for NAFA_DM3_Pool2 extents 1003520 ( 63%).           Updating bitmap for NAFA_DM3_Pool2 extents 1013760 ( 63%).           Updating bitmap for NAFA_DM3_Pool2 extents 1024000 ( 64%).           Updating bitmap for NAFA_DM3_Pool2 extents 1034240 ( 65%).           Updating bitmap for NAFA_DM3_Pool2 extents 1044480 ( 65%).           Updating bitmap for NAFA_DM3_Pool2 extents 1054720 ( 66%).           Updating bitmap for NAFA_DM3_Pool2 extents 1064960 ( 67%).           Updating bitmap for NAFA_DM3_Pool2 extents 1075200 ( 67%).           Updating bitmap for NAFA_DM3_Pool2 extents 1085440 ( 68%).           Updating bitmap for NAFA_DM3_Pool2 extents 1095680 ( 69%).           Updating bitmap for NAFA_DM3_Pool2 extents 1105920 ( 69%).           Updating bitmap for NAFA_DM3_Pool2 extents 1116160 ( 70%).           Updating bitmap for NAFA_DM3_Pool2 extents 1126400 ( 71%).           Updating bitmap for NAFA_DM3_Pool2 extents 1136640 ( 71%).           Updating bitmap for NAFA_DM3_Pool2 extents 1146880 ( 72%).           Updating bitmap for NAFA_DM3_Pool2 extents 1157120 ( 73%).           Updating bitmap for NAFA_DM3_Pool2 extents 1167360 ( 73%).           Updating bitmap for NAFA_DM3_Pool2 extents 1177600 ( 74%).           Updating bitmap for NAFA_DM3_Pool2 extents 1187840 ( 74%).           Updating bitmap for NAFA_DM3_Pool2 extents 1198080 ( 75%).           Updating bitmap for NAFA_DM3_Pool2 extents 1208320 ( 76%).           Updating bitmap for NAFA_DM3_Pool2 extents 1218560 ( 76%).           Updating bitmap for NAFA_DM3_Pool2 extents 1228800 ( 77%).           Updating bitmap for NAFA_DM3_Pool2 extents 1239040 ( 78%).           Updating bitmap for NAFA_DM3_Pool2 extents 1249280 ( 78%).           Updating bitmap for NAFA_DM3_Pool2 extents 1259520 ( 79%).           Updating bitmap for NAFA_DM3_Pool2 extents 1269760 ( 80%).           Updating bitmap for NAFA_DM3_Pool2 extents 1280000 ( 80%).           Updating bitmap for NAFA_DM3_Pool2 extents 1290240 ( 81%).           Updating bitmap for NAFA_DM3_Pool2 extents 1300480 ( 82%).           Updating bitmap for NAFA_DM3_Pool2 extents 1310720 ( 82%).           Updating bitmap for NAFA_DM3_Pool2 extents 1320960 ( 83%).           Updating bitmap for NAFA_DM3_Pool2 extents 1331200 ( 84%).           Updating bitmap for NAFA_DM3_Pool2 extents 1341440 ( 84%).           Updating bitmap for NAFA_DM3_Pool2 extents 1351680 ( 85%).           Updating bitmap for NAFA_DM3_Pool2 extents 1361920 ( 85%).           Updating bitmap for NAFA_DM3_Pool2 extents 1372160 ( 86%).           Updating bitmap for NAFA_DM3_Pool2 extents 1382400 ( 87%).           Updating bitmap for NAFA_DM3_Pool2 extents 1392640 ( 87%).           Updating bitmap for NAFA_DM3_Pool2 extents 1402880 ( 88%).           Updating bitmap for NAFA_DM3_Pool2 extents 1413120 ( 89%).           Updating bitmap for NAFA_DM3_Pool2 extents 1423360 ( 89%).           Updating bitmap for NAFA_DM3_Pool2 extents 1433600 ( 90%).           Updating bitmap for NAFA_DM3_Pool2 extents 1443840 ( 91%).           Updating bitmap for NAFA_DM3_Pool2 extents 1454080 ( 91%).           Updating bitmap for NAFA_DM3_Pool2 extents 1464320 ( 92%).           Updating bitmap for NAFA_DM3_Pool2 extents 1474560 ( 93%).           Updating bitmap for NAFA_DM3_Pool2 extents 1484800 ( 93%).           Updating bitmap for NAFA_DM3_Pool2 extents 1495040 ( 94%).           Updating bitmap for NAFA_DM3_Pool2 extents 1505280 ( 95%).           Updating bitmap for NAFA_DM3_Pool2 extents 1515520 ( 95%).           Updating bitmap for NAFA_DM3_Pool2 extents 1525760 ( 96%).           Updating bitmap for NAFA_DM3_Pool2 extents 1536000 ( 96%).           Updating bitmap for NAFA_DM3_Pool2 extents 1546240 ( 97%).           Updating bitmap for NAFA_DM3_Pool2 extents 1556480 ( 98%).           Updating bitmap for NAFA_DM3_Pool2 extents 1566720 ( 98%).           Updating bitmap for NAFA_DM3_Pool2 extents 1576960 ( 99%).           Updating bitmap for NAFA_DM3_Pool2 extents 1584394 (100%).                    Checking for dead inodes 307200 ( 29%).        Clearing  inode 0x80000529ca64.Checking for dead inodes 614400 ( 58%).        Clearing  inode 0x8000077ee888.Checking for dead inodes 819200 ( 77%).        Clearing  inode 0x8b84e72.Clearing  inode 0x8b84e74.Checking for dead inodes 1052672 (100%).          Checking directories 58747 (100%).         Scanning for orphaned inodes 1052672 (100%).        Verifying link &amp; subdir counts 1052672 (100%).          Super Block -  Free Inodes of 0xcf84 is wrong! Should be 0xcf8a.Repairing free list.Repairing free list. 1052672 (100%)Free inode count is wrong (was 0xcf84 should be 0xcf88)Repairing free list. 1052672 (100%)Verifying Free List Extents.Checking pending free list.                        Checking Arbitration Control Block.Checking NAFA_DM3_Metadata allocation bit maps (100%).         Checking NAFA_DM3_Pool1 allocation bit maps (100%).         Checking NAFA_DM3_Pool2 allocation bit maps (100%).         Updating super block.File system &#x27;NAFA_DM3&#x27; was modified.File system &#x27;NAFA_DM3&#x27;. Blocks-6835932800 free-2939148886 Inodes-1052672 free-53128.File System Check finished - file system modified.mdcthird:~ root# ________________________________________________________________________________ ____________Please advise if the Volume was really has corrupted file System and was it fixed after I ran the -wv command.Regards.

Blaidd Drwg · Answer

It looks like there were a few errors:Scanning inodes 307200 ( 29%).       *Error*: XATTR ERROR detected for inode 0x529ca63 (0x80000529ca64)*Error*: *** RESETTING RPL DATA FOR FILE SYSTEM ***Scanning inodes 819200 ( 77%).       *Error*: XATTR ERROR detected for inode 0x8b84e71 (0x8b84e73)Scanning inodes 1052672 (100%).cvfsck said the file system was modified, so hopefully it was fixed.