Q: Xsan Clobbered?

I'm posting this on behalf of some associates overseas who lack the Engilsh skills (this might have contributed to the trouble they got themselves into). They started out innocently enough, as part of routine maintenance on their 3-year old Xsan volume, they ran cvfsck -v,then cvfsck -wv, this is the output truncated:

*Fatal*: Cannot phys buff cache block [0x8f92d800]! -Input/output error
Super Block - Number of Inodes of 0xaec00 is wrong! Should be 0x71400.
Super Block - Next Inode Chunk of 0x51c4d is wrong! Should be 0x0.
 
Verifying NT Security Descriptors (NTSDs)
Found 54 NT Security Descriptors: 0 are bad
 
*Fatal*: Cannot phys buff cache block [0x800000f6c1a648]! -Input/output error
*Fatal*: scan_fl: Cannot cache Free List Extent block [0x800000f6c1a648] - Unknown error: 0
Free List Extents are Corrupt! Free List will be rebuilt.
*Fatal*: inode_read cannot find extender inode 0x800151dbb00000 in database!
 
*Error*: Cannot read inode 0x848100754da3cf.
*Fatal*: Can't find xattr inode 0x800151dbb00108 in database.
Clearing xattr chain for inode 0x80012345d0016e.
Clearing xattr chain for inode 0x8001d032d201f8.
Clearing xattr chain for inode 0x8001d032d201f9.
Clearing xattr chain for inode 0x8001d032d201fa.
Clearing xattr chain for inode 0x8001d032d201fb.
Clearing xattr chain for inode 0x8001d032d201fe.
Clearing xattr chain for inode 0x8001d032d201ff.
Clearing xattr chain for inode 0x8001d032d20202.
Clearing xattr chain for inode 0x8001d032d20204.
Clearing xattr chain for inode 0x8001d032d20206.
Clearing xattr chain for inode 0x8001d032d20207.
Clearing xattr chain for inode 0x8001d032d2020b.
Clearing xattr chain for inode 0x8001d032d2020f.
*Fatal*: inode_read cannot find extender inode 0x10002b98e9402fb in database!
 
*Error*: Cannot read inode 0x100822114d403dc.
*Fatal*: Can't find xattr inode 0x100027bf584013d in database.
Clearing xattr chain for inode 0x100027428180368.
*Fatal*: inode_read cannot find extender inode 0x100027bf5840002 in database!
 
*Error*: Cannot read inode 0x100027428180368.
*Fatal*: Can't find xattr inode 0x100027bf584013e in database.
Clearing xattr chain for inode 0x100027428180369.
*Fatal*: Can't find xattr inode 0x100027bf584015c in database.
Clearing xattr chain for inode 0x10002742818037f.
*Fatal*: inode_read cannot find extender inode 0x100027bf5840001 in database!
 
*Error*: Cannot read inode 0x10002742818037f.
*Fatal*: Can't find xattr inode 0x100027bf584015d in database.
Clearing xattr chain for inode 0x100027428180380.
*Fatal*: Can't find xattr inode 0x100027bf5840197 in database.
Clearing xattr chain for inode 0x10002742818039b.
*Fatal*: inode_read cannot find extender inode 0x100027bf5840000 in database!
 
*Error*: Cannot read inode 0x10002742818039b.
*Fatal*: Can't find xattr inode 0x100027bf5840198 in database.
Clearing xattr chain for inode 0x10002742818039c.
*Fatal*: Can't find xattr inode 0x10002ac11b0000f in database.
Clearing xattr chain for inode 0x10002a231f803a5.
*Fatal*: inode_read cannot find extender inode 0x10002ac11b00000 in database!
 
*Error*: Cannot read inode 0x10402a231f803a5.
*Fatal*: Can't find xattr inode 0x10002ac11b00010 in database.
Clearing xattr chain for inode 0x10002a231f803a6.
*Fatal*: Can't find xattr inode 0x10002ac11b0002b in database.
Clearing xattr chain for inode 0x10002a231f803ae.
*Fatal*: inode_read cannot find extender inode 0x10002ac11b00002 in database!
 
*Error*: Cannot read inode 0x10302a231f803ae.
*Fatal*: Can't find xattr inode 0x10002ac11b0002c in database.
Clearing xattr chain for inode 0x10002a231f803af.
*Fatal*: Can't find xattr inode 0x10002ac11b00036 in database.
Clearing xattr chain for inode 0x10002a231f803d3.
*Fatal*: inode_read cannot find extender inode 0x10002ac11b00001 in database!
 
*Error*: Cannot read inode 0x10382a231f803d3.
*Fatal*: Can't find xattr inode 0x10002ac11b00037 in database.
Clearing xattr chain for inode 0x10002a231f803d4.
Clearing Orphaned Xattr Inode 0x82804c17ba0200.
Clearing Orphaned Xattr Inode 0x808073f49c0016.
Clearing  Inode 0x848100754da3cf.
*Fatal*: Inode Clear cannot find extender inode 0x800151dbb00000
Clearing Orphaned Extender Inode 0x8281730f16005e.
Clearing Orphaned Extender Inode 0x8281730f16005f.
Clearing Orphaned Extender Inode 0x8201730f160060.
Clearing Orphaned Extender Inode 0x8201730f160061.
Clearing Orphaned Extender Inode 0x8201730f160062.
Clearing Orphaned Extender Inode 0x8201730f160063.
Clearing Orphaned Extender Inode 0x8201730f160064.
Clearing Orphaned Extender Inode 0x8281730f160065.
Clearing Orphaned Extender Inode 0x8281730f160066.
Clearing Orphaned Extender Inode 0x8201730f160067.
...
Clearing Orphaned Extender Inode 0x1018010ccd40190.
Clearing Orphaned Extender Inode 0x1018010ccd40192.
Clearing Orphaned Extender Inode 0x1018010ccd40193.
Clearing Orphaned Extender Inode 0x1018010ccd40195.
Clearing Orphaned Xattr Inode 0x1018010ccd40196.
Clearing Orphaned Xattr Inode 0x1018010ccd40197.
Clearing Orphaned Extender Inode 0x1018010ccd4019c.
Clearing Orphaned Extender Inode 0x1018010ccd4019e.
Clearing Orphaned Extender Inode 0x1018010ccd401a0.
Clearing Orphaned Extender Inode 0x1018010ccd401a2.
Clearing Orphaned Xattr Inode 0x1018010ccd401a3.
Clearing Orphaned Xattr Inode 0x1010010ccd401a4.
Clearing  Inode 0x100822114d403dc.
*Fatal*: Inode Clear cannot find extender inode 0x10002b98e9402fb
Clearing  Inode 0x100027428180368.
*Fatal*: Inode Clear cannot find extender inode 0x100027bf5840002
Clearing  Inode 0x10002742818037f.
*Fatal*: Inode Clear cannot find extender inode 0x100027bf5840001
Clearing  Inode 0x10002742818039b.
*Fatal*: Inode Clear cannot find extender inode 0x100027bf5840000
Clearing Orphaned Extender Inode 0x10202a231f80000.
Clearing Orphaned Extender Inode 0x10182a231f80003.
Clearing Orphaned Extender Inode 0x10502a231f80004.
Clearing Orphaned Extender Inode 0x10182a231f80006.
Clearing Orphaned Extender Inode 0x10502a231f80007.
Clearing Orphaned Extender Inode 0x10282a231f80009.
Clearing Orphaned Extender Inode 0x10282a231f8000a.
Clearing Orphaned Extender Inode 0x10302a231f8000c.
Clearing Orphaned Extender Inode 0x10502a231f8000d.
Clearing Orphaned Extender Inode 0x10282a231f8000f.
Clearing Orphaned Extender Inode 0x10282a231f80010.
Clearing Orphaned Extender Inode 0x10302a231f80012.
...
Directory entry folders.513 (p=0x178 i=0x8001f95a4c0195) has an invalid inode. Removing entry.
Directory entry 전국top10가요쇼-286회.mov (p=0x102015257c000dd i=0x10002ba10580335) has an invalid inode. Removing entry.
Directory entry 전국top10가요쇼-287회.mov (p=0x102015257c000dd i=0x10002ba102400ae) has an invalid inode. 
....Ö•·Ü´·ÑÄ·ÖÆ·Ü®top10·ÑÄ·Ö°·Ñã·Ö≠·Ñâ·Ö≠-297·Ñí·Ö¨.mov (p=0x102015257c000dd i=0x10002ba0db802f7) has an invalid inode. Removing entry.
Directory entry ._전국top10가요쇼-296회.mov (p=0x102015257c000dd i=0x10002ba10580332) has an invalid inode. Removing entry.
Directory entry .DS_Store (p=0x101022114d403d8 i=0x100029d37bc01ad) has an invalid inode. Removing entry.
Directory entry 0708 (p=0x101022114d403d9 i=0x100027bf5840169) has an invalid inode. Removing entry.
Directory entry .DS_Store (p=0x101022114d403d9 i=0x100029d37bc01ae) has an invalid inode. Removing entry.
Directory entry Untitled Project 1 (p=0x101022114d403d9 i=0x100822114d403dc flags=0x33) is an orphan. Removed.
*Error*: Cannot get directory data block 0x0: inode 0x101022114d403da- Input/output error.
*Error*: Inode 0x101022114d403da has no physical directory file.
Clearing corrupt directory Inode 0x100022114d403da.
*Warning*: Adding orphaned inode 0xbc5aa5f to root directory as FOUND.0xbc5aa5f.
*Warning*: Adding orphaned inode 0xbc5aa61 to root directory as FOUND.0xbc5aa61.
*Warning*: Adding orphaned inode 0xbc5aa63 to root directory as FOUND.0xbc5aa63.
*Warning*: Adding orphaned inode 0xbc5aa65 to root directory as FOUND.0xbc5aa65.
*Warning*: Adding orphaned inode 0xbc5aa67 to root directory as FOUND.0xbc5aa67.
....
*Warning*: Adding orphaned inode 0x10402a231f8038b to root directory as FOUND.0x10402a231f8038b.
*Warning*: Adding orphaned inode 0x10582a231f803a6 to root directory as FOUND.0x10582a231f803a6.
*Warning*: Adding orphaned inode 0x10182a231f803af to root directory as FOUND.0x10182a231f803af.
*Warning*: Adding orphaned inode 0x10382a231f803d4 to root directory as FOUND.0x10382a231f803d4.
Directory entry Render Files (p=0x101022114d403d8 i=0x101022114d403da flags=0x123) is an orphan. Removed.
*Error*: Inode 0x2 link count is 19 - should be 835. Fixed.
*Error*: Inode 0x2 nsubdirs count is 11 - should be 17. Fixed.
*Error*: Inode 0x9 link count is 14 - should be 2. Fixed.
*Error*: Inode 0x9 nsubdirs count is 8 - should be 2. Fixed.
*Error*: Inode 0x178 link count is 14 - should be 13. Fixed.
*Error*: Inode 0x178 nsubdirs count is 14 - should be 13. Fixed.
*Error*: Inode 0x102015257c000dd link count is 453 - should be 146. Fixed.
*Error*: Inode 0x101022114d403d8 link count is 6 - should be 4. Fixed.
*Error*: Inode 0x101022114d403d8 nsubdirs count is 5 - should be 4. Fixed.
*Error*: Inode 0x101022114d403d9 link count is 5 - should be 2. Fixed.
*Error*: Inode 0x101022114d403d9 nsubdirs count is 4 - should be 2. Fixed.
 
Super Block -  Free Inodes of 0x8c2cf is wrong! Should be 0x6aa1d.
 
Repairing free list.
 
Repairing free list. 0 (  0%)
Repairing free list. 4639 (  1%)
Repairing free list. 9278 (  2%)
...
Repairing free list. 459234 ( 99%)
Repairing free list. 463872 (100%)
Repairing free list. 463872 (100%)
*Fatal*: Cannot phys buff cache block [0x800000f6c1a648]! -Input/output error
*Fatal*: scan_fl: Cannot cache Free List Extent block [0x800000f6c1a648] - Unknown error: 0
*Warning*: Posssible Free List B-Tree internal inconsistency detected.
Please refer to the -C option in the cvfsck documentation.
 
Super Block -  Free List Size of 0x71 is wrong! Should be 0x48.
 
Map file 'MDJ' is incorrect. Updating.
Map file 'POOL_1' is incorrect. Updating.
Map file 'POOL_2' is incorrect. Updating.
Map file 'POOL_3' is incorrect. Updating.
 
Updating super block.
 
File system 'Xsan_Vol' was modified.
 
File system 'Xsan_Vol'. Blocks-8180349184 free-7365347349 Inodes-463872 free-436765.

*Fatal*: Cannot phys buff cache block [0x8f92d800]! -Input/output error
 
Verifying NT Security Descriptors (NTSDs)
Found 54 NT Security Descriptors: 0 are bad
 
*Fatal*: Cannot phys buff cache block [0x800000f6c1a648]! -Input/output error
*Fatal*: scan_fl: Cannot cache Free List Extent block [0x800000f6c1a648] - Unknown error: 0
Free List Extents are Corrupt! Free List will be rebuilt.
 
Repairing free list.
 
Repairing free list. 0 (  0%)
Repairing free list. 4639 (  1%)
Repairing free list. 9278 (  2%)
Repairing free list. 13917 (  3%)
Repairing free list. 454595 ( 98%)
Repairing free list. 459234 ( 99%)
Repairing free list. 463872 (100%)
Repairing free list. 463872 (100%)
*Fatal*: Cannot phys buff cache block [0x800000f6c1a648]! -Input/output error
*Fatal*: scan_fl: Cannot cache Free List Extent block [0x800000f6c1a648] - Unknown error: 0
*Warning*: Posssible Free List B-Tree internal inconsistency detected.
Please refer to the -C option in the cvfsck documentation.
 
 
Map file 'POOL_1' is incorrect. Updating.
 
Updating super block.
 
File system 'Xsan_Vol' was modified.
 
File system 'Xsan_Vol'. Blocks-8180349184 free-7365347350 Inodes-463872 free-436765.

Now, because they saw this:

Warning: Posssible Free List B-Tree internal inconsistency detected.
Please refer to the -C option in the cvfsck documentation.

They unfortunately proceed to run cvfsck -C

*Fatal*: Cannot phys buff cache block [0x8f92d800]! -Input/output error
 
Verifying NT Security Descriptors (NTSDs)
Found 54 NT Security Descriptors: 0 are bad
 
*Fatal*: Cannot phys buff cache block [0x800000f6c1a648]! -Input/output error
*Fatal*: scan_fl: Cannot cache Free List Extent block [0x800000f6c1a648] - Unknown error: 0
Free List Extents are Corrupt! Free List will be rebuilt.
436765 free inodes marked for deferred reclaim.
0 pending inodes marked for deferred reclaim.
 
Clobbering Free List
 
Updating super block.
 
File system 'Xsan_Vol' was modified.
 
File system 'Xsan_Vol'. Blocks-8180349184 free-8180286935 Inodes-463872 free-0.

Since then, they cannot start their Xsan volume.

*Fatal*: Cannot phys buff cache block [0x8f92d800]! -Input/output error
 
Verifying NT Security Descriptors (NTSDs)
Found 54 NT Security Descriptors: 0 are bad
 
Map file 'MDJ' is incorrect. Updating.
Map file 'POOL_1' is incorrect. Updating.
Map file 'POOL_2' is incorrect. Updating.
Map file 'POOL_3' is incorrect. Updating.
 
File system 'Xsan_Vol' was modified.
 
File system 'Xsan_Vol'. Blocks-8180349184 free-7365347421 Inodes-463872 free-0.

This error seems to be the fatal one:
Fatal: Cannot phys buff cache block [0x8f92d800]! -Input/output error
And appears before and after the clobbering, so perhaps the Clobbering is not ultimately the problem.

Seems like they have moved/misplaced their metadata. Any ideas how to fix this or things for them to try? No way to re-generate a clean set of metadata? (It's late, I know...)