Unable to mount Xsan volume

Came in this morning and our Xsan volume had a connection failure on all clients. I can remote into the primary dc and the secondary dc. Connections to the volume failed on the dcs as well. I rebooted the metadata network switch. I have not bounced the fibre channel switch but all lights are green on the switch and on the drive cages. I'm seeing a warning ! on three of the LUNs in the list.

Ideas? I'm dead in the water.

Xsan Volume log on the primary dc for today is: http://pastebin.com/1fgjqvmN and also:

[1001 06:40:26] 0xb040f000 (Warning) pio_directio fd 18 retry 0 bufaddr 0x959c08 seekaddr 0x484000 len 512 rw 0x0 ERROR 5!

[1001 06:40:26] 0xb040f000 (Warning) pio_directio fd 18 retry 1 bufaddr 0x959c08 seekaddr 0x484000 len 512 rw 0x0 ERROR 5!

[1001 06:40:26] 0xb040f000 (Warning) pio_directio fd 18 retry 2 bufaddr 0x959c08 seekaddr 0x484000 len 512 rw 0x0 ERROR 5!

[1001 06:40:27] 0xb040f000 (Warning) pio_directio fd 18 retry 3 bufaddr 0x959c08 seekaddr 0x484000 len 512 rw 0x0 ERROR 5!

[1001 06:40:27] 0xb040f000 (Warning) pio_directio fd 18 retry 4 bufaddr 0x959c08 seekaddr 0x484000 len 512 rw 0x0 ERROR 5!

[1001 06:40:27] 0xa0082720 (**Error**) PioIO: pio_status/5

[1001 06:40:27] 0xa0082720 (**FATAL**) PANIC: /Library/Filesystems/Xsan/bin/fsm "Cannot read arbitration block! - [errno 36]: Operation now in progress

" file arbitration_block.c, line 186

[1001 06:40:27] 0xa0082720 (**FATAL**) PANIC: aborting threads now.

Logger_thread: sleeps/22787070 signals/0 flushes/14 writes/14 switches 0

Logger_thread: logged/52 clean/52 toss/0 signalled/0 toss_message/0

Logger_thread: waited/0 awakened/0

[1001 06:40:38] 0xa0082720 (Info) Server Revision 3.5.0 Build 7443 Branch branches_35X (411.3)

[1001 06:40:38] 0xa0082720 (Info) Built for Darwin 9.0 i386

[1001 06:40:38] 0xa0082720 (Info) Created on Mon Dec 7 12:35:35 PST 2009

[1001 06:40:38] 0xa0082720 (Info) Built in /SourceCache/XsanFS/XsanFS-411.3

[1001 06:40:38] 0xa0082720 (Info)

Configuration:

DiskTypes-6

Disks-6

StripeGroups-3

MaxConnections-139

ThreadPoolSize-256

StripeAlignSize-256

FsBlockSize-16384

BufferCacheSize-32M

InodeCacheSize-8192

RestoreJournal-Disabled

RestoreJournalDir-None

[1001 06:40:38] 0xa0082720 (**Error**) PioOpen: no device for disk "vtrak1_metadata".

[1001 06:40:38] 0xa0082720 (**FATAL**) Cannot open stripe group "MetadataAndJournal" - Operation not supported by device

Logger_thread: sleeps/1 signals/0 flushes/1 writes/1 switches 0

Logger_thread: logged/7 clean/7 toss/0 signalled/0 toss_message/0

Logger_thread: waited/0 awakened/0