Xsan crashes
Been getting some file system crashes in xsan causing the metadata controllers to freeze up and drop the clients. I've checked the volume with cvfsk and it looks fine.
The main error we're seeing on both metadata controllers is this:
Process: fsm [4168]
Path: /System/Library/Filesystems/acfs.fs/Contents/bin/fsm
Identifier: fsm
Version: 564
Code Type: X86-64 (Native)
Parent Process: fsmpm [298]
Responsible: fsm [4168]
User ID: 0
Date/Time: 2018-05-29 11:02:20.358 -0700
OS Version: Mac OS X 10.11.6 (15G20015)
Report Version: 11
Anonymous UUID: 5ED82D91-3698-A1DB-BB17-95F9F03DE7E7
Time Awake Since Boot: 330000 seconds
System Integrity Protection: enabled
Crashed Thread: 188
Exception Type: EXC_CRASH (SIGABRT)
Exception Codes: 0x0000000000000000, 0x0000000000000000
Exception Note: EXC_CORPSE_NOTIFY
Application Specific Information:
PANIC: /System/Library/Filesystems/acfs.fs/Contents/bin/fsm "OpHangLimitSecs exceeded VOP-Class-0 Type-4 Subtype-11 362 ticks Conn[6] Thread-0x7000061ce000 Pqueue-0x7ffb83429a38 Workp-0x7ffb841f2420 MsgQ-0x7ffb841f2408 Msg-0x7ffb841f2498 now 375 started 13 limit 360 (180 secs.)
" file /Library/Caches/com.apple.xbs/Sources/XsanFS/XsanFS-564/snfs/fsm/queues.c, line 330
But we have also gotten this kernel panic on one of the systems:
Anonymous UUID: 5ED82D91-3698-A1DB-BB17-95F9F03DE7E7
Thu May 17 14:55:58 2018
*** Panic Report ***
panic(cpu 0 caller 0xffffff7f834692f9): "CVFS ASSERTION FAILED: vcp_error == 0 line 2154 file /Library/Caches/com.apple.xbs/Sources/XsanFS/XsanFS-564/snfs/client/vfs/nomad/m d_vnops.c"@/Library/Caches/com.apple.xbs/Sources/XsanFS/XsanFS-564/snfs/client/v fs/nomad/md_debuglog.c:358
Backtrace (CPU 0), Frame : Return Address
0xffffff800c84b8c0 : 0xffffff80008dbb52
0xffffff800c84b940 : 0xffffff7f834692f9
0xffffff800c84b9a0 : 0xffffff8000affeaa
0xffffff800c84ba20 : 0xffffff8000aff9d6
0xffffff800c84ba60 : 0xffffff8000af91f2
0xffffff800c84baf0 : 0xffffff8000b04ca3
0xffffff800c84bb90 : 0xffffff8000b028fe
0xffffff800c84bda0 : 0xffffff8000dbab84
0xffffff800c84be30 : 0xffffff8000dba111
0xffffff800c84bf60 : 0xffffff8000e28d21
0xffffff800c84bfb0 : 0xffffff80009ed256
Kernel Extensions in backtrace:
com.apple.filesystems.acfs(564.0)[1D8DDDCE-20C1-3D8F-ACA8-05F504F6E9A2]@0xfffff f7f833c6000->0xffffff7f83509fff
dependency: com.apple.iokit.IOStorageFamily(2.1)[DC1AAB7C-F417-3238-BB3F-2A5B84D67B90]@0xff ffff7f8104c000
BSD process name corresponding to current thread: KernelEventAgent
Mac OS version:
15G20015
Kernel version:
Darwin Kernel Version 15.6.0: Tue Jan 30 11:45:51 PST 2018; root:xnu-3248.73.8~1/RELEASE_X86_64
Kernel UUID: ECB8B1A1-7E15-34CB-825F-C366E2635655
Kernel slide: 0x0000000000600000
Kernel text base: 0xffffff8000800000
__HIB text base: 0xffffff8000700000
System model name: Macmini7,1 (Mac-35C5E08120C7EEAF)
System uptime in nanoseconds: 875907419593346
last loaded kext at 106647566166: com.apple.filesystems.smbfs 3.0.1 (addr 0xffffff7f814f5000, size 413696)
last unloaded kext at 116948853907: com.apple.driver.AppleIntelLpssGspi 2.0.60 (addr 0xffffff7f82dd7000, size 45056)
loaded kexts:
com.promise.SANLink2-FC 1.2.9
com.apple.filesystems.smbfs 3.0.1
com.apple.driver.AudioAUUC 1.70