Looks like no one’s replied in a while. To start the conversation again, simply ask a new question.

Level 1

18 points

Xsan crashes

Been getting some file system crashes in xsan causing the metadata controllers to freeze up and drop the clients. I've checked the volume with cvfsk and it looks fine.

The main error we're seeing on both metadata controllers is this:

Process: fsm [4168]

Path: /System/Library/Filesystems/acfs.fs/Contents/bin/fsm

Identifier: fsm

Version: 564

Code Type: X86-64 (Native)

Parent Process: fsmpm [298]

Responsible: fsm [4168]

User ID: 0

Date/Time: 2018-05-29 11:02:20.358 -0700

OS Version: Mac OS X 10.11.6 (15G20015)

Report Version: 11

Anonymous UUID: 5ED82D91-3698-A1DB-BB17-95F9F03DE7E7

Time Awake Since Boot: 330000 seconds

System Integrity Protection: enabled

Crashed Thread: 188

Exception Type: EXC_CRASH (SIGABRT)

Exception Codes: 0x0000000000000000, 0x0000000000000000

Exception Note: EXC_CORPSE_NOTIFY

Application Specific Information:

PANIC: /System/Library/Filesystems/acfs.fs/Contents/bin/fsm "OpHangLimitSecs exceeded VOP-Class-0 Type-4 Subtype-11 362 ticks Conn[6] Thread-0x7000061ce000 Pqueue-0x7ffb83429a38 Workp-0x7ffb841f2420 MsgQ-0x7ffb841f2408 Msg-0x7ffb841f2498 now 375 started 13 limit 360 (180 secs.)

" file /Library/Caches/com.apple.xbs/Sources/XsanFS/XsanFS-564/snfs/fsm/queues.c, line 330

But we have also gotten this kernel panic on one of the systems:

Anonymous UUID: 5ED82D91-3698-A1DB-BB17-95F9F03DE7E7

Thu May 17 14:55:58 2018

*** Panic Report ***

panic(cpu 0 caller 0xffffff7f834692f9): "CVFS ASSERTION FAILED: vcp_error == 0 line 2154 file /Library/Caches/com.apple.xbs/Sources/XsanFS/XsanFS-564/snfs/client/vfs/nomad/m d_vnops.c"@/Library/Caches/com.apple.xbs/Sources/XsanFS/XsanFS-564/snfs/client/v fs/nomad/md_debuglog.c:358

Backtrace (CPU 0), Frame : Return Address

0xffffff800c84b8c0 : 0xffffff80008dbb52

0xffffff800c84b940 : 0xffffff7f834692f9

0xffffff800c84b9a0 : 0xffffff8000affeaa

0xffffff800c84ba20 : 0xffffff8000aff9d6

0xffffff800c84ba60 : 0xffffff8000af91f2

0xffffff800c84baf0 : 0xffffff8000b04ca3

0xffffff800c84bb90 : 0xffffff8000b028fe

0xffffff800c84bda0 : 0xffffff8000dbab84

0xffffff800c84be30 : 0xffffff8000dba111

0xffffff800c84bf60 : 0xffffff8000e28d21

0xffffff800c84bfb0 : 0xffffff80009ed256

Kernel Extensions in backtrace:

com.apple.filesystems.acfs(564.0)[1D8DDDCE-20C1-3D8F-ACA8-05F504F6E9A2]@0xfffff f7f833c6000->0xffffff7f83509fff

dependency: com.apple.iokit.IOStorageFamily(2.1)[DC1AAB7C-F417-3238-BB3F-2A5B84D67B90]@0xff ffff7f8104c000

BSD process name corresponding to current thread: KernelEventAgent

Mac OS version:

15G20015

Kernel version:

Darwin Kernel Version 15.6.0: Tue Jan 30 11:45:51 PST 2018; root:xnu-3248.73.8~1/RELEASE_X86_64

Kernel UUID: ECB8B1A1-7E15-34CB-825F-C366E2635655

Kernel slide: 0x0000000000600000

Kernel text base: 0xffffff8000800000

__HIB text base: 0xffffff8000700000

System model name: Macmini7,1 (Mac-35C5E08120C7EEAF)

System uptime in nanoseconds: 875907419593346

last loaded kext at 106647566166: com.apple.filesystems.smbfs 3.0.1 (addr 0xffffff7f814f5000, size 413696)

last unloaded kext at 116948853907: com.apple.driver.AppleIntelLpssGspi 2.0.60 (addr 0xffffff7f82dd7000, size 45056)

loaded kexts:

com.promise.SANLink2-FC 1.2.9

com.apple.filesystems.smbfs 3.0.1

com.apple.driver.AudioAUUC 1.70

Posted on May 29, 2018 1:16 PM

Similar questions

5 replies

etblack Author

Level 1

18 points

Jun 11, 2018 8:58 AM in response to Logan Wu

We had a few items that might have been behind it. The Metadata controllers and one of the clients were on El Capitan and one of the clients had been upgraded to Sierra. We upgraded everything to Sierra and in the process found that there was a mistake in the dns entry for one of the metadata controllers (it wouldn't rejoin the open directory because of that). Got that resolved and also figured out that the on one of the workstations they would regularly unplug the second monitor, which is where the ethernet connection for the metadata network runs, so had to do some user education there.

Since then they haven't reported any major issues.

Logan Wu

Level 1

4 points

Jun 11, 2018 11:15 PM in response to etblack

Many thanks. I want to know the ACL of your xsan is enabled or not. what kind of account type clients are using?

all network accounts ? network & local accounts? or just local accounts?

MDC1 mdworker[54759]: [13:53:41.422] <<< FFR_Common >>> FigFormatReaderCreateForStreamReturningMatchingContainerLabelType: *** Could not create format reader for URL file:///Volumes/Media/Movie/%E5%88%81%E6%B0%91/Output/To%20Music/180611/180611% 20cut%20Wenders%20Reel1-%E5%AE%BD%E5%B8%A6%EF%BC%88%E9%AB%98%EF%BC%89.mp4 FileType '' MIMETYPE (null) SuggFileName (null) instantiationError = -12848

MDC1 kernel[0]: **** [IOBluetoothHostControllerUSBTransport][ClearFeatureInterruptEndpointHalt] -- successfully posting another read for the mInt0InterruptPipe -- mInterruptPipeInOutstandingIOCount = 1 -- this = 0xe800

We have lots of messages like this. Although we have set the network account and permissions for client Macs, editors are still using local account. 😟

Logan Wu

Level 1

4 points

Jun 11, 2018 4:37 AM in response to etblack

I have the same problem. How is it going on about the problem?

etblack Author

Level 1

18 points

Jun 12, 2018 9:29 AM in response to Logan Wu

Yeah, all of our accounts are network accounts. They are ultimately being pulled from AD. We have all system connected to OD and AD. ACLs are active.

etblack Author

Level 1

18 points

Jun 12, 2018 9:32 AM in response to etblack

Since my last set of changes, upgrading everything to Sierra and fixing metadata connection and dns entries we haven't had a single crash or failover. Going on 150 hours of uptime.

Xsan crashes