servermgrd 100% CPU after Server 5.1 upgrade (Time Machine related)

After upgrading to Server 5.1, (from Server 5.0) the Server application became unresponsive (both locally, and via a remote connection) after I looked at the Time Machine settings.

Looking at Activity Monitor, I see that the process servermgrd is at around 100% (of a single CPU on a multi core machine), and has been that way for over a day.

Looking at the Logs, I see

Mar 26 00:05:30 REDACTED kernel[0]: process servermgrd[19195] thread 240535 caught burning CPU! It used more than 50% CPU (Actual recent usage: 99%) over 180 seconds. thread lifetime cpu usage 90.041337 seconds, (0.011734 user, 90.029603 system) ledger info: balance: 90025584783 credit: 90025584783 debit: 0 limit: 90000000000 (50%) period: 180000000000 time since last refill (ns): 90693005282

Mar 26 00:05:35 REDACTED spindump[703]: Saved cpu_resource.diag report for servermgrd version ??? (???) to /Library/Logs/DiagnosticReports/servermgrd_2016-03-26-000535_REDACTED.cpu_resource.diag

Looking at the spindump, I see

Date/Time: 2016-03-26 00:16:54 +1300

OS Version: Mac OS X 10.11.4 (Build 15E65)

Architecture: x86_64

Report Version: 19

Command: servermgrd

Path: /Applications/Server.app/Contents/ServerRoot/System/Library/CoreServices/Server ManagerDaemon.bundle/Contents/MacOS/servermgrd

Version: ??? (???)

Parent: launchd [1]

PID: 9333

Event: cpu usage (microstackshots only)

Thread: 0x67e7 (99% cpu over 91 seconds)

Duration: 91.00s

Steps: 87

Hardware model: Macmini7,1

Active cpus: 4

Fan speed: 1802 rpm

Powerstats for: servermgrd [9333] thread 0x67e7

UUID: 1082DB70-9600-3102-82A9-F83A1CB10AA8

Start time: 2016-03-26 00:16:55 +1300

End time: 2016-03-26 00:18:25 +1300

Parent: launchd

Microstackshots: 68 samples (78%)

Primary state: 68 samples Non-Frontmost App, Kernel mode, Thread QoS Legacy

User Activity: 0 samples Idle, 68 samples Active

Power Source: 0 samples on Battery, 68 samples on AC

68 start_wqthread + 13 (libsystem_pthread.dylib) [0x7fff878a5341]

68 _pthread_wqthread + 1129 (libsystem_pthread.dylib) [0x7fff878a74de]

68 _dispatch_worker_thread3 + 91 (libdispatch.dylib) [0x7fff90eedb00]

68 _dispatch_root_queue_drain + 538 (libdispatch.dylib) [0x7fff90eedd53]

68 _dispatch_queue_invoke + 549 (libdispatch.dylib) [0x7fff90ef5707]

68 _dispatch_queue_drain + 754 (libdispatch.dylib) [0x7fff90eef03b]

68 _dispatch_client_callout + 8 (libdispatch.dylib) [0x7fff90eea40b]

68 _dispatch_call_block_and_release + 12 (libdispatch.dylib) [0x7fff90ef593d]

68 __45-[GenericPlugin doCommand:forUser:withReply:]_block_invoke + 646 (libservermgrcommon.dylib) [0x10259e326]

68 -[PluginRequestHandler doProcessInputWithRequest:context:lockFileFD:] + 3304 (libservermgrcommon.dylib) [0x1025a5300]

68 -[NSInvocation invoke] + 286 (CoreFoundation) [0x7fff972b48fe]

68 __invoking___ + 140 (CoreFoundation) [0x7fff972b4a6c]

68 ??? (servermgr_timemachine + 9855) [0x1029e567f]

68 -[NSEnumerator countByEnumeratingWithState:objects:count:] + 56 (CoreFoundation) [0x7fff97290c48]

68 -[NSURLDirectoryEnumerator nextObject] + 68 (Foundation) [0x7fff8c7f1d69]

68 _URLEnumeratorGetNextURL + 167 (CoreServicesInternal) [0x7fff980391fe]

68 _GetDirectoryURLs(_CFURLEnumerator*) + 383 (CoreServicesInternal) [0x7fff98039bfb]

68 getattrlistbulk + 10 (libsystem_kernel.dylib) [0x7fff96998be2]

Binary Images:

0x102579000 - 0x10257afff servermgrd (33) <1082DB70-9600-3102-82A9-F83A1CB10AA8> /Applications/Server.app/Contents/ServerRoot/System/Library/CoreServices/Server ManagerDaemon.bundle/Contents/MacOS/servermgrd

0x102584000 - 0x1025b6ff7 libservermgrcommon.dylib (289) <AEE9069D-FE11-3A21-AD36-EA3846D95302> /Applications/Server.app/Contents/ServerRoot/usr/lib/libservermgrcommon.dylib

0x1029e3000 - 0x1029e6fff com.apple.servermgr.timemachine 10.11 (22) <6086D67C-13B8-3BF3-BB3E-9CAF2D1371C5> /Applications/Server.app/Contents/ServerRoot/usr/share/servermgrd/bundles/serve rmgr_timemachine.bundle/Contents/MacOS/servermgr_timemachine

0x7fff878a4000 - 0x7fff878adff7 libsystem_pthread.dylib (138.10.4) <3DD1EF4C-1D1B-3ABF-8CC6-B3B1CEEE9559> /usr/lib/system/libsystem_pthread.dylib

0x7fff8c776000 - 0x7fff8cacafff com.apple.Foundation 6.9 (1258) <51833143-6CAE-3E1C-9FBA-BCDEB48D4ADF> /System/Library/Frameworks/Foundation.framework/Versions/C/Foundation

0x7fff90ee8000 - 0x7fff90f15fff libdispatch.dylib (501.40.12) <C7499857-61A5-3D7D-A5EA-65DCC8C3DF92> /usr/lib/system/libdispatch.dylib

0x7fff96981000 - 0x7fff9699fff7 libsystem_kernel.dylib (3248.40.184) <88C17B7F-1CD8-3979-A1A9-F7BDB4FCE789> /usr/lib/system/libsystem_kernel.dylib

0x7fff9722e000 - 0x7fff976a4fff com.apple.CoreFoundation 6.9 (1258.1) <943A1383-DA6A-3DC0-ABCD-D9AEB3D0D34D> /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation

0x7fff98016000 - 0x7fff98044ff7 com.apple.CoreServicesInternal 248.2 (248.2) <6E111F0A-D7F1-3738-ADE7-CF983BD4EC8B> /System/Library/PrivateFrameworks/CoreServicesInternal.framework/Versions/A/Cor eServicesInternal

Powerstats for: mdworker

UUID: 7B5EAB64-3986-3267-9097-E954D8ED544E

Start time: 2016-03-26 00:17:05 +1300

End time: 2016-03-26 00:17:19 +1300

Any idea what might be causing this? I have several TB of existing Time Machine backups, in case the system is checking them for some unfathomable reason. I have left the system for over a day, in case it might eventually finish, but it shows no signs so far.

The rest of the system works OK. Server reports out of data TM backups; File Sharing works; DNS works; OD works; Websites works; machines can back up to the Time Machine sparesebundles...

I have rebooted several times. I have moved Server out /Applications and back in to 're-insatll' Several times. Whenever I go to look at the Time Machine, the Server GUI more or less locks up (some things still work) and servermgrd sits and spins.

Things were working fine on Server 5.0.

Posted on Mar 26, 2016 8:50 PM

26 replies

Apr 20, 2016 3:21 AM in response to Hans Eriksson

Aha! This is sounding promising, for we have something in common: multiple TimeMachine entries!

I have a couple of Servers running. One has a single Backup Destination. It doesn't exhibit the problem.

The other has seven. This is the one with the problem.

Can we verify that this is the sole necessary condition?

Apr 20, 2016 4:02 AM in response to Michael Newbery

I would agree. I had that suspicion myself so I merged my two TM share-points into one two days ago and the servermgrd has behaved nicely since then.

It sure points towards a problem with more than one TM entry.

I am not familiar with the procedures, how is this raised to Apple as a bug? Or are these discussions monitored by Apple?

Apr 20, 2016 4:07 AM in response to Linc Davis

Linc, I run the script, but when it finishes, there is nothing on the clipboard. Command-V doesn't do anything.

Apr 20, 2016 4:12 AM in response to Hans Eriksson

Hans, we have one more thing in common: we are both running ZFS. Are your (multiple) Backup Destinations on ZFS? Some (more than one) of mine are.

Apr 20, 2016 7:49 AM in response to Michael Newbery

Ah, another ZFS friend :-)

Yes, the targets were both on ZFS volumes. I had that setup running since 6-7 months without a hiccup. So it points to the latest version of Server which did "something" differently and fatally.

Apr 22, 2016 7:09 PM in response to Linc Davis

OK, I finally managed to get Linc's script to save to the clipboard. Restarting the server may have helped—I don't know. Anyway, I can't see anything particularly suspicious, just servermgrd spinning away…

http://pastebin.com/3EJJ1v80

Apr 22, 2016 11:36 PM in response to Michael Newbery

After I removed the second TM share-point the server has acted like it used to. I will let the dust settle for a few days more and then I will add a second TM share-point and I will see what happens.

I guess having more than one share-point is unusual so the testing was not extensive enough.

Linc Davis

Level 10

209,739 points

Apr 23, 2016 6:55 PM in response to Michael Newbery

Please uninstall the ZFS driver and see whether there's an improvement.

Apr 23, 2016 9:42 PM in response to Linc Davis

If I uninstall ZFS, I don't have any TimeMachine datasets. So, that's not changing just a single variable.

I'm currently playing with Server in a VM to see if I can reproduce a minimal set of required conditions. Meanwhile, it would be helpful to hear from anyone else experiencing the same problem.

Linc Davis

Level 10

209,739 points

Apr 23, 2016 9:44 PM in response to Michael Newbery

Relying on that driver for the integrity of your backups is very unwise, in my opinion. Good luck.

Apr 23, 2016 10:26 PM in response to Linc Davis

The TM backups are just one of several layers of backup. That server is just a backup machine. It (the system, not the backups) is backed up to another server, meanwhile its backups are backed up offsite.

The ZFS backups have daily snapshots, so if anything fails, I can roll back to an earlier TimeMachine—which I've done many times, owing to TM's unhappy habit of corrupting itself. As TM stores the backups in sparsebundles, ZFS snapshots are extremely efficient spacewise. The sparsebundles themselves are further backed up using one of the offsite backup companies.

The ZFS file systems are a RAID, to protect against disk failure.

I have been using ZFS for some years now, and it has never lost anything, indeed it has protected me against catastrophic disk failure on multiple occasions, without which protection I would have lost my TM backups.

This thread has been closed by the system or the community team. You may vote for any posts you find helpful, or search the Community for additional answers.

servermgrd 100% CPU after Server 5.1 upgrade (Time Machine related)