PMU Failure codes

This is the watchdog timer expired.

Mac OS X Server has a watchdog that detects when the systems hangs and automatically reboots the server.
Despite what it says, the error message you're seeing is the effect of the watchdog, but not the cause.
There's nothing I've found anywhere that explains the conditions that trigger the watchdog.

I've been having the same problem over and over again. So far, I've found that that '-93' relates to a process locking up or whatever, and '-122' relates to power issues, such as if your UPS is going bonkers. We solved the latter, but still haven't found which process locks up the box and causes the watchdog timer to expire.

We run OD, SMB, AFP, Filemaker Server, CommuniGate Pro plus print and DNS services on a lightly loaded single CPU Xserve running Panther (10.3.9).

Dec 1 15:37:27 localhost kernel: ApplePMU::PMU FORCED SHUTDOWN, CAUSE = -93
Dec 12 01:39:32 localhost kernel: ApplePMU::PMU FORCED SHUTDOWN, CAUSE = -93
Dec 14 09:33:41 localhost kernel: ApplePMU::PMU FORCED SHUTDOWN, CAUSE = -93
Dec 19 13:48:03 localhost kernel: ApplePMU::PMU FORCED SHUTDOWN, CAUSE = -93
Jan 2 13:41:47 localhost kernel: ApplePMU::PMU FORCED SHUTDOWN, CAUSE = -93

No clue what causes it. The fact that every app has its own logfiles, crash logs per user etc. isn't exactly helping with the search for the cause of this problem.

Doesn't seem to be any real pattern to it, and it is becoming quite annoying, really.

Does anyone have any clue where to go look for possible sources of this problem? Our Xserve just went down again, and somehow couldn't find its 'DataDisk' volume, mounted it under 'DataDisk 1', which basically meant that nothing works.

We'll be moving services (Filemaker, CommuniGate Pro, Samba) to Linux or Windows boxes, because at this point, the OS X Server is too friggin' unreliable. Sheesh.

Any suggestions more than welcome...

It happened again today, twice.

Jan 3 09:43:42 localhost kernel: ApplePMU::PMU FORCED SHUTDOWN, CAUSE = -93
Jan 3 10:54:29 localhost kernel: ApplePMU::PMU FORCED SHUTDOWN, CAUSE = -93

Checked the 'watchdog.event.log' in '/Library/Logs':

Start-Date: 2006-01-03 09:45:13 CET
2006-01-03 09:45:13 CET Started child "/usr/sbin/PasswordService" as pid 337.
2006-01-03 09:45:13 CET Started child "/usr/sbin/PrintServiceMonitor" as pid 338.
2006-01-03 09:45:13 CET Started child "/usr/libexec/postfix/master" as pid 339.
2006-01-03 09:45:13 CET Started child "/usr/sbin/hwmond" as pid 340.
2006-01-03 09:45:13 CET Started child "/usr/sbin/kadmind" as pid 341.
2006-01-03 09:45:13 CET Started child "/usr/sbin/krb5kdc" as pid 342.
2006-01-03 09:45:13 CET Automatic reboot timer enabled.
2006-01-03 09:56:15 CET Terminating due to signal.
2006-01-03 09:56:15 CET Automatic reboot timer disabled.
2006-01-03 09:56:15 CET Reaped child process 341 ("/usr/sbin/kadmind"); quit with exit status 2.
2006-01-03 09:56:15 CET Reaped child process 339 ("/usr/libexec/postfix/master"); quit due to signal 15.
2006-01-03 09:56:15 CET Reaped child process 342 ("/usr/sbin/krb5kdc"); quit with exit status 0.
2006-01-03 09:56:15 CET Reaped child process 340 ("/usr/sbin/hwmond"); quit due to signal 15.
2006-01-03 09:56:17 CET Reaped child process 337 ("/usr/sbin/PasswordService"); quit with exit status 0.
2006-01-03 09:56:21 CET Waiting for KILLed child (pid 338)
2006-01-03 09:56:21 CET Reaped child process 338 ("/usr/sbin/PrintServiceMonitor"); quit due to signal 9.
#End-Date: 2006-01-03 09:56:21 CET
#Start-Date: 2006-01-03 09:58:23 CET
2006-01-03 09:58:23 CET Started child "/usr/sbin/PasswordService" as pid 316.
2006-01-03 09:58:23 CET Started child "/usr/sbin/PrintServiceMonitor" as pid 317.
2006-01-03 09:58:23 CET Started child "/usr/libexec/postfix/master" as pid 318.
2006-01-03 09:58:23 CET Started child "/usr/sbin/hwmond" as pid 319.
2006-01-03 09:58:23 CET Started child "/usr/sbin/kadmind" as pid 320.
2006-01-03 09:58:23 CET Started child "/usr/sbin/krb5kdc" as pid 321.
2006-01-03 09:58:23 CET Automatic reboot timer enabled.
2006-01-03 10:03:49 CET Terminating due to signal.
2006-01-03 10:03:49 CET Automatic reboot timer disabled.
2006-01-03 10:03:49 CET Reaped child process 321 ("/usr/sbin/krb5kdc"); quit with exit status 0.
2006-01-03 10:03:49 CET Reaped child process 320 ("/usr/sbin/kadmind"); quit with exit status 2.
2006-01-03 10:03:49 CET Reaped child process 318 ("/usr/libexec/postfix/master"); quit due to signal 15.
2006-01-03 10:03:49 CET Reaped child process 319 ("/usr/sbin/hwmond"); quit due to signal 15.
2006-01-03 10:03:51 CET Reaped child process 316 ("/usr/sbin/PasswordService"); quit with exit status 0.
2006-01-03 10:03:55 CET Waiting for KILLed child (pid 317)
2006-01-03 10:03:55 CET Reaped child process 317 ("/usr/sbin/PrintServiceMonitor"); quit due to signal 9.
#End-Date: 2006-01-03 10:03:55 CET
#Start-Date: 2006-01-03 10:05:51 CET
2006-01-03 10:05:51 CET Started child "/usr/sbin/PasswordService" as pid 323.
2006-01-03 10:05:51 CET Started child "/usr/sbin/PrintServiceMonitor" as pid 324.
2006-01-03 10:05:51 CET Started child "/usr/libexec/postfix/master" as pid 325.
2006-01-03 10:05:51 CET Started child "/usr/sbin/hwmond" as pid 326.
2006-01-03 10:05:52 CET Started child "/usr/sbin/kadmind" as pid 327.
2006-01-03 10:05:52 CET Started child "/usr/sbin/krb5kdc" as pid 328.
2006-01-03 10:05:52 CET Automatic reboot timer enabled.
#Start-Date: 2006-01-03 10:55:03 CET
2006-01-03 10:55:03 CET Started child "/usr/sbin/PasswordService" as pid 318.
2006-01-03 10:55:03 CET Started child "/usr/sbin/PrintServiceMonitor" as pid 319.
2006-01-03 10:55:03 CET Started child "/usr/libexec/postfix/master" as pid 320.
2006-01-03 10:55:03 CET Started child "/usr/sbin/hwmond" as pid 321.
2006-01-03 10:55:03 CET Started child "/usr/sbin/kadmind" as pid 322.
2006-01-03 10:55:03 CET Started child "/usr/sbin/krb5kdc" as pid 323.
2006-01-03 10:55:04 CET Automatic reboot timer enabled.


Apparently, the watchdog only keeps an eye on a couple of services, mostly related to authentication. CommuniGate Pro uses a perl script to authenticate users against the Open Directory on the same machine, thru pam. Maybe that's what's causing it somehow, since it's basically the only non-standard software left on this machine; disabled Filemaker Server to rule that out.

So, we'll try moving to built-in authentication for CGP, and see if that stabilizes the server. Makes our planned move to a Linux box easier too...

Jonathan,

You didn't note what OS X Server version you were using.

Most of watchdog's functionality has been assumed by launchd with 10.4. So the rebooting in 10.4 is done by watchdogtimerd (see the man page for details).

The crux of the matter is that watchdog (or watchdogtimerd) must be "tickled" every 5 minutes by the PMU (Power Management Unit), or it will hard reboot the machine, assuming you have "automatically restart after power failure" turned on in your system preferences. Whatever is causing the issue will show up before the -93 error in the log. The logs you've posted come up after the first reboot, and then have a big gap between 10:05 and 10:55 CET, when the reboot happens. You could always disable the auto-reboot, and then come back and see if a service is dead, or if the machine is sitting there at a kernel panic. With the logs you've posted, there's no hint as to what specifically is going on.

Quoting one of my replies; "We run OD, SMB, AFP, Filemaker Server, CommuniGate Pro plus print and DNS services on a lightly loaded single CPU Xserve running Panther (10.3.9)." 🙂

We tried 10.4.2 and 10.4.3 on this machine, but we couldn't solve the 'samba locks the entire server' type of problems that several others have been experiencing as well. So we moved back to Panther.

Thanks for the 'auto reboot after power failure' tip. Hadn't thought of that one yet, will check what it's set to right now.