Looks like no one’s replied in a while. To start the conversation again, simply ask a new question.

Server 3.1 Issues

Two steps forward, one step back...


I updgraded OS X Server 3.0.3 to 3.1 yesterday afternoon, and two things happened that weren't happening before:

  1. Processor Usage (both System CPU and User CPU) jumped from under 5% on average (combined, over the last few months) to about 18-22%. I rebooted the machine this morning and it is now running anywhere from 40-60% CPU over the last hour. The dmrunnerd process is the biggest hog, taking up over 99% of the CPU according to Activity Monitor. There are also multiple instances of php-fpm, most under 6% but some jumping up to 75% or more. (This is on a quad-core 2.6GHz Mac Mini with 16GB RAM.)
  2. Logs Gone Wild. I woke up this morning to find that my drive had been almost completely filled. The biggest log file was a php log in /Library/Logs/ProfileManager, which reached well over 500GB by the time I got to it, followed by /Library/Logs/WebConfig.log at just under 2GB. I also have a PostgreSQL log in /Library/Logs/ProfileManager that is over 19GB at the moment, but it hasn't been modified since midnight.


It looks like the intense processor usage is related to the logging. Turning off all services, then re-enabling them, has not helped. Restarting the server hasn't fixed it either (the WebConfig.log file has stopped growing, but the php.log file is growing again (after I deleted the old one). I've got a chron job deleting it every 5 minutes at this point until I figure out a longer term solution.


Ideas, folks?

Posted on Mar 19, 2014 8:49 AM

Reply
Question marked as Best reply

Posted on Mar 19, 2014 9:45 AM

brucefromamherst,


I think this should stop this problem for one month:


sudo psql -U _devicemgr -d devicemgr_v2m0 -h /Library/Server/ProfileManager/Config/var/PostgreSQL -c "UPDATE devices SET last_update_info_time = dm_current_timestamp() + '1 month'"


Copy the above and paste it into Terminal (or remote ssh shell) on the server as an admin user. You'll almost certainly be prompted to enter your admin password.


Note that until a proper fix is released by Apple, it would appear that enrolling a number of new devices could lead to this problem recurring in less than a month, but you can always run the above command again.


Please let me know if this seems to bring things "under control" for you. (It won't be immediate, but it should show signs of working within a few minutes.)

29 replies
Question marked as Best reply

Mar 19, 2014 9:45 AM in response to brucefromamherst

brucefromamherst,


I think this should stop this problem for one month:


sudo psql -U _devicemgr -d devicemgr_v2m0 -h /Library/Server/ProfileManager/Config/var/PostgreSQL -c "UPDATE devices SET last_update_info_time = dm_current_timestamp() + '1 month'"


Copy the above and paste it into Terminal (or remote ssh shell) on the server as an admin user. You'll almost certainly be prompted to enter your admin password.


Note that until a proper fix is released by Apple, it would appear that enrolling a number of new devices could lead to this problem recurring in less than a month, but you can always run the above command again.


Please let me know if this seems to bring things "under control" for you. (It won't be immediate, but it should show signs of working within a few minutes.)

Mar 19, 2014 10:14 AM in response to mscott_mdm

I just came across your post of the same advice on the devicemgrd null ptr error loop causing high cpu load. thread and tried it. Worked great...my overall Processor Usage dropped immediately and is pretty steady around the 10% range. Thanks for the suggestion. Hopefully there will be a longer-term fix rather than something that needs to be reapplied every month.


I really hope Apple fixes the various issues with the Server app soon...I'd like to get on with my real work.

Mar 19, 2014 3:53 PM in response to mscott_mdm

Thank You so much for posting the Problem (AND THE SOLUTION)


Had absolutely the same issue with the server after upgrade to Mavericks Server 3.1 . Even before I have noticed that the drive filled with 180GB of log files (in multiple locations) I had problems imaging a computer lab, the Mac would spontanously stop imaging and Quit. - There were multiple postgres_real tasks that would hog 3-4 cpu cores. also PHP-FPM and Bzip processes would hog the server at 100% times. this BZIP led me to find that the Logs are filling up ... under ProcessManager directory. . It was a havoc. I have changed the above command from "1 month" to "1 week" .... but in truth will be looking for official Apple solution soon!


Thanks Again!!!!

Mar 19, 2014 4:21 PM in response to brucefromamherst

I ran in to the same Profile Manager issue. I solved it temporarily by making the logs unwritable:

sudo chmod 000 /Library/Logs/ProfileManager/php.log


I see this exception thrown a lot in php.log:

Caught error 'Undefined index: CommonName' of severity 8 at /Applications/Server.app/Contents/ServerRoot/usr/share/devicemgr/backend/php/ta sk.php:634

Mar 20, 2014 7:17 AM in response to brucefromamherst

Working with Apple on this issue, debugging might be on causing the Logs to swell. Check the Profile Manager logs and see if debugging is indeed on, switch it off with:


sudo defaults write /Library/Preferences/com.apple.ProfileManager debugOutput 0


The other issue that isn't solved for us yet (2 servers) is that Profile Manager's database back up has gone hog wild doing backups every few minutes, swelling the backup directory to 6GB in about 18 hours (and counting) for us with no end in sight.


Check that directory at /Library/Server/ProfileManager/Data/backups


Please let me know if any of you see this backup issue on upgraded servers too (and a fix would be nice if you know one!)


Apple is always very helpful when we call about these issues. I sometimes wonder if they test at all with upgrading a server rather than plopping it on a fresh machine and declaring it working. Most problems seem to happen with people trying to upgrade in a production enviroment rather than clean installs.

Mar 21, 2014 6:04 AM in response to jwzg2

jwzg2 wrote:


Backups, backups, backups. I know that doesn't help you now, but...


Same issue here. php.log was growing at a rate of over 50 mb/second and I came in this morning to a full 1TB server.


We're rolling back, since I've added no new devices/settings. When Apple gets their fecal matter consolidated, we'll move forward.

Hello jwzg2,


which steps did you make to recover to a Backup?

Only restored the old /Applications/Server.app? The hole System? Or just the Database?

I am missing exactly instructions how to do this and keeping the database and the "connected/enrolled" devices and policies.


Thank you, ...

Mar 21, 2014 7:35 AM in response to Patrick Fist

I had the same issues as the OP after upgrading from 3.0.2 to 3.1. After having gotten burned upgrading to Mavericks/3.0 from Mountain Lion/2.2.2 I now make a practice of taking a full disk image backup of the server before doing an upgrade, using Carbon Copy Cloner.


I boot from an external drive with CCC and dump the entire server HD (using block copy to a compressed disk image on the attached promise RAID). It does require taking the server offline, but the entire process takes < 5 minutes (backing up ~35GB of data from an the server's boot drive - an SSD). I also have a script that does nightly database dumps of the wiki server, profile manager, OD, and a couple of custm apps I have running on the server. If my full image is a bit out of date, I can still restore and then bring it up to date by restoring wiki or PM databases as needed.

Mar 21, 2014 3:22 PM in response to Chris Marriott

I just got off the phone with Apple Enterprise support about the Server 3.1 Profile Manager log issue (my server had a 455GB php.log this morning!)


From Apple Enterprise:


sudo psql -U _devicemgr -d devicemgr_v2m0 -h /Library/Server/ProfileManager/Config/var/PostgreSQL -c "UPDATE devices SET last_update_info_time = dm_current_timestamp() + '1 month'"


sudo psql -U _devicemgr -d devicemgr_v2m0 -h /Library/Server/ProfileManager/Config/var/PostgreSQL -c "ALTER TABLE devices ALTER COLUMN last_update_info_time SET DEFAULT dm_current_timestamp() + '1 month'"


A fix is in the works allegedly, the above commands take care of it for 1 month. I haven't compared specifically, but it looks like the same two commands posted by mscott_mdm. I didn't know it, but you can just highlight the command and drag it into a terminal window (thanks Apple support guy!). No copy/paste necessary! At least something positive came of this!


THANKS!

--Dan

Server 3.1 Issues

Welcome to Apple Support Community
A forum where Apple customers help each other with their products. Get started with your Apple ID.