brucefromamherst

Q: Server 3.1 Issues

Two steps forward, one step back...

 

I updgraded OS X Server 3.0.3 to 3.1 yesterday afternoon, and two things happened that weren't happening before:

  1. Processor Usage (both System CPU and User CPU) jumped from under 5% on average (combined, over the last few months) to about 18-22%. I rebooted the machine this morning and it is now running anywhere from 40-60% CPU over the last hour. The dmrunnerd process is the biggest hog, taking up over 99% of the CPU according to Activity Monitor. There are also multiple instances of php-fpm, most under 6% but some jumping up to 75% or more. (This is on a quad-core 2.6GHz Mac Mini with 16GB RAM.)
  2. Logs Gone Wild. I woke up this morning to find that my drive had been almost completely filled. The biggest log file was a php log in /Library/Logs/ProfileManager, which reached well over 500GB by the time I got to it, followed by /Library/Logs/WebConfig.log at just under 2GB. I also have a PostgreSQL log in /Library/Logs/ProfileManager that is over 19GB at the moment, but it hasn't been modified since midnight.

 

It looks like the intense processor usage is related to the logging. Turning off all services, then re-enabling them, has not helped. Restarting the server hasn't fixed it either (the WebConfig.log file has stopped growing, but the php.log file is growing again (after I deleted the old one). I've got a chron job deleting it every 5 minutes at this point until I figure out a longer term solution.

 

Ideas, folks?

Posted on Mar 19, 2014 8:49 AM

Close

Q: Server 3.1 Issues

  • All replies
  • Helpful answers

Page 1 Next
  • by mscott_mdm,Helpful

    mscott_mdm mscott_mdm Mar 19, 2014 9:45 AM in response to brucefromamherst
    Level 2 (225 points)
    Mar 19, 2014 9:45 AM in response to brucefromamherst

    brucefromamherst,

     

    I think this should stop this problem for one month:

     

    sudo psql -U _devicemgr -d devicemgr_v2m0 -h /Library/Server/ProfileManager/Config/var/PostgreSQL -c "UPDATE devices SET last_update_info_time = dm_current_timestamp() + '1 month'"

     

    Copy the above and paste it into Terminal (or remote ssh shell) on the server as an admin user. You'll almost certainly be prompted to enter your admin password.

     

    Note that until a proper fix is released by Apple, it would appear that enrolling a number of new devices could lead to this problem recurring in less than a month, but you can always run the above command again.

     

    Please let me know if this seems to bring things "under control" for you. (It won't be immediate, but it should show signs of working within a few minutes.)

  • by brucefromamherst,

    brucefromamherst brucefromamherst Mar 19, 2014 10:14 AM in response to mscott_mdm
    Level 1 (0 points)
    Mar 19, 2014 10:14 AM in response to mscott_mdm

    I just came across your post of the same advice on the devicemgrd null ptr error loop causing high cpu load. thread and tried it. Worked great...my overall Processor Usage dropped immediately and is pretty steady around the 10% range. Thanks for the suggestion. Hopefully there will be a longer-term fix rather than something that needs to be reapplied every month.

     

    I really hope Apple fixes the various issues with the Server app soon...I'd like to get on with my real work.

  • by mscott_mdm,Helpful

    mscott_mdm mscott_mdm Mar 19, 2014 10:44 AM in response to mscott_mdm
    Level 2 (225 points)
    Mar 19, 2014 10:44 AM in response to mscott_mdm

    I would also recommend that you run this command, to keep newly-enrolled devices from causing the same problem after a couple of days:

     

    sudo psql -U _devicemgr -d devicemgr_v2m0 -h /Library/Server/ProfileManager/Config/var/PostgreSQL -c "ALTER TABLE devices ALTER COLUMN last_update_info_time SET DEFAULT dm_current_timestamp() + '1  month'"
  • by LibaryTechnologyDivision,

    LibaryTechnologyDivision LibaryTechnologyDivision Mar 19, 2014 3:53 PM in response to mscott_mdm
    Level 1 (0 points)
    Mar 19, 2014 3:53 PM in response to mscott_mdm

    Thank You so much for posting the Problem (AND THE SOLUTION)

     

    Had absolutely the same issue with the server after upgrade to Mavericks Server 3.1 .  Even before I have noticed that the drive filled with 180GB of log files (in multiple locations)   I had problems imaging a computer lab, the Mac would spontanously stop imaging and Quit.   - There were multiple postgres_real  tasks that would hog 3-4 cpu cores.  also PHP-FPM and Bzip processes would hog the server at 100% times.    this BZIP led me to find that the Logs are filling up ... under ProcessManager directory.        .  It was a havoc.  I have changed the above command from "1 month" to "1 week"   .... but in truth will be looking for official Apple solution soon!

     

    Thanks Again!!!!

  • by 4nNtt,

    4nNtt 4nNtt Mar 19, 2014 4:21 PM in response to brucefromamherst
    Level 1 (0 points)
    Mar 19, 2014 4:21 PM in response to brucefromamherst

    I ran in to the same Profile Manager issue.  I solved it temporarily by making the logs unwritable:

    sudo chmod 000 /Library/Logs/ProfileManager/php.log

     

    I see this exception thrown a lot in php.log:

    Caught error 'Undefined index: CommonName' of severity 8 at /Applications/Server.app/Contents/ServerRoot/usr/share/devicemgr/backend/php/ta sk.php:634

  • by Steven Major,

    Steven Major Steven Major Mar 20, 2014 7:17 AM in response to brucefromamherst
    Level 1 (44 points)
    Mar 20, 2014 7:17 AM in response to brucefromamherst

    Working with Apple on this issue, debugging might be on causing the Logs to swell.  Check the Profile Manager logs and see if debugging is indeed on, switch it off with:

     

    sudo defaults write /Library/Preferences/com.apple.ProfileManager debugOutput 0

     

    The other issue that isn't solved for us yet (2 servers) is that Profile Manager's database back up has gone hog wild doing backups every few minutes, swelling the backup directory to 6GB in about 18 hours (and counting) for us with no end in sight.

     

    Check that directory at /Library/Server/ProfileManager/Data/backups

     

    Please let me know if any of you see this backup issue on upgraded servers too (and a fix would be nice if you know one!)

     

    Apple is always very helpful when we call about these issues. I sometimes wonder if they test at all with upgrading a server rather than plopping it on a fresh machine and declaring it working. Most problems seem to happen with people trying to upgrade in a production enviroment rather than clean installs.

  • by Chris Marriott,

    Chris Marriott Chris Marriott Mar 20, 2014 8:01 AM in response to brucefromamherst
    Level 1 (80 points)
    Mar 20, 2014 8:01 AM in response to brucefromamherst

    Have the same issue here since upgrading to Server 3.1

     

    Apple, please fix your software that we all depend on.  There's no excuse for these kinds of bugs in seemly every release of your server app.

  • by Chris Marriott,

    Chris Marriott Chris Marriott Mar 20, 2014 8:05 AM in response to Chris Marriott
    Level 1 (80 points)
    Mar 20, 2014 8:05 AM in response to Chris Marriott

    Now after a reboot, Profile Manager won't start up and the Server App won't display any interface for managing the service.

     

    Nice work, Apple.

  • by mscott_mdm,

    mscott_mdm mscott_mdm Mar 20, 2014 8:11 AM in response to Chris Marriott
    Level 2 (225 points)
    Mar 20, 2014 8:11 AM in response to Chris Marriott

    Chris,

     

    Did you run the two commands I posted above? If so, do you have the output of those commands that you can paste here? (You can run them again and paste the output from the 2nd attempt, too.)

     

    Thanks.

  • by Chris Marriott,

    Chris Marriott Chris Marriott Mar 20, 2014 8:22 AM in response to mscott_mdm
    Level 1 (80 points)
    Mar 20, 2014 8:22 AM in response to mscott_mdm

    I ran the commands but I don't have output.  Since Profile Manager won't even start now, I'm screwed.  Like so many times before, I'm going to have to blow away the server and rebuilt it.  Same old, same old when it comes to Apple's 'server' software.

  • by jwzg2,

    jwzg2 jwzg2 Mar 20, 2014 2:10 PM in response to Chris Marriott
    Level 1 (0 points)
    Mar 20, 2014 2:10 PM in response to Chris Marriott

    Backups, backups, backups.  I know that doesn't help you now, but...

     

    Same issue here. php.log was growing at a rate of over 50 mb/second and I came in this morning to a full 1TB server.

     

    We're rolling back, since I've added no new devices/settings.  When Apple gets their fecal matter consolidated, we'll move forward.

  • by Patrick Fist,

    Patrick Fist Patrick Fist Mar 21, 2014 6:04 AM in response to jwzg2
    Level 1 (50 points)
    Mar 21, 2014 6:04 AM in response to jwzg2

    jwzg2 wrote:

     

    Backups, backups, backups.  I know that doesn't help you now, but...

     

    Same issue here. php.log was growing at a rate of over 50 mb/second and I came in this morning to a full 1TB server.

     

    We're rolling back, since I've added no new devices/settings.  When Apple gets their fecal matter consolidated, we'll move forward.

    Hello jwzg2,

     

    which steps did you make to recover to a Backup?

    Only restored the old /Applications/Server.app? The hole System? Or just the Database?

    I am missing exactly instructions how to do this and keeping the database and the "connected/enrolled" devices and policies.

     

    Thank you, ...

  • by jwzg2,

    jwzg2 jwzg2 Mar 21, 2014 6:16 AM in response to Patrick Fist
    Level 1 (0 points)
    Mar 21, 2014 6:16 AM in response to Patrick Fist

    https://discussions.apple.com///thread/4940719

    Check the last post for the link to restoring the ODM.

     

    If you aren't getting at least a Time Machine backup daily, and a weekly image, you are playing with fire.

  • by JKersten,

    JKersten JKersten Mar 21, 2014 7:35 AM in response to Patrick Fist
    Level 1 (0 points)
    Mar 21, 2014 7:35 AM in response to Patrick Fist

    I had the same issues as the OP after upgrading from 3.0.2 to 3.1. After having gotten burned upgrading to Mavericks/3.0 from Mountain Lion/2.2.2 I now make a practice of taking a full disk image backup of the server before doing an upgrade, using Carbon Copy Cloner.

     

    I boot from an external drive with CCC and dump the entire server HD (using block copy to a compressed disk image on the attached promise RAID). It does require taking the server offline, but the entire process takes  < 5 minutes (backing up ~35GB of data from an the server's boot drive - an SSD). I also have a script that does nightly database dumps of the wiki server, profile manager, OD, and a couple of custm apps I have running on the server. If my full image is a bit out of date, I can still restore and then bring it up to date by restoring wiki or PM databases as needed.

Page 1 Next