brucefromamherst

Q: Server 3.1 Issues

Two steps forward, one step back...

 

I updgraded OS X Server 3.0.3 to 3.1 yesterday afternoon, and two things happened that weren't happening before:

  1. Processor Usage (both System CPU and User CPU) jumped from under 5% on average (combined, over the last few months) to about 18-22%. I rebooted the machine this morning and it is now running anywhere from 40-60% CPU over the last hour. The dmrunnerd process is the biggest hog, taking up over 99% of the CPU according to Activity Monitor. There are also multiple instances of php-fpm, most under 6% but some jumping up to 75% or more. (This is on a quad-core 2.6GHz Mac Mini with 16GB RAM.)
  2. Logs Gone Wild. I woke up this morning to find that my drive had been almost completely filled. The biggest log file was a php log in /Library/Logs/ProfileManager, which reached well over 500GB by the time I got to it, followed by /Library/Logs/WebConfig.log at just under 2GB. I also have a PostgreSQL log in /Library/Logs/ProfileManager that is over 19GB at the moment, but it hasn't been modified since midnight.

 

It looks like the intense processor usage is related to the logging. Turning off all services, then re-enabling them, has not helped. Restarting the server hasn't fixed it either (the WebConfig.log file has stopped growing, but the php.log file is growing again (after I deleted the old one). I've got a chron job deleting it every 5 minutes at this point until I figure out a longer term solution.

 

Ideas, folks?

Posted on Mar 19, 2014 8:49 AM

Close

Q: Server 3.1 Issues

  • All replies
  • Helpful answers

Previous Page 2
  • by DNCUSD-IT,

    DNCUSD-IT DNCUSD-IT Mar 21, 2014 3:22 PM in response to Chris Marriott
    Level 1 (10 points)
    Mar 21, 2014 3:22 PM in response to Chris Marriott

    I just got off the phone with Apple Enterprise support about the Server 3.1 Profile Manager log issue (my server had a 455GB php.log this morning!)

     

    From Apple Enterprise:

     

    sudo psql -U _devicemgr -d devicemgr_v2m0 -h /Library/Server/ProfileManager/Config/var/PostgreSQL -c "UPDATE devices SET last_update_info_time = dm_current_timestamp() + '1 month'"

     

    sudo psql -U _devicemgr -d devicemgr_v2m0 -h /Library/Server/ProfileManager/Config/var/PostgreSQL -c "ALTER TABLE devices ALTER COLUMN last_update_info_time SET DEFAULT dm_current_timestamp() + '1 month'"

     

    A fix is in the works allegedly, the above commands take care of it for 1 month.  I haven't compared specifically, but it looks like the same two commands posted by mscott_mdm.  I didn't know it, but you can just highlight the command and drag it into a terminal window (thanks Apple support guy!).  No copy/paste necessary!  At least something positive came of this!

     

    THANKS!

    --Dan

  • by ceotjoe,

    ceotjoe ceotjoe Mar 22, 2014 9:43 AM in response to brucefromamherst
    Level 1 (0 points)
    Mar 22, 2014 9:43 AM in response to brucefromamherst

    It's not just regarding the logs. In my case Profile Manager sent pushs like **** to all devices. I noticed it, because my iPhone drained a lot of power in only a few hours. Examining the logs of the iPhone showed mdmd getting push notifications all the time. This caused installd (it had a couple of hours CPU time, which seemed very unsusual for me comparing to the other processes) running also consuming a lot of CPU and in the end draining the battery.

     

    Applying the two SQL statements mentioned in this thread also fixed the exessive pushing in my case.

     

    Thanks

    Jörg

  • by R-M-B,

    R-M-B R-M-B Mar 24, 2014 11:24 AM in response to ceotjoe
    Level 1 (14 points)
    Mar 24, 2014 11:24 AM in response to ceotjoe

    Mac OS X Server 3.1.1 is available now, hopefully with embedded fix for this issue!

  • by ziondotcom,

    ziondotcom ziondotcom Mar 24, 2014 3:39 PM in response to R-M-B
    Level 1 (10 points)
    Mar 24, 2014 3:39 PM in response to R-M-B

    From Apple Enterprise Support:

    "It appears though that the log files and CPU usage from profile manager should be resolved in 3.1.1, which was released today."

  • by Steven Major,

    Steven Major Steven Major Mar 24, 2014 3:53 PM in response to ziondotcom
    Level 1 (44 points)
    Mar 24, 2014 3:53 PM in response to ziondotcom

    Has anyone tried it yet?

     

    I no longer have faith that Apple's QA team (assuming there still is one) for Server can get a release out the door without some castrophy happening to my systems.

  • by Steven Major,

    Steven Major Steven Major Mar 24, 2014 5:38 PM in response to Steven Major
    Level 1 (44 points)
    Mar 24, 2014 5:38 PM in response to Steven Major

    I should quit my computing gig and tell fortunes. I can see the future. I cloned one of our servers running 3.1, and tried the 3.1.1 "fix".

     

    It's been stuck on "Upgrading Services" for over an hour.

     

    I'm not sure whether to laugh at the absurdity of it all or just shake my head in disbelief that a billion dollar company just can't make their own software work properly.

  • by jwzg2,

    jwzg2 jwzg2 Mar 24, 2014 5:49 PM in response to Steven Major
    Level 1 (0 points)
    Mar 24, 2014 5:49 PM in response to Steven Major

    Steven, ours actually updated quickly.  Now I'm bug hunting. 

     

    By the way, how many users do you have running on your server?  Over 425 here.

    Steven Major wrote:

     

    I should quit my computing gig and tell fortunes. I can see the future. I cloned one of our servers running 3.1, and tried the 3.1.1 "fix".

     

    It's been stuck on "Upgrading Services" for over an hour.

     

    I'm not sure whether to laugh at the absurdity of it all or just shake my head in disbelief that a billion dollar company just can't make their own software work properly.

  • by Steven Major,

    Steven Major Steven Major Mar 24, 2014 6:09 PM in response to jwzg2
    Level 1 (44 points)
    Mar 24, 2014 6:09 PM in response to jwzg2

    This isn't even one of our production servers, this is more or less a test server where we do some proof of concept testing.  It's got less than 12 enrolled devices and 6 users on it.  It's pretty clean too.

     

    Production server has about 265 devices.

     

    We're still ramping up too, having 10.6 and Workgroup Manager running on 500 Macs.

     

    Kicking the tires on Profile Manager since Lion hasn't instilled much confidence in us and I'm wary of bring our fleet of 900 devices onboard.

     

    We're just shy of paranoid about backups here, but sooner or later some corruption will creep in along the way that you don't notice dispite best practices.  We know from history with Apple more often than not the answer is "just wipe the server" or "just reset Profile Manager".  That answer becomes less and less acceptable with 100's of devices to re-populate and tie-ins with the Enterprise App Store - which we'd like to do.

     

    Server needs a simple, built-in way to verify it's integetry and backup data somewhere else (not Time Machine).

     

    Yes, some of this is possible via the command line, and I love diving into the shell as much as the next guy, but if I REALLY wanted to do that, I'd manage the whole shebang with Linux.

  • by GreybeardCT,

    GreybeardCT GreybeardCT Mar 25, 2014 4:53 PM in response to brucefromamherst
    Level 1 (5 points)
    Mar 25, 2014 4:53 PM in response to brucefromamherst

    On the issue of the extremely large WebConfig.log and the large php.log I have a suggestion that's worked for me so far.

     

    The newsyslog is run hourly at *:30 (at least on my system) and it can be used to rotate and trim the logs.  I have added the following file: /private/etc/newsyslog.d/webconfig.conf

     

    # logfilename          [owner:group]    mode count size when  flags [/pid_file] [sig_num]
    /Library/Logs/WebConfig.log                 644  7     *    $W1D4   J
    /Library/Logs/WikiConfig.log                 644  7     *    $W1D4   J
    

     

    This will rotate the WebConfig.log weekly and compress it. YMMV and you may want to do that more/less often or keep fewer than 7 copies.

     

    See man newsyslog and man newsyslog.conf for the specifics.

  • by hc999,

    hc999 hc999 Mar 25, 2014 5:30 PM in response to brucefromamherst
    Level 1 (10 points)
    Mar 25, 2014 5:30 PM in response to brucefromamherst

    Here's my rule of thumb when it comes to Server.app and OS X upgrades: wait at least 1 to 2 weeks before upgrading. This is definitely the case with a new major release of the OS (I still haven't upgraded all of my servers to Mavericks from ML - there is no rush as long as the security updates keep coming).

  • by Patrick Fist,

    Patrick Fist Patrick Fist Mar 26, 2014 2:08 AM in response to Steven Major
    Level 1 (50 points)
    Mar 26, 2014 2:08 AM in response to Steven Major

    Steven Major wrote:

     

    Has anyone tried it yet?

     

    I no longer have faith that Apple's QA team (assuming there still is one) for Server can get a release out the door without some castrophy happening to my systems.

     

    Hey Steven,

     

    I just installed 3.1.1 and did not solved my increasing Log issue. But it looks like that the server isnt sending non-stop Push messages now. Little issue solved but still on high cpu load of opdendirectoryd and devicemgrd.

  • by netzen7,

    netzen7 netzen7 Mar 26, 2014 3:53 PM in response to brucefromamherst
    Level 1 (0 points)
    Mar 26, 2014 3:53 PM in response to brucefromamherst

    3.1 and 3.1.1 are junk for me.  I've spent 3 days trying to stabilize - including a failed server build from scratch.  Errors and inconsistencies throughout, performance problems, run away CPU and disk utilization, entire websites missing, 3rd party certificates that won't apply.

     

    This isn't OK. Maybe for a "consumer" company, but I'm running this in an enterprise where thousands of users depend on working software.  Thankfully this is only messing with about 120 devices, but that is 120 too many.  I've reverted everything back to 3.0.2...and holding.

  • by Steven Major,

    Steven Major Steven Major Mar 26, 2014 4:03 PM in response to netzen7
    Level 1 (44 points)
    Mar 26, 2014 4:03 PM in response to netzen7

    For the large Profile Manager logs, did anyone have luck with this? It worked for me:

     

    sudo defaults write /Library/Preferences/com.apple.ProfileManager debugOutput 0

     

    debug for me, after the update, was set to 1 and very verbose.

     

    Call Apple if you haven't already. Get a case number. Get esclated. They need to know the problems and every call and case are logged. Hopefully, sooner or later, someone will realize that something is amiss when each update breaks things.

     

    Should we have to do this at the enterprise level? No.

  • by jmelashenko,

    jmelashenko jmelashenko Apr 4, 2014 10:26 AM in response to mscott_mdm
    Level 1 (0 points)
    Apr 4, 2014 10:26 AM in response to mscott_mdm

    Thanks mscott_mdm this has helped me a great deal this morning.

  • by mscott_mdm,

    mscott_mdm mscott_mdm Apr 21, 2015 12:52 PM in response to mscott_mdm
    Level 2 (225 points)
    Apr 21, 2015 12:52 PM in response to mscott_mdm

    Please note, if you applied the "ALTER TABLE" workaround I mentioned in this thread, you will need to apply the workaround I posted in Server 4.1 Profile Manager Migration Failed before you can successfully upgrade to Server 4.1. I would recommend doing this before you install Server 4.1 if you haven't already.

     

    sudo psql -U _devicemgr -d devicemgr_v2m0 -h /Library/Server/ProfileManager/Config/var/PostgreSQL -c "ALTER TABLE mdm_targets ALTER COLUMN last_update_info_time SET DEFAULT 'epoch'"

Previous Page 2