Memory Leak and 100% CPU in slapd process when using Open Directory OS X 10.8.5

Question

Level 1

0 points

Memory Leak and 100% CPU in slapd process when using Open Directory OS X 10.8.5

Has anyone seen the slapd (LDAP daemon from OpenLDAP which OS X Server uses) run away when running Open Directory? We've tried disabling replication but it doesn't seem to matter. The slapd process will start up and take up about 400MB and then at night will consume all the available memory and crashing the server.

Any thoughts? Any benefit in upgrading to Mavericks' Server app?

Thanks,

John

OS X Server-OTHER, OS X Mountain Lion (10.8.5)

Posted on Dec 6, 2013 10:24 AM

Reply

Answer 1

Dec 6, 2013 12:04 PM in response to teampayoff

Have you tried enabling Open Directory debug logging?

sudo odutil set log debug

Have you tried a db_recover on the LDAP databases?

Before you do anything, make sure you have a valid backup of the server or at the very least, the OD database. If you are experiencing a problem, upgrading to Mavericks will likely not solve the problem. It will likely make it worse. Try to resolve the issue that you have first.

Reply

Answer 2

teampayoff Author

Level 1

0 points

Dec 6, 2013 2:14 PM in response to Strontium90

@Strontium90

Found this in opendirectory.log which seems to just show ldap requests starting to slow down.

2013-12-06 09:54:19.736670 PST - State information (some requests have been active for extended period):

Sessions: {

2839 -- opendirectoryd:

- {

Session ID: CBAE3152-1A3B-4C7E-89D6-2F836C28F5BA

Refs: singleton

Type: Default

Target: localhost

}

}

Nodes: {

2839 -- opendirectoryd:

- {

Node ID: CC6FC6C3-AD02-4068-9E8F-8FDC228F4C48

Nodename: /Search

Session ID: <Default>

Refs: 3

Internal Use: X

}

2816 -- automount:

- {

Node ID: 8FC03A24-46F2-4DAA-A0EC-3FF98B458FB4

Nodename: /Search

Session ID: <Default>

Refs: 2

}

277 -- SystemUIServer:

- {

Node ID: FFF97865-2FBE-474A-BD53-43A35DA09D3A

Nodename: /Search

Session ID: <Default>

Refs: 3

Internal Use: X

}

}

Requests: {

2816 -- automount:

- {

Nodename: /Search

Refs: 5

Active Time: 65170959

Type: ODNodeCopySubnodeNames

Current Module: search

Node ID: 8FC03A24-46F2-4DAA-A0EC-3FF98B458FB4

Request ID: 5

}

2839 -- opendirectoryd:

- {

Nodename: /Search

Refs: 5

Active Time: 65971987

Type: ODQueryCreateWithNode

Current Module: search

Node ID: CC6FC6C3-AD02-4068-9E8F-8FDC228F4C48

Request ID: 4

}

277 -- SystemUIServer:

- {

Current Module: SystemCache

Type: getpwuid

Request ID: 10

Refs: 4

Active Time: 13397955

}

- {

Nodename: /Search

Parent Request: 10

Refs: 5

Active Time: 11041999

Type: ODQueryCreateWithNode

Current Module: search

Request ID: 12

Node ID: FFF97865-2FBE-474A-BD53-43A35DA09D3A

}

68 -- coreservicesd:

- {

Current Module: SystemCache

Type: getpwuid

Request ID: 8

Refs: 3

Active Time: 13484131

}

2832 -- sshd:

- {

Current Module: SystemCache

Type: getpwnam

Request ID: 9

Refs: 3

Active Time: 13483916

}

640 -- SystemUIServer:

- {

Current Module: SystemCache

Type: getpwuid

Request ID: 7

Refs: 3

Active Time: 13522530

}

}

2013-12-06 09:54:19.737704 PST - requesting spindump generation

In slapd.log around the same time:

Dec 6 06:55:23 dms-out-01.int.payoff.com slapd[118]: conn=-1 op=0: attribute "entryCSN" index delete failure

Dec 6 06:58:25 dms-out-01.int.payoff.com slapd[118]: connection_read(46): no connection!

Dec 6 07:00:23 dms-out-01.int.payoff.com slapd[118]: connection_read(48): no connection!

Dec 6 07:27:34 dms-out-01.int.payoff.com slapd[118]: connection_read(47): no connection!

Dec 6 07:56:30 dms-out-01.int.payoff.com slapd[118]: connection_read(47): no connection!

Dec 6 08:10:24 dms-out-01.int.payoff.com slapd[118]: => bdb_idl_delete_key: c_get failed: DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock (-30994)

Dec 6 08:10:24 dms-out-01.int.payoff.com slapd[118]: conn=-1 op=0: attribute "entryCSN" index delete failure

Dec 6 08:10:25 dms-out-01.int.payoff.com slapd[118]: => bdb_idl_delete_key: c_get failed: DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock (-30994)

Dec 6 08:10:25 dms-out-01.int.payoff.com slapd[118]: conn=-1 op=0: attribute "entryCSN" index delete failure

Dec 6 08:18:29 dms-out-01.int.payoff.com slapd[118]: connection_input: conn=12372 deferring operation: binding

Dec 6 08:18:55 dms-out-01.int.payoff.com slapd[118]: connection_input: conn=12373 deferring operation: binding

Dec 6 08:19:50 dms-out-01.int.payoff.com slapd[118]: connection_read(53): no connection!

Dec 6 08:27:16 dms-out-01.int.payoff.com slapd[118]: => bdb_idl_delete_key: c_get failed: DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock (-30994)

Dec 6 08:27:16 dms-out-01.int.payoff.com slapd[118]: conn=-1 op=0: attribute "entryCSN" index delete failure

Dec 6 09:08:00 dms-out-01.int.payoff.com slapd[118]: connection_input: conn=12871 deferring operation: binding

Dec 6 09:16:10 dms-out-01.int.payoff.com slapd[118]: connection_input: conn=12871 deferring operation: pending operations

The deadlock from what I can tell is a more of a warning from OpenLDAP, but it is a common pattern and occurring every few minutes.

Not much useful in slapconfig.log.

Any insights? Thanks so much.

~john

Reply

Answer 3

Mar 10, 2014 1:05 PM in response to teampayoff

I am having this exact same problem and have not been able to find a solution.

Reply

Answer 4

teampayoff Author

Level 1

0 points

Mar 10, 2014 1:34 PM in response to St Margaret's College

St. Margaret's,

We ended up upgrading to OS X Mavericks Server.app, and that greatly stabilized our OD server. This might be a good solution for you if you don't utilize OS X Server's other broken services. However, we have since moved to Windows Active Directory. Now we're rock solid. Good luck.

~Team Payoff

Reply

Answer 5

twillsie

Level 1

4 points

Sep 25, 2014 10:04 AM in response to teampayoff

Our server has this problem as well (on Mountain Lion). We aren't ready to upgrade yet so we have just done the following to restart SLAPD:

$ sudo pkill slapd

$ sudo /usr/libexec/slapd -d 255

our server has 16 gigs of ram, so if slapd does start to memory leak it will take a good day before the server grinds to a halt, so we just run a CRON script to restart slapd every night at 1:00am.

With this specific problem, when your server's memory is maxed, it's really difficult to remote login using VNC/Apple Remote Desktop, so make sure your server has got SSH enabled (so you can terminal in). And you will be able to fix this problem without having to actually reboot the whole server.

We are still trying to figure out what the actual problem is, but at least now we can figure it out without the pressure of everyone's account not logging in. For us the memory leak is not always persistent, some times it would go a whole week before maxing the ram, some days it would max the ram before the next morning.

Reply

Answer 6

Oct 6, 2014 12:18 AM in response to twillsie

Hello twillsie

We have exactly the same problem since upgrading to Mavericks, how do you script the CRON job ?

Is it possible to do without putting any password into the script ?

SLAP needs root privileges can you post here how you do it please !

Reply

Answer 7

twillsie

Level 1

4 points

Nov 13, 2014 3:06 PM in response to Rolf Kocherhans

Hi Rolf, I suppose I should have been more accurate. I scripted a launchd plist and bash shell script. So here goes.

I was going to origingally tell you to just run your CRON job as sudo, but seeing as CRON is deprecated, I figured I'd give you the following properly apple way to create a scheduled task script that runs as root on system launch and run every night at 1:00am (system time)

I'll do this step by step.

1. Use Terminal and navigate to your LaucnhDaemons folder

cd /Library/LaunchDaemons

2. Create your launchd script (try and use apple naming conventions to avoid file conflicts)

sudo touch com.yourname.restartslapd.plist

3. Edit the file as root (you can use vi, or vim, I prefer to use nano)

sudo nano com.yourname.restartslapd.plist

4. Enter in your launchd configurations (make sure your Label matches your filename).

<?xml version="1.0" encoding="UTF-8"?>

<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">

<plist version="1.0">

<dict>

<key>Label</key>

<string>com.yourname.restartslapd</string>

<key>ProgramArguments</key>

<array>

<string>/Users/Shared/scripts/killstartslapd.sh</string>

</array>

<key>Nice</key>

<integer>1</integer>

<key>StartCalendarInterval</key>

<dict>

<key>Hour</key>

<integer>1</integer>

</dict>

<key>RunAtLoad</key>

<true/>

<key>StandardErrorPath</key>

<string>/tmp/restartslapd.err</string>

<key>StandardOutPath</key>

<string>/tmp/restartslapd.out</string>

</dict>

</plist>

5. Create your script (be sure to use sudo when you create it so that root can run it)

/Users/Shared/scripts/killstartslapd.sh

#!/bin/bash

date >> /tmp/restartslapd.out

echo "restarting slapd..."

pkill slapd

sleep 5

/usr/libexec/slapd -d 255

echo "completed!"

6. Add your plist into launchctl as sudo.

sudo launchctl load com.yourname.restartslapd.plist

7. You can check to see if your daemon is running with this command.

sudo launchctl list

Also back in the CRON days (all of just a year or two ago lol). You simply just used "sudo cron -e" and removed sudo from your scripts (because root is already evoked by cron before your scripts ran anyways).

I hope this helps you out, and sorry about the late reply. I didn't want to give you a solution using a deprecated process on your mac.

Reply

Answer 8

twillsie

Level 1

4 points

Nov 14, 2014 10:19 AM in response to twillsie

There is one problem with my launchd script above.

In the launchd .plist, this key...

<key>StartCalendarInterval</key>

<dict>

<key>Hour</key>

<integer>1</integer>

</dict>

Should be this...

<key>StartCalendarInterval</key>

<dict>

<key>Hour</key>

<integer>1</integer>

<key>Minute</key>

<integer>0</integer>

</dict>

If you don't set the minute, launchd will run your script every minute for the hour. I figured I'd just make that change on here.

Reply

Answer 9

shryocdj

Level 1

30 points

Dec 26, 2014 10:06 AM in response to twillsie

Hey twillsie, any reason you didn't just issue a "serveradmin restart dirserv"?

Nice solution, btw.

Reply