slapd crashes when multiple simultaneous connections from same IP address

Question

Level 1

4 points

slapd crashes when multiple simultaneous connections from same IP address

Previously in Snow Leopard, when linux and/or other systems authenticated against OpenDirectory's openldap slapd service... things behaved fine.

In Moutain Lion OSX Server ... slapd will crash if multiple authentication request connections are received from the same client IP at the same time.

Client:

# csshX is a cluster ssh... it's the equivalant of opening 5 terminal windows and then typing "ssh myserver"

# "myserver" will authenticate agains the opendirectory master (slapd)

csshX myserver myserver myserver myserver myserver

# enter your password ...

# at best 1 of 5 terminals connect ... the rest reprompt for the password as they were not able to complete the connection

Server slapd Crash:

Sep 5 12:36:52 server.company.com ReportCrash[73128]: LaunchServices/5123589: Unable to lookup coreservices session port for session 0x186a0 uid=0 euid=0

Sep 5 12:36:52 server.company.com ReportCrash[73128]: failed looking up LS service ( scCreateSystemService returned MACH_PORT_NULL, called from SetupCoreApplicationServicesCommunicationPort, so using client-side NULL calls.

Sep 5 14:22:49 server.company.com slapd[74709]: @(#) $OpenLDAP: slapd 2.4.28 (Jun 20 2012 15:31:09) $ root@b1032.apple.com:/private/var/tmp/OpenLDAP/OpenLDAP-208~28/servers/slapd Sep 5 14:22:49 server.company.com slapd[74709]: slap_add_listener: opened additional listener 'ldaps:///' Sep 5 14:22:49 server.company.com slapd[74709]: bdb_db_open: database "dc=server,dc=company,dc=com": unclean shutdown detected; attempting recovery. Sep 5 14:22:49 server.company.com slapd[74709]: bdb_monitor_db_open: monitoring disabled; configure monitor database to enable Sep 5 14:22:49 server.company.com slapd[74709]: bdb_db_open: database "cn=authdata": unclean shutdown detected; attempting recovery. Sep 5 14:22:50 server.company.com slapd[74709]: slapd starting Sep 5 14:22:50 server.company.com slapd[74709]: daemon: posting com.apple.slapd.startup notification

# the connections in lsof -i drop ... and the PID is cycled out as slapd is restarted

The same procedure authenticating against Snow Leopard Server was fine.

We believe there is a bug in this 2.4.28 slapd build shipped with mountain lion surrounding concurrency and mutex locks???

The problem does not appear to exist if you open connections to 5 different servers with 5 different IP addresses.

Performing multiple ldapsearch requests at the exact same time is fine so long as you do not bind / authenticate to the openldap server. It appears binding is related.

Additional Info:

-- Changelog for Apple openldap: http://www.opensource.apple.com/source/OpenLDAP/OpenLDAP-208/OpenLDAP/CHANGES

-- Suspiciously similar bug that RedHat openldap used to have (probably unrelated)

---- "Previously, multiple concurrent connections to an OpenLDAP server could cause the slapd service to terminate unexpectedly with an assertion error. This update adds mutexes to protect multiple threads from accessing a structure with a connection, and the slapd service no longer crashes. (BZ#677611)"

---- http://www.redhat.com/archives/enterprise-watch-list/2011-March/msg00022.html

---- this could be entirely unrelated

-- http://www.openldap.org/software/release/changes.htm

---- Fixed slapd crash when attrsOnly is true (ITS#7143) ???

---- the above bug was patched after the release that ships with mountain lion ... however... I have not been able to mess with this yet:

---- ./slapd.d/cn=config/cn=schema.ldif:olcAttributeTypes: ( 1.3.6.1.4.1.4203.666.11.5.1.23 NAME 'reqAttrsOnly' DESC '

---- ./slapd.d/cn=config/cn=schema.ldif: ses $ reqAttrsonly ) MAY ( reqFilter $ reqAttr $ reqEntries $ reqSizeLimit $

---- the above may be a factor

-- We've adjusted the following slapd config variables in /etc/openldap/slapd_macosxserver.conf with no success

---- conn_max_pending 2048

---- conn_max_pending_auth 4096

---- concurrency 1000

---- threads 100

Posted on Sep 5, 2012 3:46 PM

Reply

Answer 1

dpilone

Level 1

0 points

Sep 7, 2012 2:26 PM in response to ionepoch

I'm seeing a similar problem on our Mini running 10.8.1. We have a TeamCity server with 8 builds setup in it. They're configured to check the repository once every minute, which results in 8 more or less simultaneous ssh authentications against our repository server which is bound to our Mini's OpenDirectory. Like clockwork I see a slapd crash in the logs. Right now I'm working around it by configuring each build to be off from the others as I haven't found any way of keeping slapd alive with multiple simultaneous authentication requests.

Reply

Answer 2

ionepoch Author

Level 1

4 points

Sep 7, 2012 8:45 PM in response to dpilone

Thank you for confirming this. It's helpful to start ruling out something on our end.

I honestly don't think we have anything configured wrong ... we have multiple OS X server's we have test this on... one is a vanilla basic open directory server with 3 users... the other is under larger load. They both behave the same.

My gut says it is due to being a few versions behind on an Apple build of slapd .. as compared to the official open source project. We don't have time at the moment to debug this... but if I were to dive into this... I would try and remove the AttrsOnly fields in the following statements:

-- http://www.openldap.org/software/release/changes.htm

---- Fixed slapd crash when attrsOnly is true (ITS#7143) ???

---- the above bug was patched after the release that ships with mountain lion ... however... I have not been able to mess with this yet:

---- ./slapd.d/cn=config/cn=schema.ldif:olcAttributeTypes: ( 1.3.6.1.4.1.4203.666.11.5.1.23 NAME 'reqAttrsOnly' DESC '

---- ./slapd.d/cn=config/cn=schema.ldif: ses $ reqAttrsonly ) MAY ( reqFilter $ reqAttr $ reqEntries $ reqSizeLimit $

---- the above may be a factor

(that's the same info in my original post... just wanted to highlight it).

Reply

Answer 3

senatore

Level 1

0 points

Oct 2, 2012 10:19 AM in response to ionepoch

Hello,

I believe we are experiencing the same issue on our newly built 10.8.2 directory server. We are experiencing intermittent authentication failures, which seem to be directly related to slapd crashing consistently. We have been running this way for about 1 month now, and it is not unusual for slapd to crash over 100 times in a single day. It could function normally for an hour without crashing, but then crash 5-10 times over a few minutes time. I have not noticed multiple connectons from the same IP specifically when the crash occurs, but I would not doubt it. Overall, authentication works, but it is a very serious issue that we need to resolve.

Have you made any progress in rectifying this issue? I have disabled spotlight, due to the logs referencing spotlight prior to almost every crash, however, that had no obvious effect.

Thanks in advance for any information you can provide.

This is the system.log around crash time consistenly:

Oct 2 08:55:36 servername ReportCrash[23600]: DebugSymbols was unable to start a spotlight query: spotlight is not responding or disabled.

Oct 2 08:55:36 servername ReportCrash[23600]: failed looking up LS service ( scCreateSystemService returned MACH_PORT_NULL, called from SetupCoreApplicationServicesCommunicationPort, so using client-side NULL calls.