OD intermittently fails to authenticate LDAP users
Hi there! I have a strange issue with Open Directory and I'm hoping someone might have some insight on it. Here's some background info to start:
- We have 2 Mac minis (one OD master, one OD replica) in our environment that, among other things, use Open Directory to provide an LDAP authentication backend for a few of our on-premises applications (JIRA, Confluence, OpenVPN, etc). We have about 100 users or so.
- I created a bind user called ldap_search to enable the aforementioned 3rd party applications to use OD LDAP.
- Both Mac minis are running OS X Server 5.015 on OS X 10.11.3.
- The on-premises applications are authenticating to the Open Directory master using LDAP over SSL. The SSL cert was obtained from a 3rd party trusted root CA.
Everything was mostly working great until recently. About 3 weeks ago, a few users notified me that they weren't able to log into any applications using their LDAP credentials. I tested this myself and verified that I was unable to log in. I checked the OD master and verified that slapd was running. The server was up and accessible through Apple Remote Desktop and ssh. I checked opendirectoryd logs, password service error and server logs, slapd config logs and finally the slapd logs themselves. The only clues I found in all of these logs were about 100+ errors in the slapd log related to the ldap_search bind user:
Mar 21 12:14:55 myserver slapd[205]: odusers_krb_auth: Error obtaining credentials for ldap_search: -1765328228
Mar 21 12:15:28 myserver slapd[205]: odusers_krb_auth: Error obtaining credentials for ldap_search: -1765328228
Mar 21 12:16:00 myserver slapd[205]: odusers_krb_auth: Error obtaining credentials for ldap_search: -1765328228
Mar 21 12:17:02 myserver slapd[205]: odusers_krb_auth: Error obtaining credentials for ldap_search: -1765328228
Mar 21 12:17:33 myserver slapd[205]: odusers_krb_auth: Error obtaining credentials for ldap_search: -1765328228
I ended up rebooting the server and things went back to normal fairly quickly. Fast forward to yesterday. The same thing happened for about 5 minutes. No one could log into any of the applications using their OD LDAP credentials. This time, the issue disappeared on its own with no intervention on my part. And then again, it happened today for about 30 minutes. Another admin rebooted the server and the issue was resolved. Again, the only clue to the issue was in the slapd logs:
Apr 13 09:56:21 myserver slapd[221]: odusers_krb_auth: Error obtaining credentials for ldap_search: -1765328228
Apr 13 09:58:37 myserver slapd[221]: odusers_krb_auth: Error obtaining credentials for ldap_search: -1765328228
Apr 13 09:59:19 myserver slapd[221]: odusers_krb_auth: Error obtaining credentials for ldap_search: -1765328228
Apr 13 10:00:06 myserver slapd[221]: odusers_krb_auth: Error obtaining credentials for ldap_search: -1765328228
Apr 13 10:03:00 myserver slapd[221]: odusers_krb_auth: Error obtaining credentials for ldap_search: -1765328228
I've checked the error logs on the "clients" (i.e. the application servers for JIRA, Confluence, etc, that are hooked into LDAP) and the failed logins show "LDAP: error code 50 - Insufficient Access Rights." I think this is just a standard "password was incorrect" error from LDAP.
I'd really like to avoid having to reboot the server all the time, as it's quite disruptive. I've scoured Google and these forums for clues as to what could be causing this, but I haven't found anything yet. Has anyone else run into anything like this? Is this a known bug? I've been holding off on upgrading to OS X 10.11.4/Server 5.1, but perhaps that will fix this?
Thanks!
Mac mini, OS X El Capitan (10.11.3), null