Password service won't start

Question

Level 1

4 points

Password service won't start

Following an unplanned hard restart of our server (Mac OS X 10.9.5, Server 3.2.2), the password service won't start, so our staff can't log into mail, calendars or file sharing.

When I launch Open Directory, the main window has the message "Unable to load replica list", then it switches itself off after a minute or so.

I've looked at the certificates in Server and they have the green tick so presumably are OK.

DNS is working OK and running the command "sudo changeip -checkhostname" reports success.

Here's the section of log that repeats every few seconds as PasswordService repeatedly starts up and exits:

13/03/2015 20:55:00.617 com.apple.launchd[1]: (com.apple.PasswordService[5890]) Exited with code: 1

13/03/2015 20:55:00.617 com.apple.launchd[1]: (com.apple.PasswordService) Throttling respawn: Will start in 10 seconds

13/03/2015 20:55:02.540 xscertd[249]: Failed sending LookupCRLByCARecordName command to com.apple.xscertd.helper: The operation couldn’t be completed. (com.apple.certificateserver error 42005.)

13/03/2015 20:55:08.708 com.apple.launchd[1]: (org.openldap.slapd[5894]) Exited with code: 1

13/03/2015 20:55:08.708 com.apple.launchd[1]: (org.openldap.slapd) Throttling respawn: Will start in 7 seconds

13/03/2015 20:55:10.206 xscertd-helper[5897]: ldap_search_ext_s returned -1 - Can't contact LDAP server when searching for bdb suffix, exiting

13/03/2015 20:55:10.207 com.apple.launchd[1]: (com.apple.xscertd-helper[5897]) Exited with code: 1

13/03/2015 20:55:10.207 com.apple.launchd[1]: (com.apple.xscertd-helper) Throttling respawn: Will start in 10 seconds

13/03/2015 20:55:10.639 PasswordService[5901]: -[PasswordServerPrefsObject getSearchBase]: Unable to locate search base: -1 Can't contact LDAP server

13/03/2015 20:55:10.639 PasswordService[5901]: -[PasswordServerPrefsObject loadXMLData]: Unable to locate passwordserver config record's plist attribute: -1 Can't contact LDAP server

13/03/2015 20:55:10.640 PasswordService[5901]: -[PasswordServerPrefsObject getSearchBase]: Unable to locate search base: -1 Can't contact LDAP server

13/03/2015 20:55:10.640 PasswordService[5901]: -[PasswordServerPrefsObject saveXMLData]: ldap_modify_ext_s of the passwordserver config record's plist attribute: -1 Can't contact LDAP server

13/03/2015 20:55:10.684 PasswordService[5901]: int pwsf_GetPublicKey(char *): ldap_search_ext_s cn=authdata for Public Key returned -1

13/03/2015 20:55:10.687 com.apple.launchd[1]: (com.apple.PasswordService[5901]) Exited with code: 1

I have backups (both Time Machine and clones of the hard drive) that I could use to restore the corrupted bit of the configuration, if necessary, but I don't know what to restore.

Mac Mini Server (2009)-OTHER, OS X Mavericks (10.9.5)

Posted on Mar 13, 2015 3:35 PM

Reply

Answer 1

Linc Davis

Level 10

209,541 points

Mar 13, 2015 6:16 PM in response to stevie_west

How to fix failing Open Directory (database "cn=authdata" cannot be opened, err 12)

Reply

Answer 2

stevie_west Author

Level 1

4 points

Mar 14, 2015 1:42 AM in response to Linc Davis

Thanks for that , Linc. The unload and db_recover commands seemed to work (no error messages), but slapd tool mode gave this:

$ sudo launchctl unload /System/Library/LaunchDaemons/org.openldap.slapd.plist

Password:

$ sudo db_recover -h /var/db/openldap/authdata/

$ sudo /usr/libexec/slapd -Tt

5503ed34 bdb_monitor_db_open: monitoring disabled; configure monitor database to enable

5503ed34 bdb(cn=authdata): file id2entry.bdb has LSN 10/5735425, past end of log at 10/4280990

5503ed34 bdb(cn=authdata): Commonly caused by moving a database from one database environment

5503ed34 bdb(cn=authdata): to another without clearing the database LSNs, or by removing all of

5503ed34 bdb(cn=authdata): the log files from a database environment

5503ed34 bdb(cn=authdata): /var/db/openldap/authdata/id2entry.bdb: unexpected file type or format

5503ed34 bdb_db_open: database "cn=authdata": db_open(/var/db/openldap/authdata/id2entry.bdb) failed: Invalid argument (22).

5503ed34 backend_startup_one (type=bdb, suffix="cn=authdata"): bi_db_open failed! (22)

slap_startup failed (test would succeed using the -u switch)

Reply

Answer 3

stevie_west Author

Level 1

4 points

Mar 14, 2015 5:34 AM in response to stevie_west

OK, some progress…

The instructions provided by neocodesoftware and the extra step provided by S-N-Y in this thread moved things on for me:

Mac OS Server Open Directory Will Not Turn On

Open Directory now launches successfully.

I’d previously stopped the Calendar, File Sharing and Mail services, so I rebooted then started these. However, nobody can log into them. Tried another couple of restarts but no joy.

These two blocks keep repeating in the logs (I’ve replaced bits that look sensetive with xxxx):

14/03/2015 12:22:06.444 kdc[89]: UNKNOWN -- xxx.xxxxxxx.xxx$@XXX.XXXXXXX.XXX: no such entry found in hdb

14/03/2015 12:22:06.600 kdc[89]: AS-REQ xxx.xxxxxxx.xxx$@XXX.XXXXXXX.XXX from 127.0.0.1:63305 for krbtgt/XXX.XXXXXXX.XXX@XXX.XXXXXXX.XXX

14/03/2015 12:22:06.602 kdc[89]: received message with invalid client_id 323

14/03/2015 12:22:06.602 kdc[89]: try fetching result again

14/03/2015 12:22:06.604 kdc[89]: received message with invalid client_id 324

14/03/2015 12:22:06.604 kdc[89]: try fetching result again

14/03/2015 12:22:06.605 kdc[89]: received message with invalid client_id 325

and

14/03/2015 12:24:07.212 PasswordService[303]: -[AuthDBFile getPasswordRec:putItHere:unObfuscate:]: user with slot xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx not found. Result: 80 Other (e.g., implementation specific) error

14/03/2015 12:24:07.215 emond[154]: Host at 192.168.xxx.xxx will be blocked for at least 15 minutes

The blocked host is the server’s own IP address on the LAN.

Reply

Answer 4

Linc Davis

Level 10

209,541 points

Mar 14, 2015 7:08 AM in response to stevie_west

Many Open Directory problems can be resolved by taking the following steps. Test after each one, and back up all data before making any changes.

1. The OD master must have a static IP address on the local network, not a dynamic address. It must not be connected to the same network with more than one interface; e.g., Ethernet and Wi-Fi.

2. You must have a working DNS service, and the server's hostname must match its fully-qualified domain name. To confirm, select the server by name in the sidebar of the Server application window, then select the Overview tab. Click the Edit button on the Host Name line. On the Accessing your Server sheet, Domain Name should be selected. Change the Host Name, if necessary. The server must have at least a three-level name (e.g. "server.yourdomain.com"), and the name must not be in the ".local" top-level domain, which is reserved for Bonjour.

3. The primary DNS server used by the server must be itself, unless you're using another server for internal DNS. The only DNS server set on the clients should be the internal one, which they should get from DHCP if applicable.

4. If you have accounts with network home directories, make sure the URL's are correct in the user settings. A return status of 45 from the authorizationhost daemon in the log may mean that the URL for mounting the home directory was not updated after a change in the hostname.

5. Only if you're still running Mavericks server, follow these instructions to rebuild the Kerberos configuration on the server.

6. If you use authenticated binding, check the validity of the master's certificate. The common name must match the hostname and domain name. Deselecting and then reselecting the certificate in Server.app has been reported to have an effect in some cases. Otherwise delete all certificates and create new ones.

7. Unbind and then rebind the clients in the Users & Groups preference pane. Use the fully-qualified domain name of the master.

8. Reboot the master and the clients.

9. Don't log in to the server with a network user's account.

10. Disable any internal firewalls in use, including third-party "security" software.

11. If you've created any replica servers, delete them.

12. If OD has only recently stopped working when it was working before, you may be able to restore it from the automatic backup in /var/db/backups, or from a Time Machine snapshot of that backup.

13. As a last resort, export all OD users. In the Open Directory pane of Server, delete the OD server. Then recreate it and import the users. Ensure that the UID's are in the 1001+ range.

If you get this far without solving the problem, then you'll need to examine the logs in the Open Directory section of the log list in the Server app, and also the system log on the clients.

Reply

Answer 5

stevie_west Author

Level 1

4 points

Mar 14, 2015 11:37 AM in response to Linc Davis

Phew! We're up and running again. As I had full backups of the server, I was able to eventually restore the Open Directory Data using the link in step 12:

http://help.apple.com/advancedserveradmin/mac/3.0/#apdD88A9F25-BCA9-4AD8-A317-60 DB8A52353C

I had to boot another Mac from one of the backups (disconnected from the network to avoid confusion) and export the data from there, then import it on the actual server. Then I just had to create a new certificate to get it all working.

Lessons I've learned are:

A hard restart of the server can corrupt the Open Directory setup.
You can never have too many backups.
Backing up the Open Directory data separately is worthwhile, even if you have other backups of the whole machine.

Thanks!

Reply