10.8 Server: Problems Updating an Open Directory Replica

Question

Level 1

0 points

10.8 Server: Problems Updating an Open Directory Replica

Dear community

I have two new 10.8.2 servers – one is an OD master, and the other – an OD replica.

Initial replica creation goes well, but for some reason the master can't update it after that. So, for example, a password change does not propagate to the replica. I have to delete and re-create the replica, in order to get the changes.

Here are the LDAP log entries that repeat every minute:

Feb  1 12:43:03 odmaster.example.com slapd[102]: slap_client_connect: URI=ldap://odreplica.example.com:389 ldap_sasl_interactive_bind_s failed (-2)
Feb  1 12:43:03 odmaster.example.com slapd[102]: do_syncrep1: client_connect failed (-1) - searchbase(cn=authdata)
Feb  1 12:43:03 odmaster.example.com slapd[102]: slap_client_connect: URI=ldap://odreplica.example.com:389 ldap_sasl_interactive_bind_s failed (-2)
Feb  1 12:43:03 odmaster.example.com slapd[102]: do_syncrep1: client_connect failed (-1) - searchbase(dc=odmaster,dc=example,dc=com)
Feb  1 12:43:03 odmaster.example.com slapd[102]: do_syncrepl: rid=004 rc -1 retrying
Feb  1 12:43:33: --- last message repeated 1 time ---
Feb  1 12:43:38 odmaster.example.com slapd[102]: SASL [conn=5714] Failure: no secret in database
Feb  1 12:43:38 odmaster.example.com slapd[102]: int slap_sasl_bind(Operation *, SlapReply *): Error to increment failed login count for uid=updater637333537,cn=users,dc=odmaster,dc=example,dc=com

A similar if not the same issue has been reported here, with no solution:

Note sure what to try.

Thank you for your help.

Mac mini, OS X Mountain Lion (10.8.2)

Posted on Feb 1, 2013 9:58 AM

Reply

Answer 1

ryanhoye

Level 1

0 points

Feb 6, 2013 1:10 PM in response to wimpog

Do you have any other dns machine records pointing to your master? the slave?

The AuthenticationAuthority is a Kerberos ID, if you go to computers in the directory and look at your master's dns name you will see authenticationauthority again, this is a different hash because the dns name is a user in itself.

If you sudo into the /var/db/openldap/authdata folder and look at the files there, you will see similar hashes on those users. There is where I was refering to when I was comparing. I specifically looked at _ldap_replicator on both systems to determine if the hash was the same. The files themselves appear to be almost exactly the same.

Reply

Answer 2

ryanhoye

Level 1

0 points

Feb 6, 2013 7:58 PM in response to ryanhoye

Partial success!

IT IS a Kerberos issue.

I cleared up an DNS issue that was compounding the problems, and used kinit to test a connection. ( I should note that I have been running 2 OD Masters in order to achieve functionality while I've been testing this. In the following: Master reffers to my previously named replica2 and replica will refer to replica1. The setup is the Master is running critical services and cannot be altered extensively, and the replica is being used to test replication functionality from a machine that will not replicate properly)

The non-communicating master would not authenticate _ldap_replicator to the KDC. And through testing in terminal I found that the issue was specific to this Master.

at 1847 I authenticated diradmin on the replica by using kinit diradmin

This resulted in authenticating against the KDC realm on the master. MASTER.EXAMPLE.COM

At 1847, the ldap log stops reporting syncrep1 sasl errors and has successfully pulled a test user account from the master, it did not successfully pull over service related information, so the user shows as 'not allowed'.

However, the Master continues to report syncrep errors, meaning that this could be a temporary/fluke fix. Gravy is that I now get this log in the system logs:

Feb 6 19:44:20 master.example.com kdc[47]: AS-REQ _ldap_replicator@MASTER.EXAMPLE.COM from 127.0.0.1:63200 for krbtgt/MASTER.EXAMPLE.COM@MASTER.EXAMPLE.COM

Feb 6 19:44:20 --- last message repeated 1 time ---

Feb 6 19:44:20 master.example.com kdc[47]: Client sent patypes: REQ-ENC-PA-REP

Feb 6 19:44:20 master.example.com kdc[47]: Need to use PA-ENC-TIMESTAMP/PA-PK-AS-REQ

Feb 6 19:44:20 master.example.com kdc[47]: AS-REQ _ldap_replicator@MASTER.EXAMPLE.COM from 127.0.0.1:57243 for krbtgt/MASTER.EXAMPLE.COM@MASTER.EXAMPLE.COM

Feb 6 19:44:20 --- last message repeated 1 time ---

Feb 6 19:44:20 master.example.com kdc[47]: Client sent patypes: REQ-ENC-PA-REP

Feb 6 19:44:20 master.example.com kdc[47]: Need to use PA-ENC-TIMESTAMP/PA-PK-AS-REQ

Feb 6 19:44:20 master.example.com kdc[47]: AS-REQ _ldap_replicator@MASTER.EXAMPLE.COM from 127.0.0.1:61874 for krbtgt/MASTER.EXAMPLE.COM@MASTER.EXAMPLE.COM

Feb 6 19:44:20 --- last message repeated 1 time ---

Feb 6 19:44:20 master.example.com kdc[47]: Client sent patypes: ENC-TS, REQ-ENC-PA-REP

Feb 6 19:44:20 master.example.com kdc[47]: ENC-TS pre-authentication succeeded -- _ldap_replicator@MASTER.EXAMPLE.COM

Feb 6 19:44:20 master.example.com kdc[47]: Client supported enctypes: aes256-cts-hmac-sha1-96, aes128-cts-hmac-sha1-96, des3-cbc-sha1, arcfour-hmac-md5, using aes256-cts-hmac-sha1-96/aes256-cts-hmac-sha1-96

Feb 6 19:44:20 master.example.com kdc[47]: Requested flags: forwardable

Feb 6 19:44:20 master.example.com kdc[47]: AS-REQ _ldap_replicator@MASTER.EXAMPLE.COM from 127.0.0.1:61473 for krbtgt/MASTER.EXAMPLE.COM@MASTER.EXAMPLE.COM

Feb 6 19:44:20 --- last message repeated 1 time ---

Feb 6 19:44:20 master.example.com kdc[47]: Client sent patypes: ENC-TS, REQ-ENC-PA-REP

Feb 6 19:44:20 master.example.com kdc[47]: ENC-TS pre-authentication succeeded -- _ldap_replicator@MASTER.EXAMPLE.COM

Feb 6 19:44:20 master.example.com kdc[47]: Client supported enctypes: aes256-cts-hmac-sha1-96, aes128-cts-hmac-sha1-96, des3-cbc-sha1, arcfour-hmac-md5, using aes256-cts-hmac-sha1-96/aes256-cts-hmac-sha1-96

Feb 6 19:44:20 master.example.com kdc[47]: Requested flags: forwardable

Take that ML server!

I will let you know if the errors return in the morning after the kerberos ticket for diradmin expires.

Also. Anyone have any thoughts on why I continue to see authentication for some users, particularly those using DIGEST-MD5 and the _ldap_replicator authenticating four times every time they authenticate against the KDC?

Reply

Answer 3

ryanhoye

Level 1

0 points

Feb 7, 2013 8:54 AM in response to ryanhoye

So, logged into server, checked the logs, Replica was still not reporting any syncrep errors. The master was still reporting errors. Restart the Master, errors return to the Replica, and the Master continues to report the message from the System Log above.

I have not yet tried to destroy the ldap on the Master to try rebinding it to another master or anything else. I should also note that my Master in this situation has 2 SSL certificates, but the OD uses the one issued by the OD.

Closer, still not fixed.

Reply

Answer 4

wimpog Author

Level 1

0 points

Feb 7, 2013 9:55 AM in response to ryanhoye

My DNS – both the master and the slave is correct, I checked it over and over again.

/var/db/openldap/authdata contains several binary files, not sure where you found the hashes.

Does your replication work now?

Reply

Answer 5

ryanhoye

Level 1

0 points

Feb 7, 2013 11:17 AM in response to wimpog

No my replication is not working at the moment. I have not yet tried to rebuild the master after my changes.

The hashes are in cn=authdata. As far as I remember they are in that folder.

Reply

Answer 6

wimpog Author

Level 1

0 points

Feb 7, 2013 1:47 PM in response to ryanhoye

How do you plan to rebuild it? Just delete and re-create from start?

What is your hardware?

Reply

Answer 7

ryanhoye

Level 1

0 points

Feb 7, 2013 1:49 PM in response to wimpog

I'll use slapconfig -destoryldapserver after backing it up, and then trying to replicate the master that is working.

I successfully replicated my original setup again. Maybe it will work once this config is cleared up.

Reply

Answer 8

ryanhoye

Level 1

0 points

Feb 10, 2013 10:04 AM in response to ryanhoye

I just reimplimented my original setup with the Master and two replicas and it is successfully replicating and _ldap_replicator is successfully authenticating against the kerberos master.

I think the solution was to clear out some remnants of the old config by moving some old remaining ldif files from inside /var/db/openldap/replication (on the master) and triple checking my DNS.

The DNS proved to be an issue where I found that the replica in question was resolving to another dns name from outside of the machine. Checkhostname was working fine, but kerberos couldn't authenticate against that machine because it couldn't find the realm that matched.

I will update if anything changes, but for now, everything is happy.

Reply

Answer 9

wimpog Author

Level 1

0 points

Feb 10, 2013 6:29 PM in response to ryanhoye

What did you exactly do? Couldy you describe in details?

Did you essentially delete your master and re-created it?

How did you verify your DNS settings?

Thanks, and I'm glad it is working for you.

Reply

Answer 10

ryanhoye

Level 1

0 points

Feb 13, 2013 9:59 AM in response to wimpog

Start by checking if you can kinit from one machine to the other, and check if both machines can "see" eachother as the correct DNS names. IE: you should checking that the reverse lookup of either machine ends up being the domain name you are trying to authenticate to on either end.

So If master.example.com is supposed to be 192.168.0.1 and replica1 is supposed to be 192.168.0.2 be sure that a reverse lookup from the opposite machine results in those entries matching up. Once that is complete, make sure that when you try to kinit from either machine, make sure that the error that presents itself is actually refering to the domain that it should be trying to authenticate against. It may say that it is trying to reach the KDC in the realm, but it may say that it is trying to authenticate against somethingelse.example.com. If you find then that the realm it is trying to authenticate against is correct, and the DNS entries are validated, then you will need to export all of your users using workgroup manager, and then destory the ldap servers (replica first, then master) and start over.

I had stupidly left an A record for redundancy in my DNS config that messed the whole thing up. It was trying to authenticate against that instead of the domain it was supposed to. And the partial success I had was removing that record, and successfully kinit against the correct realm with diradmin. The issue didn't solve itself, presumably because during the replica creation process the record is stored somewhere. Once I cleaned out that information and recreated the replica, it successfully authenticated and the errors stopped. I have successfully created a few test users and had permissions and information populate properly across all three servers.

My other suggestion is to also remove the ldif files that are in /var/db/openldap/replication from the master so that they don't interfere with the recreation of the master and replicas.

So far it is still working. The LDAP log on the master and replicas don't show any errors since the fix.

Reply

Answer 11

Mar 25, 2013 11:30 AM in response to wimpog

So after all the headaches I've been through with OD replicas on both 10.7 and 10.8 I decided to share some hopefully helpful notes.

If you're having trouble with your OD replicas after upgrading your servers to 10.8, do the following:

If your replicas are in AD, unbind them from AD

On the replica, open terminal and enter "sudo slapconfig -destroyldapserver"

remove or move the /etc/krb5.keytab file (not doing this can cause problems. Even if your replica promotes you could have problems with OD accounts authing to services)

verify the forward/reverse DNS entries for your server using nslookup

on your OD Master, open Directory Utility

Navigate to the Directory Editor

Switch the view to /LDAPv3/127.0.0.1/

Switch to the "Computers" section

find any computer accounts for your OD Replica(s)

Note the GeneratedUID of the computer accounts

Delete the computer accounts of your OD replicas

Switch to the "Config" section

Select "ldapreplicas"

edit XMLPlist and remove any entries for your OD Replicas

Select the "ComputerGroups" section

Select the com.apple.opendirectory computer group

Remove your replicas from GroupMembers (this is why we made note of the GeneratedUID earlier), GroupMembership and Members

reboot both master and replica

promote your replica

Reply

Answer 12

May 22, 2013 2:08 PM in response to Joe Swenson

So... is this reliable once working? I have heard that Apple enterprise services won't even tackle an OD replication becasue it is broken.

I have had no luck with getting replication working on an upgraded ldap. I want to do a clean rebuild anyway, so I may try to get a replica working in the process.

Reply

Answer 13

Sep 11, 2013 7:43 AM in response to ryanhoye

Thanks ! Your Post and Joe Swenson´s fixed my problem!

No everthing is replicating! 🙂

What I have done:

First of all check your DNS!!!! Forward & Reverse Lookup has to work!

(nslookup in terminal)

!!!-only the replica server-!!!

1. destroyed replica-server with terminal command (command was mentioned before)

2. deleted etc/krb5.keytab (sudo rm)

3. deleted /var/db/openldap/replication (complete folder ; sudo rm -r <path-to-folder>)

4. created new replica OD

->

Sep 11 15:02:49 <MYREPLICADNSNAME> slapd[395]:

do_syncrep1: CONNECTED(0x7fe502c00040) searchbase(cn=authdata) URI(ldap://<MYREPLICADNSNAME>:389)

Sep 11 15:02:49 <MYREPLICADNSNAME> slapd[395]:

do_syncrep1: CONNECTED(0x7fe5029045e0) searchbase(dc=XXX,dc=XXX,dc=XXX,dc=XX) URI(ldap://<MYREPLICADNSNAME>389)

Created some accounts on my master and sync is working now on my replica.

BTW:

I have NOT unbound my replica from AD (we are running the magic triangle)

AAAAAND:

If you try to connect a client to the replica server, client says, that
<MYREPLICADNSNAME> is a replica of <MYMASTERDNSNAME> and thats why it is not necessary!

Failover works like a charm!

Great! Thanks guys!

Reply