Authentication Delays / Slow Authentication for Open Directory Users

Question

Level 1

5 points

Authentication Delays / Slow Authentication for Open Directory Users

I'm experiencing delays when authenticating Open Directory users and it absolutely has me at my wit's end.

The problem is quite simple: any time an Open Directory user authenticates his password there is a delay of at least 5-10 seconds. This goes for clients that are bound to the directory server and also authenticating locally on the server. Here are some examples:

* On the server, there is a several second delay on the Login Window screen when trying to log in using an Open Directory account. Logging in as a local user is instantaneous.
* In Workgroup manager, authenticating as the Directory Administrator takes several seconds.
* On a remote computer, sharing the screen using an Open Directory user take several seconds and again, a local user is instantaneous. Screen sharing takes particularly long and often temporarily shows a sheet saying it has lost the connection with the server while authenticating.
* Connecting with AFP takes several seconds when using an Open Directory login
* On a client computer, unlocking the screen after sleep or screen saver takes several seconds for Open Directory users
* Connecting with SSH does NOT exhibit the behavior

In addition to all of this, I've seen periodic random unexplainable freezes for several seconds on client computers that are bound to the directory even when logged in as a local user account (and with no other users logged in.) For example, launching applications often results in a freeze. After unbinding the computer from the directory the problem goes away entirely.

The history of the problem:
Used Tiger Server for over a year = no problems
Clean install of Leopard Server 10.5.0 back in October = no problems
Update to Leopard Server 10.5.1 = no problems

Then, all of the sudden one day several weeks back I started having problems. The server had been up for a few weeks. I didn't install any updates. I didn't change any configuration. Literally the only thing that I had done recently was unplug the Apple Cinema Display and keyboard+mouse that was connected to the server. Then I started having problems so I plugged the display, keyboard and mouse back in to troubleshoot it. I cleared the directory services caches on my server and clients and rebooted the Airport Base Station that's serving as my router and eventually the problem went away. I wish I could tell you which of those things resolved the problem but I have no idea. It was fine for a couple more weeks (and incidentally I once again unplugged the display, keyboard and mouse from the server). Then last week I started having problems again and this time no amount of rebooting, cache clearing, rebinding, troubleshooting using information in these forums or anything else will fix the problem. I only mention the display/keyboard/mouse thing because it's literally the only thing I changed around the time the problems started happening. I truly don't think it has anything to do with it.

So in desperation I backed up and did a clean install today. Here's the process I used:
0. Erase the disk
1. Install Leopard Server 10.5.0 from the install DVD
2. In the setup assistant, use the Advanced Configuration option but I didn't enable any services. Set up network settings and host name of myserver.mydomain.private.
3. Reboot
4. Use Software Update to update to 10.5.1 and Security Update 2007-009 v1.1
5. Reboot
6. Configure DNS (see below for detailed configuration)
7. Reboot
8. Change role to Open Directory Master
9. Reboot

... and the problem is still there. Simply logging into the server GUI with the Directory Administrator account has the delay. Authenticating in Workgroup Manager has the delay. I haven't even bothered to set up AFP or any other users yet. I'm truly at my wit's end and I'm ready to chuck the server out the window.

I've done a lot of googling and searching of these forums looking for answers. All of the responses seem to point to a problem with DNS or with the Kerberos realm. I believe all of my setup is correct. Here it is:

== Basic Configuration ==
OS: Mac OS X Server 10.5.1 (9B18) with Security Update 2007-009 v.1.1
Services Enabled:
DNS
Open Directory
(All other services are not yet enabled)

== DNS Setup ==
Primary Zone: mydomain.private.
Allows zone transfer: no
Nameservers: ns.mydomain.private.

myserver (Machine) 10.0.22.201
ns (Alias) myserver.mydomain.private.

Reverse Zone: 22.0.10.in-addr.arpa.
10.0.22.201 (Reverse Mapping) myserver.mydomain.private.

Accept recursive queries from the following networks:
localnets

Forwarder IP Addresses:
208.67.222.222
208.67.220.220

== Open Directory Setup ==
Role: Open Directory Master
LDAP Search Base: dc=myserver,dc=mydomain,dc=private
Kerberos Realm: myserver.mydomain.private

== Network Configuration ==
Configure: Manually
IP Address: 10.0.22.201
Subnet Mask: 255.255.255.0
Router: 10.0.22.1
DNS Server: 127.0.0.1
Search Domains: mydomain.private

== Other Stuff ==
Using 'changeip -checkhostname' verifies that the hostname and DNS hostname are both myserver.mydomain.private.

I set the realm to myserver.mydomain.private (though the default was myserver.local) based on the advice of another poster to this forum. Kerberos.app reveals something interesting: the kdc and admin servers are both myserver.local and the domains are .local and local. I tried changing all instances of 'local' to 'mydomain.private' to see if that would solve the problem. No luck.

I verified on a client that 'host myserver' and 'host 10.0.22.201' return proper DNS and reverse DNS resolutions.

Hopefully one of the gurus out there will be able to help me out.

Thanks,
jeff

iMac 24", MacBook Pro, Mac Mini, Mac OS X (10.5.1)

Posted on Dec 29, 2007 11:30 PM

Reply

Answer 1

Dec 30, 2007 10:35 AM in response to Jeff Krueger

Hi

On the server issue:

sudo changeip -checkhostname

and post the result. It does seem as if there is something wrong with the Kerberos Realm. Possibly your DNS retained some incorrect information from a previous config attempt. You may end up having to take steps to demote to Standalone, sort out the DNS and to re-promote afterwards.

I’ve always used the server’s own IP address in the DNS Server’s Field rather than the loopback address.

Hope this helps, Tony

Reply

Answer 2

Jeff Krueger Author

Level 1

5 points

Dec 30, 2007 3:10 PM in response to Antonio Rocco

Here's the info:

==

myserver:~ admin$ sudo changeip -checkhostname

Primary address = 10.0.22.201

Current HostName = myserver.mydomain.private
DNS HostName = myserver.mydomain.private

The names match. There is nothing to change.

==

Prior to the clean install I did follow your advice from another thread about demoting to a Standalone Server and then re-promoting to an Open Directory Master. It didn't solve the problem before which is why I did a clean install.

For kicks I changed the DNS address to the server's IP (10.0.22.201) instead of the loopback address to see if it would make any difference. No dice.

Reply

Answer 3

Jeff Krueger Author

Level 1

5 points

Dec 30, 2007 4:44 PM in response to Jeff Krueger

I gathered together some log information for when I try to authenticate user 'diradmin' in Workgroup Manager. You can see from the log messages that this authentication took 4 seconds. There's an interesting error message in slapd.log (see below) but it doesn't say what it's looking for in the keytab that it's not finding. Grr! I've provided a listing of the principles in my keytab. I haven't monkeyed around with it at all -- this is just what resulted from promoting the server to an Open Directory Master.

== kdc.log ==
Dec 30 18:21:48 myserver.mydomain.private krb5kdc[79](debug): handling authdata
Dec 30 18:21:48 myserver.mydomain.private krb5kdc[79](debug): handling authdata
Dec 30 18:21:48 myserver.mydomain.private krb5kdc[79](debug): .. .. ok
Dec 30 18:21:48 myserver.mydomain.private krb5kdc[79](debug): .. .. ok
Dec 30 18:21:48 myserver.mydomain.private krb5kdc[79](info): AS_REQ (7 etypes {18 17 16 23 1 3 2}) fe80::216:cbff:fea5:f3ce: ISSUE: authtime 1199060508, etypes {rep=16 tkt=16 ses=16}, diradmin@MYSERVER.MYDOMAIN.PRIVATE for krbtgt/MYSERVER.MYDOMAIN.PRIVATE@MYSERVER.MYDOMAIN.PRIVATE
Dec 30 18:21:48 myserver.mydomain.private krb5kdc[79](info): AS_REQ (7 etypes {18 17 16 23 1 3 2}) fe80::216:cbff:fea5:f3ce: ISSUE: authtime 1199060508, etypes {rep=16 tkt=16 ses=16}, diradmin@MYSERVER.MYDOMAIN.PRIVATE for krbtgt/MYSERVER.MYDOMAIN.PRIVATE@MYSERVER.MYDOMAIN.PRIVATE
Dec 30 18:21:52 myserver.mydomain.private krb5kdc[79](info): TGS_REQ (7 etypes {18 17 16 23 1 3 2}) fe80::216:cbff:fea5:f3ce: ISSUE: authtime 1199060508, etypes {rep=16 tkt=16 ses=16}, diradmin@MYSERVER.MYDOMAIN.PRIVATE for ldap/myserver.local@MYSERVER.MYDOMAIN.PRIVATE
Dec 30 18:21:52 myserver.mydomain.private krb5kdc[79](info): TGS_REQ (7 etypes {18 17 16 23 1 3 2}) fe80::216:cbff:fea5:f3ce: ISSUE: authtime 1199060508, etypes {rep=16 tkt=16 ses=16}, diradmin@MYSERVER.MYDOMAIN.PRIVATE for ldap/myserver.local@MYSERVER.MYDOMAIN.PRIVATE

== slapd.log ==
Dec 30 18:21:48 myserver slapd[36]: <= bdb substringcandidates: (authAuthority) index_param failed (18)
Dec 30 18:21:52 myserver slapd[36]: SASL [conn=20] Failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (No principal in keytab matches desired name)

== sudo klist -k ==
Keytab name: FILE:/etc/krb5.keytab
KVNO Principal
---- --------------------------------------------------------------------------
3 afpserver/LKDC:SHA1.D711BEA4D0DDB570D64ED88C5D06A78A34B7167C@LKDC:SHA1.D711BEA4 D0DDB570D64ED88C5D06A78A34B7167C
3 afpserver/LKDC:SHA1.D711BEA4D0DDB570D64ED88C5D06A78A34B7167C@LKDC:SHA1.D711BEA4 D0DDB570D64ED88C5D06A78A34B7167C
3 afpserver/LKDC:SHA1.D711BEA4D0DDB570D64ED88C5D06A78A34B7167C@LKDC:SHA1.D711BEA4 D0DDB570D64ED88C5D06A78A34B7167C
3 cifs/LKDC:SHA1.D711BEA4D0DDB570D64ED88C5D06A78A34B7167C@LKDC:SHA1.D711BEA4D0DDB 570D64ED88C5D06A78A34B7167C
3 cifs/LKDC:SHA1.D711BEA4D0DDB570D64ED88C5D06A78A34B7167C@LKDC:SHA1.D711BEA4D0DDB 570D64ED88C5D06A78A34B7167C
3 cifs/LKDC:SHA1.D711BEA4D0DDB570D64ED88C5D06A78A34B7167C@LKDC:SHA1.D711BEA4D0DDB 570D64ED88C5D06A78A34B7167C
3 vnc/LKDC:SHA1.D711BEA4D0DDB570D64ED88C5D06A78A34B7167C@LKDC:SHA1.D711BEA4D0DDB5 70D64ED88C5D06A78A34B7167C
3 vnc/LKDC:SHA1.D711BEA4D0DDB570D64ED88C5D06A78A34B7167C@LKDC:SHA1.D711BEA4D0DDB5 70D64ED88C5D06A78A34B7167C
3 vnc/LKDC:SHA1.D711BEA4D0DDB570D64ED88C5D06A78A34B7167C@LKDC:SHA1.D711BEA4D0DDB5 70D64ED88C5D06A78A34B7167C
3 cifs/myserver.local@MYSERVER.MYDOMAIN.PRIVATE
3 cifs/myserver.local@MYSERVER.MYDOMAIN.PRIVATE
3 cifs/myserver.local@MYSERVER.MYDOMAIN.PRIVATE
3 ldap/myserver.local@MYSERVER.MYDOMAIN.PRIVATE
3 ldap/myserver.local@MYSERVER.MYDOMAIN.PRIVATE
3 ldap/myserver.local@MYSERVER.MYDOMAIN.PRIVATE
3 xgrid/myserver.local@MYSERVER.MYDOMAIN.PRIVATE
3 xgrid/myserver.local@MYSERVER.MYDOMAIN.PRIVATE
3 xgrid/myserver.local@MYSERVER.MYDOMAIN.PRIVATE
3 vpn/myserver.local@MYSERVER.MYDOMAIN.PRIVATE
3 vpn/myserver.local@MYSERVER.MYDOMAIN.PRIVATE
3 vpn/myserver.local@MYSERVER.MYDOMAIN.PRIVATE
3 ipp/myserver.local@MYSERVER.MYDOMAIN.PRIVATE
3 ipp/myserver.local@MYSERVER.MYDOMAIN.PRIVATE
3 ipp/myserver.local@MYSERVER.MYDOMAIN.PRIVATE
3 xmpp/myserver.local@MYSERVER.MYDOMAIN.PRIVATE
3 xmpp/myserver.local@MYSERVER.MYDOMAIN.PRIVATE
3 xmpp/myserver.local@MYSERVER.MYDOMAIN.PRIVATE
3 XMPP/myserver.local@MYSERVER.MYDOMAIN.PRIVATE
3 XMPP/myserver.local@MYSERVER.MYDOMAIN.PRIVATE
3 XMPP/myserver.local@MYSERVER.MYDOMAIN.PRIVATE
3 host/myserver.local@MYSERVER.MYDOMAIN.PRIVATE
3 host/myserver.local@MYSERVER.MYDOMAIN.PRIVATE
3 host/myserver.local@MYSERVER.MYDOMAIN.PRIVATE
3 smtp/myserver.local@MYSERVER.MYDOMAIN.PRIVATE
3 smtp/myserver.local@MYSERVER.MYDOMAIN.PRIVATE
3 smtp/myserver.local@MYSERVER.MYDOMAIN.PRIVATE
3 nfs/myserver.local@MYSERVER.MYDOMAIN.PRIVATE
3 nfs/myserver.local@MYSERVER.MYDOMAIN.PRIVATE
3 nfs/myserver.local@MYSERVER.MYDOMAIN.PRIVATE
3 http/myserver.local@MYSERVER.MYDOMAIN.PRIVATE
3 http/myserver.local@MYSERVER.MYDOMAIN.PRIVATE
3 http/myserver.local@MYSERVER.MYDOMAIN.PRIVATE
3 HTTP/myserver.local@MYSERVER.MYDOMAIN.PRIVATE
3 HTTP/myserver.local@MYSERVER.MYDOMAIN.PRIVATE
3 HTTP/myserver.local@MYSERVER.MYDOMAIN.PRIVATE
3 pop/myserver.local@MYSERVER.MYDOMAIN.PRIVATE
3 pop/myserver.local@MYSERVER.MYDOMAIN.PRIVATE
3 pop/myserver.local@MYSERVER.MYDOMAIN.PRIVATE
3 imap/myserver.local@MYSERVER.MYDOMAIN.PRIVATE
3 imap/myserver.local@MYSERVER.MYDOMAIN.PRIVATE
3 imap/myserver.local@MYSERVER.MYDOMAIN.PRIVATE
3 ftp/myserver.local@MYSERVER.MYDOMAIN.PRIVATE
3 ftp/myserver.local@MYSERVER.MYDOMAIN.PRIVATE
3 ftp/myserver.local@MYSERVER.MYDOMAIN.PRIVATE
3 afpserver/myserver.local@MYSERVER.MYDOMAIN.PRIVATE
3 afpserver/myserver.local@MYSERVER.MYDOMAIN.PRIVATE
3 afpserver/myserver.local@MYSERVER.MYDOMAIN.PRIVATE

Reply

Answer 4

Dec 31, 2007 9:14 AM in response to Jeff Krueger

Hi

Whatt exactly is your server called? myserver.mydomain.private or myserver.local.mydomain.private?

There is something not quite right somewhere. Review your DNS Service again and make sure the name in the Nameservers field is what it should be as well as making sure Fully Qualified is not ticked in the Machine Record for that server.

Hope this helps, Tony

Reply

Answer 5

Jeff Krueger Author

Level 1

5 points

Jan 1, 2008 10:53 PM in response to Antonio Rocco

My server is called myserver.mydomain.private. There is no .local in the name. The name in the Machine record in DNS is not fully qualified. Verification of correctly operating DNS:

myserver:~ admin$ host myserver.mydomain.private
myserver.mydomain.private has address 10.0.22.201

myserver:~ admin$ host 10.0.22.201
201.22.0.10.in-addr.arpa domain name pointer myserver.mydomain.private.

I'm starting to think there's a bug somewhere in the Server Admin application. From the Open Directory Admin Guide, page 99: "Realm Name: This field is preset to be the same as the server’s DNS name converted to capital letters. This is the convention for naming a Kerberos realm. If necessary, you can enter a different name."

The problem is, even though DNS appears to be set up correctly and all of the checks outlined on page 99 are correct, the default realm name is coming up as MYSERVER.LOCAL and NOT MYSERVER.MYDOMAIN.PRIVATE. If I manually change the realm name to MYSERVER.MYDOMAIN.PRIVATE, I end up with what you saw in my earlier post where the principals are messed up, e.g. afpserver/myserver.local@MYSERVER.MYDOMAIN.PRIVATE.

I thought about hand-editing /Library/Preferences/edu.mit.Kerberos but I assume the principals would still be wrong and I'd have to edit those, too. I don't really want to go that route.

So I'm stuck. It seems that Server Admin is somehow coming up with the Bonjour name instead of the DNS hostname and I can't figure out why. Does anyone have any other suggestions of what to look at?

Reply

Answer 6

Jan 2, 2008 8:30 AM in response to Jeff Krueger

Hi Jeff

Personally I don’t know of any bug in Server Admin. I don’t see the problems you are seeing although judging by some of the other posts - some do.

All I can think of now is perhaps you mistakenly entered the wrong information at the Server Assistant stage, the bit after you enter the network settings. The part where it asks you to enter the DNS hostname as well as the server name.

Whatever it is Open Directory seems to be hanging onto something which it should not be. You could demote to Standalone and restart the server and re-promote to see if that clears it.

Hope this helps, Tony

Reply

Answer 7

Jeff Krueger Author

Level 1

5 points

Jan 2, 2008 7:06 PM in response to Antonio Rocco

Well, I've solved the problem. I'm going to post the details in hopes that it helps somebody else.

In short: the authentication delays I was seeing were caused by messed up Kerberos principals. In my case, they were appearing as servicename/myserver.local@MYSERVER.MYDOMAIN.PRIVATE instead of servicename/myserver.mydomain.private@MYSERVER.MYDOMAIN.PRIVATE.

The moral of the story is *if you are configuring Kerberos and the default realm name comes up as MYSERVER.LOCAL (your server's Bonjour name), cancel and fix the problem* before proceeding with changing the role.

I decided to do another clean install of Leopard Server so I could document the steps and possibly file a bug report with Apple. I gave the server the Computer Name (Bonjour name) My Server and the hostname myserver.mydomain.private. Then I configured DNS and verified that lookups were working properly for my server's hostname and IP. I also verified that 'changeip -checkhostname' was correct as discussed earlier in this thread.

I began to change my server's role to Open Directory Master. When I reached the sheet where you specify the Kerberos realm, it defaulted MY-SEVER.LOCAL again. (Resisting the urge to throw something breakable) I canceled and started looking around for what was wrong.

I noticed that in Server Admin > My Server > Settings > Network that the network information was not as I would expect. It said:

Name Family IP Address DNS Name
en0 IPv4 10.0.22.201 myserver.mydomain.private
Ethernet (en0) IPv6 xxxx:xxxx:xxxx... My-Server.local

That is, Server Admin seemed to think that IPv6 is the primary interface (this is my guess based on the designation "Ethernet".) I don't know why it assumed this. When I set up the server I just left IPv6 at the default setting (Configure Automatically). I didn't configure DNS for IPv6 (in fact, I confess complete ignorance as to how to even do that.)

On the hunch that this had something to do with it, I went to the Network preference pane and turned off IPv6. I quit Server Admin and relaunched and it no longer showed any interfaces on the Network tab, so I rebooted the server. After the reboot, I saw:

Name Family IP Address DNS Name
Ethernet (en0) IPv4 10.0.22.201 myserver.mydomain.private

Ahh. That's more like it. I went to change the role to Open Directory Master and the Kerberos realm defaulted as MYSERVER.MYDOMAIN.PRIVATE and after changing the role, the resulting list of principles looks like this:

3 cifs/myserver.mydomain.private@MYSERVER.MYDOMAIN.PRIVATE
3 cifs/myserver.mydomain.private@MYSERVER.MYDOMAIN.PRIVATE
3 cifs/myserver.mydomain.private@MYSERVER.MYDOMAIN.PRIVATE
3 ldap/myserver.mydomain.private@MYSERVER.MYDOMAIN.PRIVATE
3 ldap/myserver.mydomain.private@MYSERVER.MYDOMAIN.PRIVATE
3 ldap/myserver.mydomain.private@MYSERVER.MYDOMAIN.PRIVATE
...
3 afpserver/myserver.mydomain.private@MYSERVER.MYDOMAIN.PRIVATE
3 afpserver/myserver.mydomain.private@MYSERVER.MYDOMAIN.PRIVATE
3 afpserver/myserver.mydomain.private@MYSERVER.MYDOMAIN.PRIVATE

And as a result, authentication of Open Directory users is once again instantaneous. Woo-hoo!

So I guess the remaining questions are:
1. Why does Server Admin (or more likely, the behind-the-scenes tools) use the IPv6 hostname when creating the Kerberos realm?
2. If this messes things up and there's apparently no GUI to configure DNS for IPv6, why is it enabled by default?
3. Should I re-enable IPv6 now that Kerberos is set up correctly?

Anyway, I hope this information saves some others from the headaches I've endured trying to figure this out.

Reply

Answer 8

Jan 29, 2008 12:12 PM in response to Jeff Krueger

Jeff you are my hero. I was having the exact same Kerberos issue and it likely would have taken me days of frustration (if ever) to locate the problem.

You rock. Thank you!

Reply

Answer 9

rkovelman

Level 2

320 points

Mar 5, 2008 4:43 PM in response to Jeff Krueger

When ever you configure a Mac server it MUST ALWAYS be the FQDN. .local is used for Bonjour and Windows servers which still is not allowed to be used. Always make sure you use or see a .com, .net, .org etc.

Reply