Password problem after migrating to Mountain Lion Server

Question

Level 1

25 points

Password problem after migrating to Mountain Lion Server

Hello everyone,

Yesterday, I migrated our Lion Server to Mountain Lion Server. Everything seemed to work fine. Except since this morning, none of the network user cannot connect to their calendar, reminders, and wiki. They can connect to their account and to their mails. The following line appears multiple times in ApplePasswordServer.Error.log:

Aug 16 2012 16:52:50 700250us client response doesn't match what we generated

It seems that only web services are concerned (vpn, mobile accounts, and mails are working). My initial guess is that the hash computed on the basis of the user password is not computed on the same way on the client machine (which is running Mountain Lion by the way) and on the server. On the other hand, this would be very surprising, since all this stuff is based on standards and unlikely to have changed since Lion.

I tried to create a new "Test" user. Even this new user, created after the migration, cannot connect to its calendar, etc. I also tried to reset my user password using the Server App. It makes no difference, the same lines appear in the logs.

Is anyone experiencing a similar problem ? Does anyone have a clue of what to try next ?

Thanks a lot !

OS X Server, Mountain Lion Server

Posted on Aug 16, 2012 8:14 AM

Reply

Answer 1

elgringito Author

Level 1

25 points

Aug 16, 2012 9:06 AM in response to elgringito

Update to my own post : Looking at ApplePasswordServer.Server.log, I see

Aug 16 2012 18:00:27 113394us AUTH2: {0xd6ace...., elgringito} WEBDAV-DIGEST authentication failed, SASL error -13 (password incorrect).

This line appears when I try to connect to the wiki. Yet, a few lines before :

Aug 16 2012 18:00:12 75192us AUTH2: {0xd6ace...., elgringito} DIGEST-MD5 authentication succeeded

This line appears when I log in to my mobile account, using the same password that doesn't get accepted when trying to log in on the wiki.

Anybody ? Thanks !

Reply

Answer 2

mbresink

Level 1

54 points

Aug 23, 2012 12:39 PM in response to elgringito

I can only confirm that I have exactly the same problem with one of my Open Directory user accounts. Resetting the password does not change anything. However, all other accounts (with basically the same setup) mysteriously work fine.

This would suggest that something in the password server itself is damaged. Due to total lack of documentation, it's difficult to find out what exactly is going on here.

Reply

Answer 3

mbresink

Level 1

54 points

Aug 23, 2012 2:39 PM in response to mbresink

I found out the following:

After looking at the contents of the password server database using the slot numbers of several user accounts, it seems that all users where WEBDAV-DIGEST authentication is failing, have two entries for the digest method "*cmusaslsecretDIGEST". This is obviously wrong. All users who can authenticate successfully have only one such entry.

Deleting and recreating the user account has no effect. In fact, updating the password server with a new entry may actually trigger this error. It could be that all users where this is failing have changed their passwords after the server was updated to Mountain Lion.

It would be interesting to know if you also see duplicate entries for "*cmusaslsecretDIGEST" in the database. You can display a password server record via the user account's slot number (in your example, the 0xd6ace...) using the command

sudo mkpassdb -dump <slot-number>

At the end of the record dump, you should see 10 digest entries with their method identifiers.

Reply

Answer 4

elgringito Author

Level 1

25 points

Aug 24, 2012 2:04 AM in response to mbresink

Hello mbresink,

thanks for sharing ! I did as you said. Only one user changed his password since the migration to mountain Lion. If I try to display his password's hashes, using the mkpassdb -dump command, I obtain 10 digests (some of which are empty). The "digest 7" indicates method: *cmusaslsecretDIGEST but indicates no digest value. Strange, isn't it ? It seems "half"-empty. The other digests are either completely empty (e.g. digest 8) or indicate both a method and a value.

If I try to display the password digests of a user who did not change his passwords since the migration, the "digest 7" slot is completely empty (juste as digest 8 for example).

There is obviously something wrong for me here. It is different for me than for you though, since I do not see two digest values in slot 7 (since, in all cases, I do not see any digest value at all for digest 7).

Even if I knew how to compute the digest value from the password (I guess this is feasible, but I do not think that wasting time on this is worth it), and insert the hash in the database, I would still need to update the "slot checksum" at the end (and God knows what other slot should be updated as well).

The only hope I had was to be able to use the command mkpassdb -setpassword, in order to avoid the GUI (I guess it won't change anything, but it might be worth a try). But I couldn't make it work. The password digests are simply not updated.

Problem not solved yet... But thanks for your help.

Reply

Answer 5

mbresink

Level 1

54 points

Aug 24, 2012 2:33 AM in response to elgringito

Hi elgringito,

thanks a lot for your confirmation!

There might have been some slight misunderstanding: The effect you are seeing appears to be exactly the same as mine. The digest 7 is "half empty", but there is another entry for *cmusaslsecretDIGEST at digest 2. This digest might be the correct one, possibly conflicting with the one at 7.

Digests 1, 8, and 9 are always empty for me, and digest 7 is empty for users who didn't change their passwords.

When I fnd some time, I'll try to reproduce this on an empty server and file a bug report with Apple.

Reply

Answer 6

mbresink

Level 1

54 points

Aug 24, 2012 6:05 AM in response to mbresink

Unfortunately, this theory did not prove to be correct.

Although it is the case that users who changed their passwords have duplicate cmusaslsecretDIGEST entries, this does not seem to be the cause of the problem.

A "clean" test server also shows the duplicate entries, but WEBDAV-DIGEST authentication is working anyway.

Hm... :-(

Reply

Answer 7

elgringito Author

Level 1

25 points

Aug 24, 2012 6:10 AM in response to mbresink

Hello mbresink,

you are completely right. The list of digests of the user who changed his password indeed contains a full "digest 2" (which indeed indicates the method: *cmusaslsecretDIGEST) and a "half empty" digest 7. The other users (who did not update their password since the migration) also have a full "digest 2" but their digest 7 is completely empty. None of the users can connect to the services I mentioned in my first post.

I will also try to set up a new server, maybe under Lion, just to make sure that it is indeed a problem to have two digest entries with the same "cmusaslsecretDIGEST" method. But I won't be able to do so before the 3rd of September.

Thanks again for your help !

Reply

Answer 8

elgringito Author

Level 1

25 points

Aug 24, 2012 10:07 AM in response to mbresink

Too bad the theory did not prove to be correct !

Maybe the problem comes from a bad web server configuration ? E.g. a misplaced/misconfigured .htaccess file ? I am not in front the server right now. I will investigate and report whatever I find.

Bye

Reply

Answer 9

elgringito Author

Level 1

25 points

Sep 12, 2012 3:37 AM in response to mbresink

Hello mbresink,

We did not manage to solve our problem and eventually had to completely re-install the server.

We think that the problem was due to Open Directory. Possibly a "changeip" issue (long before upgrading to ML, we had to change the server ip address, this worked fine on Lion).

Good luck!

Reply

Answer 10

Kostas B

Level 1

94 points

Sep 19, 2012 6:44 AM in response to elgringito

Count me in as well, I have the same issue.

Best regards

Kostas

Reply

Answer 11

mbresink

Level 1

54 points

Sep 19, 2012 8:10 AM in response to Kostas B

I have reproduced this problem on an empty server now. This is definitively a bug in the Mountain Lion Server upgrade procedure.

It seems that in all cases where one of the available authentication mechanisms had been disabled in Lion Server (before the upgrade), WEBDAV-DIGEST authentication will fail in OS X Server on Mountain Lion for all Open Directory user accounts which have either been created after the upgrade, or for which the password has been changed.

Could you either confirm or deny that you had deactivated one of the authentication mechanisms in Lion?

It was common practice to disable unsafe an unneeded mechanisms, recommended by Apple in the administration reference manual.

Reply

Answer 12

Kostas B

Level 1

94 points

Sep 19, 2012 8:24 AM in response to mbresink

Thank you for your answer.

I had Snow Leopard before upgrading. However, I can confirm that no new or old OD accounts can use ANY web services (Web sites realms or Profile Manager).

Best regards

Kostas

Reply

Answer 13

elgringito Author

Level 1

25 points

Sep 19, 2012 8:24 AM in response to mbresink

Hello mbresink,

I just booted a cloned version of my "old" Lion Server. The state of this clone is exactly the one of the server just before I tried to upgrade to Mountain Lion Server. On the clone, when I open "Server Admin", and go to

Open Directory -> Policies -> Authentication

there is indeed one unchecked authentication method (in this particular case, APOP, since we do not use POP for emails).

This confirms what you say !

Reply

Answer 14

felddy

Level 1

11 points

Sep 19, 2012 8:33 AM in response to mbresink

I too had disabled APOP on Lion Server and am seeing the exact same issues with digest authentications failing after a migration to ML Server.

They say misery loves company, but it is even more reassuring finding an active thread on the path to a solution.

If there are any theories I can help explore let me know.

Reply

Answer 15

mbresink

Level 1

54 points

Sep 19, 2012 9:52 AM in response to elgringito

Thanks a lot for your confirmation.

I now have found a solution which works as least for me! YMMV, but it's worth a try. You may test the following steps:

On a client or the server, launch /System/Library/CoreServices/Directory Utility.
Open the Directory Editor.
Connect to the affected /LDAPv3/... node of the OD server and authenticate as directory administrator.
Set the Viewing record type to Config.
Navigate to the entry passwordserver.
Select the attribute XMLPlist.
Search for the section SASLPluginStates. The next lines contain a dictionary data structure between the tags <dict> and </dict>.
Replace this dictionary by the one shown below which is the default configuration of OS X Server Mountain Lion. (For most of you, it will be sufficient to replace OFF by ON for the APOP entry.) WARNING: XML syntax errors may cause serious problems with the Password Server. Don't change any other parts of the XMLPlist entry.
Press Save and quit Directory Utility. You won't need to reboot the server.
Change the password of the affected user account, e.g. using Workgroup Manager.

<dict>

<key>CRYPT</key>

<key>DIGEST-MD5</key>

<key>GSSAPI</key>

<key>KERBEROS_V4</key>

<key>MS-CHAPv2</key>

<key>SMB-LAN-MANAGER</key>

<key>SMB-NTLMv2</key>

<key>TWOWAYRANDOM</key>

<key>WEBDAV-DIGEST</key>

</dict>

Reply