Wiki won't accept OD user logins after upgrading to server 3.1.1

I took a chance and (after backing up) upgraded to Server 3.1.2. Problem is resolved. Don't know if it was a bug in 3.1.1 or if the upgrade to 3.1.1 hadn't gone properly for Wiki and Profile Manager (which also has the same issue). Both working okay now.

Okay, this problem has re-surfaced (on server 3.1.2). Here are the symptoms:

1. It only affects network (Open Directory) users. It affects all network users.

2. It only affects the Wiki service; OD users can authenticate for all other services (Mail, Contacts, Calendar, VPN, Time Machine, OS X login on client machines) without problem.

3. Stopping and restarting the Wiki Servcie and the OD servcie does not fix the problem, but rebooting the server does. Things will then be fine for a while (I have not yet quantified how long 'a while' is) and then the problems occurs again.

4. When the problem is present, even a newly created user cannot login to the Wiki servcie.

5. When the login failure occurs, the collabd.log file contains the following:

Jun 1 09:52:23 anarion.home.thejenkinsfamily.org.uk collabd[73366] <Info>: [CSServiceDispatcher.m:237 b904000 +10979ms] TIMER: 0ms ---> Executed [http]Request{SettingsService.authSettings()}

Jun 1 09:52:23 anarion.home.thejenkinsfamily.org.uk collabd[73366] <Info>: [CSServiceDispatcher.m:237 b904000 +0ms] TIMER: 1ms ---> Executed [http]Request{AuthService.challengeForUsername:advanced:(newtesty,1)}

Jun 1 09:52:23 anarion.home.thejenkinsfamily.org.uk collabd[73366] <Info>: [CSExecutionTimer.m:14 b904000 +0ms] TIMER: 2ms ---> Executed batch BatchRequest{(

"[http]Request{AuthService.challengeForUsername:advanced:(newtesty,1)}"

)}

Jun 1 09:52:23 anarion.home.thejenkinsfamily.org.uk collabd[73366] <Info>: [CSServiceDispatcher.m:237 b904000 +119ms] TIMER: 0ms ---> Executed [http]Request{SettingsService.authSettings()}

Jun 1 09:52:23 anarion.home.thejenkinsfamily.org.uk collabd[73366] <Error>: [CSAuthService.m:316 b904000 +0ms] Digest did not validate

Jun 1 09:52:23 anarion.home.thejenkinsfamily.org.uk collabd[73366] <Error>: [CSServiceDispatcher.m:260 b904000 +0ms] Caught exception "Invalid Credentials" [CSAuthBadDigest] executing [http]Request{AuthService.validateUsernameAndPasswordDigest:remember:(<<scrubbe d>>)}:

(

0 CoreFoundation 0x00007fff920e325c __exceptionPreprocess + 172

1 libobjc.A.dylib 0x00007fff95709e75 objc_exception_throw + 43

2 CSService 0x000000010897ca2e -[CSAuthService sessionForDigest:remember:] + 1583

3 CSService 0x000000010897c3a7 -[CSAuthService validateUsernameAndPasswordDigest:remember:] + 65

4 CoreFoundation 0x00007fff91fcea5c __invoking___ + 140

5 CoreFoundation 0x00007fff91fce8c4 -[NSInvocation invoke] + 308

6 CSService 0x00000001088eba40 -[CSServiceDispatcher executeRequest:asPartOfBatch:usingServiceImpl:] + 4589

7 CSService 0x00000001088ec4dc __43-[CSServiceDispatcher executeBatchRequest:]_block_invoke_3 + 83

8 CSService 0x00000001088f1df0 -[NSArray(CollabBlockMethods) map:] + 233

9 CSService 0x00000001088ec435 __43-[CSServiceDispatcher executeBatchRequest:]_block_invoke_2 + 160

10 CSService 0x00000001088f2490 +[CSExecutionTimer recordTime:ofBlock:] + 78

11 CSService 0x00000001088f22c2 +[CSExecutionTimer timerNamed:aroundBlock:] + 77

12 CSService 0x00000001088ec1dd __43-[CSServiceDispatcher executeBatchRequest:]_block_invoke + 296

13 PostgreSQLClient 0x0000000108855941 -[PGCConnection transactionInBlock:onError:] + 150

14 CSService 0x00000001088ec03b -[CSServiceDispatcher executeBatchRequest:] + 285

15 CSService 0x000000010896f748 +[CSServiceDispatchHTTPRouter routeServiceRequest:response:] + 999

16 CSService 0x00000001088f2d1f __21-[CSServiceBase init]_block_invoke_6 + 48

17 CSService 0x000000010896c81a __53-[CSRoutingHTTPConnection httpResponseForMethod:URI:]_block_invoke + 95

18 CSService 0x000000010896fd6c -[CSHTTPBackgroundResponse bounce:] + 286

19 Foundation 0x00007fff91a2276b __NSThread__main__ + 1318

20 libsystem_pthread.dylib 0x00007fff98b0f899 _pthread_body + 138

21 libsystem_pthread.dylib 0x00007fff98b0f72a _pthread_struct_init + 0

22 libsystem_pthread.dylib 0x00007fff98b13fc9 thread_start + 13

)

Jun 1 09:52:23 anarion.home.thejenkinsfamily.org.uk collabd[73366] <Info>: [CSExecutionTimer.m:14 b904000 +0ms] TIMER: 3ms ---> Executed batch BatchRequest{(

"[http]Request{AuthService.validateUsernameAndPasswordDigest:remember:(<<scrubb ed>>)}"

)}

Anyone have any suggestions before I log a bug with Apple?

I don't think there can be a general issue with OD Passwords and Wiki in Server 3.1.1 / 3.1.2 since mine work most of the time without needing to enable plain text authentication; something I am not keen to do even though WiKi access is forced to use HTTPS in my setup.

I have narrowed the trigger for this down to a procedure that runs once a week late at night on my server that:

1. Stops most services via 'serveradmin'

2. Runs a Carbon Copy Cloner 'clone' backup of the server system disk to a partition on an external drive.

3. Restarts the stopped services using 'serveradmin'

After this procedure has completed I always see the authentication problem. A server reboot resolves the problem so I added a reboot step as step 4 above and now I no longer experience the issue. Not ideal so I have logged a bug with Apple and I have been contacted by the Server dev team who tell me they are investigating it...

Chris