Apple Event: May 7th at 7 am PT

Looks like no one’s replied in a while. To start the conversation again, simply ask a new question.

Level 1

145 points

Directory Service Crashing

I have been testing leopard server and have run into a problem that puzzles me. I hope that somebody can provide some insights on this issue.

Installed 10.5 server, advanced with mobile users and time machine backups. Upgraded mobile accounts from Tiger were less than desirable (sync issues). 10.5 users were fine except that the directory service crashed consistently during home directory syncing.
Did a fresh re-install with upgrade to 10.5.1 before configuration; still the same.

Did even a standard install with standard users only (basic services to make sure that no manual DNS or OD config. caused the problem ) and the Directory Services still go out on me.

This is running on a Xserve G5 dual with 6gig of RAM & 500gb raid1 mirror. That machine ran 10.4 perfectly for the last year plus.

Below are the shortened results form the crash reporter. Any ideas? I was really hoping that I can use leopard server to use iCal server & time machine.

Thanks!

Process: DirectoryService [26]
Path: /usr/sbin/DirectoryService
Identifier: DirectoryService
Version: ??? (???)
Code Type: PPC (Native)
Parent Process: launchd [1]

Date/Time: 2007-11-25 20:33:56.179 -0600
OS Version: Mac OS X Server 10.5.1 (9B18)
Report Version: 6

Exception Type: EXC BADACCESS (SIGSEGV)
Exception Codes: KERN INVALIDADDRESS at 0x000000004b009698
Crashed Thread: 2

Thread 0:
0 libSystem.B.dylib 0x93f689d8 mach msgtrap + 8
1 libSystem.B.dylib 0x93f6f8fc mach_msg + 56
2 com.apple.CoreFoundation 0x922f9664 CFRunLoopRunSpecific + 1828
3 com.apple.CoreFoundation 0x922f9ef4 CFRunLoopRun + 60
4 DirectoryService 0x000156e4 main + 2356
5 DirectoryService 0x00014d78 start + 64

Thread 1:
0 libSystem.B.dylib 0x93f689d8 mach msgtrap + 8
1 libSystem.B.dylib 0x93f6f8fc mach_msg + 56
2 com.apple.CoreFoundation 0x922f9664 CFRunLoopRunSpecific + 1828
3 com.apple.CoreFoundation 0x922f9ef4 CFRunLoopRun + 60
4 DirectoryService 0x00020eb0 CPluginRunLoopThread::ThreadMain() + 228
5 ...ectoryServiceCore.Framework 0x0015a09c DSCThread::Run() + 52
6 ...ectoryServiceCore.Framework 0x0015a2f4 DSLThread::_RunWrapper(void*) + 96
7 libSystem.B.dylib 0x93faabf8 pthreadstart + 316

Thread 2 Crashed:
0 libsqlite3.0.dylib 0x94ae6850 sqlite3 resultvalue + 33024
1 libsqlite3.0.dylib 0x94ae5728 sqlite3 resultvalue + 28632
2 libsqlite3.0.dylib 0x94af1e5c sqlite3 resultvalue + 79628
3 libsqlite3.0.dylib 0x94af1db4 sqlite3 resultvalue + 79460
4 libsqlite3.0.dylib 0x94b3c3dc sqlite3 gettable + 31548
5 libsqlite3.0.dylib 0x94b3fbf4 sqlite3 gettable + 45908
6 libsqlite3.0.dylib 0x94b3fe64 sqlite3_step + 148
7 DirectoryService 0x0006d31c CDSLocalPluginNode::sqlExecSync(char const*, unsigned long) + 100
8 DirectoryService 0x0006eda4 CDSLocalPluginNode::AddRecordIndex(char const*, char const*) + 512
9 DirectoryService 0x0006f9b0 CDSLocalPluginNode::FlushRecord(__CFString const*, __CFString const*, __CFDictionary const*) + 1308
10 DirectoryService 0x0006972c CDSLocalPlugin::CloseRecord(sCloseRecord*) + 304
11 DirectoryService 0x000a4d80 BaseDirectoryPlugin::ProcessRequest(void*) + 520
12 DirectoryService 0x00069564 CDSLocalPlugin::ProcessRequest(void*) + 600
13 DirectoryService 0x00012874 CHandlerThread::RefDeallocProc(unsigned long, unsigned long, CServerPlugin*) + 240
14 DirectoryService 0x0000a73c CRefTable::RemoveRef(unsigned long, unsigned long, long, unsigned long, bool) + 880
15 DirectoryService 0x0001d6fc CRequestHandler::DoCloseRecord(sComData*, long*) + 220
16 DirectoryService 0x00009ce4 CRequestHandler::GetRequestData(sComData*, long*, bool*) + 808
17 DirectoryService 0x00002360 CRequestHandler::HandlePluginCall(sComData**) + 192
18 DirectoryService 0x00003068 CRequestHandler::HandleRequest(sComData**) + 76
19 DirectoryService 0x0002d4b8 dsmig do_apicall + 380
20 DirectoryService 0x0005de5c Xapicall + 452
21 DirectoryService 0x0005da80 DirectoryServiceMIG_server + 164
22 DirectoryService 0x00024330 dsmig demux_notify(mach_msg_headert*, mach msg_headert*) + 96
23 libSystem.B.dylib 0x93ff209c mach msgserver + 364
24 DirectoryService 0x00021130 CMigHandlerThread::ThreadMain() + 312
25 ...ectoryServiceCore.Framework 0x0015a09c DSCThread::Run() + 52
26 ...ectoryServiceCore.Framework 0x0015a2f4 DSLThread::_RunWrapper(void*) + 96
27 libSystem.B.dylib 0x93faabf8 pthreadstart + 316

Xserve's, misc. Mac models, Mac OS X (10.4.10)

Posted on Nov 25, 2007 6:58 PM

74 replies

Feb 11, 2008 10:34 PM in response to Jason Tallman

Because I had no idea that 10.5 was such a badly-tested heap of bugs and was indeed enticed by Apple's suggestions to the contrary 😟

Anyway I see 10.5.2 is available today so I guess I am spending my time on yet another re-install.

Thanks

Feb 12, 2008 8:42 AM in response to James Knight3

10.5.2 did not fix the Directory Service crash issue on our server. Nor did it fix the issue with new connections being "locked out" after the crash.

The least painful workaround (for me, a non-techie) is to have Server Admin running, click on the AFP green bubble, click on the Settings tab, change Authentication from Any Method to Standard, save, change Authentication back to Any Method and save again.

This seems to get the various services talking again and does not interrupt other users.

Feb 12, 2008 6:26 PM in response to Phil Houtz

Phil,

How many users do you have on your open directory server? We are seeing the same problem with 3500+ users. So far I it has been very sporadic sometimes going for 2-3 days before directory service crashes.

Jason

Feb 12, 2008 9:06 PM in response to Jason Tallman

This is just a workgroup server with 22 AFP users and a half dozen Windows clients with guest access through SMB.

I'm having the Directory Services crash at least once per hour. It appears that disabling sleep mode on the clients might be a partial workaround inasmuch as they will stay connected. But it's no way to run a railroad.

Some of the crashes we've had appear to be related to Time Machine. Another culprit might be authentication on the wiki server, but I'm not sure about that.

BASD IT

Level 1

0 points

Feb 19, 2008 11:25 AM in response to Wolfman

I am experiencing the same thing. OD is tanking quite often.
I called AppleCare and "Rodney" recommended setting up a cron job to kill AFS. Apparently, either watchdog relaunches it, or it gets pulled up some other way. Here is what I'm using:

sudo killall -HUP AppleFileServer

My /etc/crontab looks like this:

# minute hour mday month wday who command
*/2 * * * * root sudo killall -HUP AppleFileServer

This kills it every other minute. Things have been working alright so far... I just did it this morning.

Feb 20, 2008 5:30 AM in response to Phil Houtz

I'm having the same problem described in this thread with a fresh Workgroup installation upgraded to 10.5.2 on a Mac Pro quad core. The only work around I have found is to restart the entire machine. What I have found is that all checked services for users are available when I first launch the machine, but a while later all services for each user are suddenly unchecked (presumably this occurs when the OD service goes down).

I tried Phil's advice, but it doesn't get my services back up (also, I tried stopping the AFP service and re-starting it, but that didn't help either). I've only got 13 users and I freshly installed from a 10.5 DVD and upgraded to 10.5.2... I don't think I had this problem with 10.5.0.
What's unfortunate is my courtesy Apple support of 90 days has ended since I purchased this. I think I'll be relying on the "Discussions" to help sort this out.

Feb 21, 2008 10:58 AM in response to Phil Houtz

We're not running Time Machine at all on our systems, and we are also having the crashes.

Looking through the logs, it doesn't appear to matter what service is requesting a directory lookup, but basically the crashes always coincide with login password requests. Doesn't matter whether it is a kerberos or hashed password either.

As I discussed in the File Sharing forum, I just completed rebuilding the server from the ground up with absolutely no imported data, and with a completely fresh OD have reached the same problem in a matter of hours. I don't know if one of the services is perhaps corrupting directory data, but that would be my first guess.

The fact that it always crashes during directory access reads/writes is concerning, as that could undoubtedly lead to more and more corruption... perhaps that is why the crashes become more and more frequent as the installation ages...

BASD IT

Level 1

0 points

Feb 22, 2008 9:54 AM in response to BASD IT

Server is intermittently locking me out ENTIRELY now. I can't even log in with a KVM at the console. It appears that ALL authentication stops working. Cold reset only fix I could think of. gulp

Feb 22, 2008 2:45 PM in response to BASD IT

BASD IT,

I'm having the same problem (when OD goes down, I get locked out of all services). Did you ever contact Apple's support when this started to occur? I wonder if there's a workaround for this, other than restarting. This is a real deal-breaker for me.

BASD IT

Level 1

0 points

Feb 26, 2008 11:01 AM in response to Bradford Schwie

Yes, I have contacted AppleCare. They told me that it was "currently being researched by Apple's engineering team" and that they "have not provided an estimated solution date".

After contacting a local Apple Engineer It was strongly recommended that I did not run OD on the same server as AFP in 10.5.2 and lower. "OD is rock solid until you run AFP on the same server". He noted that it was "widespread" and recommended two options: Either load Tiger server on one machine with AFP and authenticate to your Leopard OD, OR go back to Tiger.

Being that we have 4200+ users authenticating against this server, I've chosen the latter solution for now. I'm eagerly anticipating a fix for this, just like everyone else.

wind out of my sails

ARG!

Feb 26, 2008 12:37 PM in response to BASD IT

Wow. I'm shocked this is the response Apple gave to you. One of the main reasons why I got Leopard Server was to be able to offer Time Machine backups (via AFP) to the Mac users in our office. It looks like I would have been better off running a plain vanilla copy of Leopard Client (this way, atleast someone could authenticate and do their backup).

I chose to get Leopard Server because I wanted to offer VPN access with a convenient interface. Again, I might have been better off by hacking to enable the nested VPN server in the client version of Leopard. Atleast the AFP functionality would work!

Anyway, I hope more people call in about this to escalate the issue.

Feb 26, 2008 9:12 PM in response to BASD IT

I also called AppleCare. Got the same cron job fix recommended. This has helped some, but I'm still experiencing a daily crash of directory services. Now it's up again after two minutes because of the cron job, but in the mean time I have 10 to 15 network users who may or may not have lost their current work, and multiple voicemail complaints to deal with.

This problem coupled with all of the System Image Utility problems I had with 10.5 and 10.5.1 really makes me want scream at apple. Don't release stuff until it's ready. By the way, if your not comfortable setting up a cron tab in UNIX, you can use Cronnix which offers a nice GUI experience. Just be sure to create the cron tab as a root user.

BASD IT

Level 1

0 points

Feb 27, 2008 5:46 AM in response to Bradford Schwie

Bradford,
Sorry, I'm not sure I understand the Leopard Client VPN server thing, but these links may help.

http://www.imulus.net/george/software/using-leopard-time-machine-to-backup-of-a- network/
http://forums.macrumors.com/showpost.php?p=4446263&postcount=126

You can Time-Machine-Backup to most any AFP share. Just run the following command in Terminal.

defaults write com.apple.systempreferences TMShowUnsupportedNetworkVolumes 1

I don't think Leopard Server changes the VPN interface much at all. It's almost verbatim to Tiger's in Server Admin.

I guess I'm not very surprised we have not yet seen a fix for this issue. Tiger server's AFP implementation wasn't rock solid until about 10.4.8, then again, you think they'd have learned from their mistakes...

Good Luck, hope this info helped.

BASD IT

Level 1

0 points

Feb 27, 2008 5:59 AM in response to Alexander Hosmer

Hmm, if your users are loosing data you may have a bigger problem.
Between what I was told by the AppleCare Technician, my experience with DirectoryService crashing, and killing the AFS daemon via cron job I've never had a user loose data. From what I understand, this issue simply locks people out and leaves the current connections connected.
Maybe it has something to do with "idle" users trying to reconnect?

I'd be right next to you screaming @ apple dude. 😉

Feb 27, 2008 1:45 PM in response to BASD IT

BASD,

I never had Tiger Server, so thats why moving to Leopard Server and getting an interface for the VPN server was attractive to me.

Using an AFP volume over a network only works reliably if the network share supports ACL's (Leopard yes, Tiger no). I would be wary of relying on a backup solution with the command you recommended based on the warning here:

<http://www.macosxhints.com/article.php?story=2007121706402296>

It may work for a while, but as soon as Time Machine needs to start re-using space on the sparsebundle, you will run into issues.

Brad

Directory Service Crashing