10.6.3 frequent crashes

MattMPS:

I observed something weird going on related to slapd. I found the following code in the com.apple.PasswordService.plist:

<key>WatchPath</key>
<string>/var/run/slapd.pid</string>

I did not put it there and was/am concerned about some sort of security problem. I am wondering if the last security update did something to this file. Note that there is an error (WatchPath should be WatchPaths) in the plist that generates a console message after launchd loads during boot. Hard for me to see how this is generating hundreds of threads.

Very interesting find ! I'm seeing an identical entry for that plist on a 10.6.3 server here.

I looked, and there doesn't appear to be any change in launchd in 10.6.3 with regards to WatchPaths

http://www.opensource.apple.com/release/mac-os-x-1063/

the instances found in the source code there, still all use WatchPaths

Note that as it exists, the "WatchPath" parameter is probably ignored.

My limited understanding suggests that launchd launches /usr/sbin/PasswordService when the /var/run/slapd.pid file is created or changed. This implies that launchd was intended to delay the PasswordService until OpenLDAP is up and running. Is this behaviour something to be concerned about?

Because of the typo the delayed launch is not actually what is happening. Also note the dates of the plist and PasswordService files are Mar 8 2010 22:25 and Feb 11 2010 00:06 respectively (SLS 10.6.3). Could someone please verify the dates of these two files on their installation?

Two days and counting. No crashes. two users without access to shares on Day 1 fixed overnight and can login. Used WGM remotely a couple of times today for a few of minutes to change passwords, add a new user and add some computers to a computer group. Have noticed the possibility of some DNS deafness for short periods of time, but other than that all good.
Still staying away from Server admin. I have been aware of not using SA and WGM on servers and for any length of time but have never had it crash any other server version- just program instability. It does seem strange not to use it for any length of time when it has graphing features to show server and AFP use. Why have this tool if it not to be used for examining the server for debugging purposes.

Same here.
Over the long weekend, the servers ran five days without any problem whatsoever.
I am now quite certain to claim that SA is indeed at least the trigger to the issue.

It is very unlikely that this thing should be related to a hardware issue.
How come, the servers worked fine with 10.6.2?
Btw using ssh keypair login does not work in case the server hangs. It does not respond.
I too have heard the theory that you should not leave SA open all the time.
But if you use it often, then it makes sense to leave it open.
And I expect such a tool from Apple to allow me to do this.
And sometimes I need to follow certain log files and I used to be able to do so in serveradmin.
If that should not be possible, then we have a serious quality issue with SA.
And a lot of monitoring tools speak to 'servermgrd' as well. They might even be causing the same effect.
@ davidh: Are you somehow related to Apple?

@qmp: I've been working with OS X Server since it first existed, and have achieved the ACSA and/or ACTC certifications for some number of server versions. Also worked for years as an Apple-certified (hardware) tech -but for an independent (Apple-authorized) service provider.

My current company is a member of the Apple Consultants Network.

But if you meant am I an Apple employee, no I am not. Keep in mind that this forum is intended for user-to-user support.

The point about hardware issues is that you (one) needs to understand that any software change (data, OS) can prompt a latent issue to manifest where it had not (not been seen or noticed) previously.

It's very important to understand that I'm not saying that is what's going on. I am saying, the first wrong move in effectively troubleshooting a problem, is to turn a blind eye to certain factors because it's not what you/we want. Be methodical, be logical, and accept that someone will always know more than you (I'm talking about life in general), and there is always more to know.

Sometimes the perspectives offered here by others are not meant to provide The Answer 😉
as there isn't one. IE, one size does not fit all. Similar seeming symptoms may not have similar underlying causes at all.

I don't mean to say that there is "not" a problem with 10.6.3 for some people in some specific scenarios, but I do mean to say that demonizing 10.6.3 is neither practical, meaningful nor helpful to anyone. Why am I not seeing the problem with the 10.6 servers I maintain ?
That's an interesting question.

I can tell you that as a matter of policy, I use Server Admin or Workgroup Manager when I need to, and then quit out of them when done.
For watching logs I use the Terminal ( tail -f /var/log/logfilename.log or tail -f /Library/Logs/<path to log> ) and occasionally the Console app (in Utilities).
But I do greatly prefer using the Terminal to look through log files ( less, tail, grep + awk ).

If it's not possible to log in via ssh keypair, then I would probably keep an active Terminal session open on the server itself.

It is now a week since I have used the server without using WGM and Server admin on the 10.6.3. Server. I have not had one crash during school time since I did this. I have been using Server admin and WGM for occasional updates from my computer but mostly from the 10.6.2 web server which I have not updated to to 10.6.3. Occasionally I have seen issues with the users not being released by AFP - sleep/disabled. When I disconnect them, the server seems to speed up access by new users. We have a few users who can log in yesterday but not today but usually can tomorrow?? weird. Anyway as far as I can see with no change except 10.6.3 update, the use of server admin on the 10.6.3 server was causing the crashes. This was not the case on 10.6.2 as can be seen by using it on my 10.6.2 webserver.

It is now a week since I have used the server without using WGM and Server admin on the 10.6.3. Server. I have not had one crash during school time since I did this. I have been using Server admin and WGM for occasional updates from my computer but mostly from the 10.6.2 web server which I have not updated to to 10.6.3. Occasionally I have seen issues with the users not being released by AFP - sleep/disabled. When I disconnect them, the server seems to speed up access by new users. We have a few users who can log in yesterday but not today but usually can tomorrow?? weird. Anyway as far as I can see with no change except 10.6.3 update, the use of server admin on the 10.6.3 server was causing the crashes. This was not the case on 10.6.2 as can be seen by using it on my 10.6.2 webserver.

I too have experienced applefileserver crashes since the 10.6.3 update. I also have discovered one way to make it happen. If I disconnect from my shares after logging in then try and reconnect to them within a "finder window" under the "shared" computers, I can make the applefileserver crash. Obviously I get a connection failed message when trying. Interestingly this does not seem to happen when using the "connect to server" way of connecting. I'm no expert, but thought I'd throw this out and see if it is happening to anyone else.

Process: AppleFileServer [66120]
Path: /System/Library/CoreServices/AppleFileServer.app/Contents/MacOS/AppleFileServer
Identifier: AppleFileServer
Version: ??? (???)
Code Type: X86-64 (Native)
Parent Process: launchd [1]

PlugIn Path: /usr/sbin/AppleFileServer
PlugIn Identifier: AppleFileServer
PlugIn Version: ??? (???)

Date/Time: 2010-05-26 07:51:35.138 -0700
OS Version: Mac OS X Server 10.6.3 (10D573)
Report Version: 6

Exception Type: EXC BADACCESS (SIGSEGV)
Exception Codes: KERN INVALIDADDRESS at 0x0000000000000000
Crashed Thread: 124

Thread 124 Crashed:
0 AppleFileServer 0x000000010006bcf2 OSLock::ReadLock() + 42
1 AppleFileServer 0x000000010001edb8 OSLockHolder::Lock() + 28
2 AppleFileServer 0x0000000100062df1 FPSession::IsMountedAsSP(OSRef<FPSharePoint>, OSRef<FPDir>) + 239
3 AppleFileServer 0x00000001000571a6 FPCNode::GetUserUnixAccessSummary(unsigned int, OSRef<FPSharePoint>) + 252
4 AppleFileServer 0x00000001000573ac FPCNode::GetUserRightsSummary(unsigned int, unsigned int*, OSRef<FPSharePoint>) + 96
5 AppleFileServer 0x0000000100060976 FPSession::UserIDCanAccessShare(unsigned int, OSRef<FPSharePoint>) + 190
6 AppleFileServer 0x00000001000632a7 FPSession::OpenSharePoint(FPName&, OSRef<FPSharePoint>&) + 483
7 AppleFileServer 0x000000010000d69d AFPRequest::OpenVol(FPOpenVol*) + 109
8 AppleFileServer 0x00000001000187a3 AFPRequest::DoCaseStatement() + 1437
9 AppleFileServer 0x0000000100018a4b AFPRequest::DoRequest() + 37
10 AppleFileServer 0x000000010001e6a0 AFPSession::HandleRequest(OSThread*) + 1384
11 AppleFileServer 0x0000000100076749 SessionManager::HandleSessionRequest(SessionManager::SessionType, long, unsigned char) + 247
12 AppleFileServer 0x000000010007d7ab WaitThread::Entry() + 421
13 AppleFileServer 0x000000010006c160 OSThread::CallEntry(OSThread*) + 22
14 AppleFileServer 0x000000010006c1e5 OSThread::_Entry(void*) + 63
15 libSystem.B.dylib 0x00007fff8851d8b6 pthreadstart + 331
16 libSystem.B.dylib 0x00007fff8851d769 thread_start + 13

I have been monitoring this thread for quite a while. I am having the same server crashing problems as many of you.

JEEKR, when you say you were able to make it happen, Are you logged in to your machine as a local user and then accessing you home directory or are you logged in as a network user or a mobile user? If logged in as a mobile or network user, what auto-mounts do you have setup? I'm just trying to cause the crash as well. I am getting at least 1 crash a day but have not been able to determine the cause. We REALLY need to get to the bottom of this.

Yesterday I experimented a little and believe I've determined that I can cause a crash by trying to log in as administrator (the admin local user account of server) from networked computer irregardless of whether or not I'm logged in as local user or a network user on that computer. Both machines are 10.6.3. Again this only seems to occur when connecting through the Finder window interface. It also does not always cause the AFP file service to go crazy with multiplication of AFP connections which in the past requires a restart of AFP or the server itself. For example, after causing a crash similar to the one I posted earlier I was able to use the "connect to server" command and log in to my server using the admin credentials with no indication of another crash. I do have my server set to show volumes when connecting to it as admin rather than sharepoints. I also must say my xserve started as a 10.4 machine and has had nothing but upgrades to get to 10.6.3 so it is probably due for a rebuild.

I finally have proof of the problem and know what causes 10.6.3 server to crash, it is DirectoryServices. I have also gotten an Apple engineer to admit that there is a DirectoryServices bug in 10.6.3 server. The thread count will clime as more Services access DirectoryServices till the server is useless. to fix just kill DirectoryServices and it will restart it self, the server will be all good for a while.

See pic from my phone of crashed server
http://www.freeimagehosting.net/uploads/a5ef32e799.jpg

I think the problem goes beyond the server edition of 10.6.3 ...

http://discussions.apple.com/thread.jspa?threadID=2383288&tstart=0&messageID=116 06056#11606056

I have been running crash free for a week now.
Here is what I did:

Went to System Preferences
Clicked Accounts
Clicked Login Options
Unlocked to make changes
Clicked Edit for Network Account Server
Clicked Open Directory Utility
Unchecked LDAPv3

Now when ever I need to make changes or add users I just enable the connection to the LDAPv3 local connection for Workgroup Manager

I "think" the problem has something to do with local kerberos connections and reverse DNS

My problem might not be your problem.

This has only helped me to use openLDAP services without it crashing, clients are able to login without LDAP crashing in a few minutes to a few hours, I have not tried any extensive logins like a mac or windows computer client network login.