afp server issue - very hign cpu load

I have been quitting the AFP process whenever it looks like it is hung, but when I do all my users get messages saying mailbox is locked and they need to quit and rebuild. Everything is fine when they do quit, but I just thought I'd share so people arn't surprised if it happens

We are having exact same problems with AFP. Apple Premium Support wasn't able to help us with this. Our case has been (reportedly) escalated and I'm expecting a call from "engineering" within 24 hours. Our school came to a halt because of this. I can't believe Apple hasn't been able to fix a known issue like this. I'm very disappointed.

I have tried wiping the server and running non-server 10.5.6 and AFP file sharing and the problem has still happened.

Before that i had tried a reinstall of server using the standard or simple setup and the problem also happened in the first few hours. we did also have directory service using a full CPU too, during any copy to the server.

Just an update on my situation...

In regards to the ticket I have open with Apple, they indicated yesterday that
Engineering had identified the cause and is testing a solution, however the testing can take several weeks. I eagerly wait in anticipation.

As I have exhausted all other possible resolution paths while the problem persists, I have implemented a cronjob to test for AppleFileServer CPU usage and kill it if the usage is over 99%. Again, this is only on 1 of my 3 Xserves. The other 2 Xserves have been merrily processing and serving data with absolutely no problems.

Cheers

Can you share the script?

Would you mind sharing your cronjob with us? I've written a few in my day.. but I would like to make sure I'm not missing something with this one. Much appreciated!

I looked thru my access logs, around the time of the problem some files have an abberation. the access logs look a lot like this:

"OpenFork FILENAME" 0 0 0

but near two of the failures i watched, i got
"OpenFork FILENAME" -5006 0 0

what is that -5006?

also during failures, half the time, its saving a png out of illustrator directly on the server - not always but it seems to be the most likely thing to start the problem

I'm no scripting guru so it's a pretty basic script cron'd to run 3 times a day...

#!/bin/sh
ps -Av > /tmp/AFPps
PROCESS=`grep AppleFileServer /tmp/AFPps | awk '{print $1}'`
CPU=`grep AppleFileServer /tmp/AFPps | awk '{print $11}'`
COUNT=`echo $CPU | wc -c`
#
if [ COUNT -gt 5 ]
then
kill -9 $PROCESS
fi



So since $CPU is basically a character string, so if this field is > 5 characters, it indicates the cpu utilization is > 99%.

today server load was again above 7.x AppleFileServer using more than 300, so the fixed ethernet/disbale ipv6 dont seem to be the solution. users did'nt complain too much. restarted whole server. (reboot)
implementing the kill-AFP script tomorrow... and w'll wait (again) to see whats going to happen and when Apple comes up with final solution.
found a remarkable user called spotlight in the acl on some folders from shared volumes removed it, user spotlight does not exist.... (not on OD not local)
what the exact difference is with today and monday/friday don't know, getting tiered, need sleep. bye

I killed -9 AppleFileServer process today while I was observing 15 connected (AFP home folders etc) users. Some received the message about server unexpectedly disconnecting and some users did not see it at all.

by the way... just noticed the formatting of my posting changed once I posted.

The line that indicates if COUNT -gt 5 should read as...
if <square left bracket><space>$COUNT -gt 5<space><square right bracket>

I've modified my approach, with good results.

Turning off Spotlight on the clients solved my AppleFileServer issues, but crippled the clients because of the inability to search.

I originally did not have my homedir sharepoints enabled for Spotlight searching. After searching the web a bit, it clearly is preferable to have the server do the indexing work vis-a-vis the clients as it does not impact the network or stress the AppleFileServer service.

So I did the following:

1) I disabled indexing of client local hard drives by issuing the UNIX command 'mdutil -i off /Volumes/<My Hard Drive Name>' via remote desktop to all my managed clients
2) I enabled spotlight searching for all my automount sharepoints (homedirs and group folders) on the server. This triggered an index process on the server which ran for quite a while.
3) I re-enabled Spotlight and the metadata indexing on the clients.

When any of my network users logged in for the first time, Spotlight ran to index their content. Interestingly though, the clients seemed to launch a server-side indexer which did not choke up AppleFileServer. While indexing was running, my CPU was very busy, but the AppleFileServer process was not the one occupying the CPU.

Once indexing completed for a network user, subsequent logins on the same or different machine did not trigger an index rebuild. When I list open Spotlight files on the server using the command 'sudo lsof | grep Spot' on the server, there are many open Spotlight index files (>>100) but the local mds process is the one that has them open. I only find a few (6-10) Spotlight files open through AppleFileServer.

Now that the initial indexing is all completed, my system continues to run smoothly with free CPU. Clients also have the ability to use Spotlight to quickly search their home directories.

Randy

Update on our situation:
The CPU usage problem is gone! (for 30 plus hours at least) Dont get your hopes up thou, i think i just have not introduced the CPU killing variable yet 😟
Here is what I did:
I have reinstalled a few times as mentioned above and the problem was always present in a matter of an hour or two. This time i reinstalled 10.5.4 off the DVD in my iMac C2D, I chose the advanced install (not the simple or standard) on boot, I DID NOT run software update. I did set all drives spotlight private, turned of SSH remote control and added ARA to one user with reports turned off. Went to Server admin, turned on AFP and ONLY AFP, authentication standard. Did not touch a single other setting in any other service - so my server is running ONLY AFP. Added my users and groups, assigned the shares permissions, not using ACL, but permission access by the group. promogatled permissions and then turned on sharing. Disable spotlight searching, set all the protocol options off but share this with AFP.
Did not touch the Terminal at all, no DNS, no nothing.... and its working.

This was a simple/minimalist setup as i can do, so this makes me think its another variable in play. I usually set the DNS, turn on a SMB with a sharepoint, run software update, so i will set thru this until it 'breaks' and voila, if i can replicate it, it means i have identified the culprit.

theory now is its either updating to 10.5.6 or some other update in software update panel, or SMB file sharing, or the Time machine backup drive thats not running 😮

Message was edited by: Eric Hilferding

I was having the same problem as all here. As soon as I updated to 10.5.5 the CPU usage on the server shot up to 100% every time specific users would log on multiple times. I'm in a k-9 school and don't what to have to manage separate accounts for the elementary students. Whenever a k-6 account was logged onto multiple times my CPU would spike. After a couple months researching, reading this post and anything else I could I realized my own account could be logged onto all our machines (about 100) at the same time and have absolutely no affect on the server. The only difference is my account is set to mobile. I know it may be a simplistic work around, but it's working for me. I set the elementary accounts to be mobile accounts with syncing off, created a sharepoint for each grade level and have that shared drive on the kids' dock and desktop. My thinking was it was AFP causing all the problems when I have one user logged on many times. By setting the accounts up as mobile, their home folder local, and syncing off it wouldn't be using afp at all until users saved to the shared drive. Even then the use in minimal. It's been working for me now for about two weeks.

One thing I could use help with is redirecting the clients' documents folder to the shared network folder. Students have been trained to save in their documents folder and getting them to use the network folder is hit and miss.

MacSpecialistGermany wrote:
The funny thing is, that I am able to kill AppleShare on the Server in ActivityMonitor without my users getting disconnected. AppleShare just gets restarted imidiately and is back to normal CPU usage (until next time).

Be wary of doing this - if a user has a file open that resides on the server it will cause issues.