afp server issue - very hign cpu load

Everyone, I opened a support ticket with Apple a couple of weeks ago. We have completed the following steps towards trying to resolve this issue, however my problem continues...

- Disabled auto-disconnect in AFP after idle time.
- Removed spotlight indexing on all afp volumes and deleted .SpotLight-V100 directories on AFP volumes.
- Verified that Host Cache Flushing is disabled on external RAID array.
- Set the following default: defaults write com.apple.desktopservices DSDontWriteNetworkStores true. Set as preference for all groups.
- Disabled kerberos for AFP authentication.
- To check the maxThreads setting for AFP, I tried to run the command defaults read com.apple.AppleFileServer maxThread, but maxThreads does not exist for com.apple.AppleFileServer.
- Changed the fibre topology to Point to Point for all 4 fiber connections to the Promise VTrak array.
- Stopped spotlight indexing by using the command: touch /Volumes/Sharename/.metadata neverindex
- Renamed odpac.bundle in /System/Library/KerberosPlugins/KerberosAuthDataPlugins/ to odpac.bundle_DISABLED

Even after all this and rebooting on Friday morning, I currently have 3 cores @ 100% and AFS process @ 300%. I just killed the AppleFileServer process and I am back to normal operation (did not reboot)

I believe you can try all the solutions I have mentioned above, however my thoughts are that there is a problem with the AFP (AppleFileServer) binary itself. The problem has been escalated to a backline Apple Engineer who we will be working with and testing a different version of the AFP binary.

Will keep you posted.

We did exactly the same with our Apple support, but as you can read in an earlier post, with no success. i agree, that there must be a bug in the AFP binary itself. But sorry, Apple published version 6 of 10.5 server, without solving this problem?????

We need help...

Have any of you guys upgraded your servers before the clients? See we weren't seeing any of these problems until I updated my clients to 10.5

I was reading the other day that great little article

*AFP. It ain't so bad....*
http://www.afp548.com/article.php?story=20060329213629494


any way to the point, I found something interesting...


+*1) Tweaking the WAN threshold and packet size on the clients*+

..."The problem seems to be that this default threshold setting is way too low, and once the AFP server starts experiencing moderate load, your LAN clients start using the WAN data chunk size. Although smaller chunks are desirable for slow connections, they induce an overhead on the server in terms of processing as the server is dealing with 16x the number of chunks, and reduce overall throughput. *A symptom of this is high CPU usage.*"...


Interesting... so what are my clients doing then??

Lets check ...

+(apple moved it to com.apple.AppleSareClient it was com.apple.AppleShareClientCore)+

defaults read /Library/Preferences/com.apple.AppleShareClient

{
"afp activetimeout" = 0;
"afp debuglevel" = 6;
"afp debugsyslog" = 1;
"afp_forceKerberosOnly" = 0;
"afp host_prefsversion" = 5;
"afp idletimeout" = 0;
"afp immutabledirs" = 1;
"afp_maxDirCache" = 60;
"afp_maxFileCache" = 60;
"afp_minDirCache" = 5;
"afp_minFileCache" = 5;
"afp mountdefaultFlags" = 0;
"afp nocaseSensitiveVols" = 0;
"afp nokQueues" = 0;
"afp no_volChangecaching" = 1;
"afp reconnectallow" = 1;
"afp reconnectinterval" = 10;
"afp reconnect_maxtime" = 600;
"afp softautomountvols" = 0;
"afp softmountvols" = 0;
"afp wanquantum" = 0;
"afp wanthreshold" = 0;
}


"afp wanquantum" = 0; and "afp wanthreshold" = 0; are zero??


So now what I am going to do, I am going to update my wan_quantum and wan_threshold to the optimized settings

afp wanquantum = 131072
afp wanthreshold = 1000

I am going to do the labs first and we will see if there is any more CPU craziness happening...

Cheers

Following on from mine, today the AppleFileServer is again running High, but this time is isn't preventing others from working, it is constantly jumping around between 80 -120% indicating to me that it is working harder than I'd expect but it isn't jammed up.

Before in Activity monitor the CPU graph would be full of green squares, at the moment the green is high but is moving.

I can live with Higher CPU usage as long as it doesn't impact on users ability to do their work, but it would be nice to get it back down to 30-40% that I'm used too.

What are you guys seeing in activity monitor?

I find that AFP might be on its way to crash, when you have it hogging CPU and it then never goes back down. Sometimes here we have 100% then 400% then even a bit more or crash

Hope that helps

Message was edited by: Rob@Bis

Rob,
Interesting idea
Can't wait to hear what you have to report after testing.
Our school is having the same problem as the rest.
I saw something today that hasn't been mentioned (I don't think) top-u shows our afp over 100% like everyone else here but it also shows mds over 100%. I think mds is related to spotlight so I'm off to look at spotlight issues following the lead of those posting above.

Jim Pattison

I agree, it seems spotlight is involved in at least some of our high CPU. trashing the users .Spotlight-V100 folders does sometime help us, but the problem returns

Good to hear that there is someone at Apple working on this, even though their previous efforts seem to have come up short.

I am managing the computers in a very small office (at most 12 or so concurrent users). All are on new Core 2 Duo iMacs running 10.5.6 connected to a Leopard 10.5.6 2 x Dual Core Xeon server. We are using network home directories on a RAID 5 set of three drives.

The AFP CPU pinning issue was there in 10.5.5. and earlier, but was manageable (it happened once or twice a month) and even while AFP was pegged at 100%, it did not rise above that level so there were essentially 3 other cores available for the people to use to do their work. So work got done, most people did not notice the slowdown and my phone did not ring. That meant that I could wait for the end of the day to either restart AFP or the server and hope that it did not crop up the next day.
It seems to me in reading this thread that there are at least two distinct problems with AFP. One is where all the CPU of all the cores is eaten up by AFP and the entire system grinds to a halt and the other more manageable problem where AFP takes up one core (ie 100%) and allows the other cores to do their business.

The other thing from reading this thread that is not completely clear; can anyone replicate this problem without resorting to starting up one of MS Office 2004 or 2008 applications?

Using or starting Microsoft Office 2004 or 2008 does not affect the problem. it doesn't matter which applications are running.

Saw this issue for the first time today on our 4 core Xserve running 10.5.4 - one cpu was maxed out and server access was hampered all over the facility. At the time I felt the situation had been caused by a user logged in from a Autodesk Smoke video editing system. This is a Linux based system. He was performing a file move operation. After reading posts here I no longer think this is the case. Ours is a basic file server system with a couple of Xserve RAIDs on it. No DNS or anything like Open Directory - just a plain old Xserve with with Sharepoint that everyone accesses with a common login. AFP and SAMBA both are served and I have a mixed bag of OSs and computers on the network. Reading the posts here and elsewhere it seems that no one really knows what causes this problem. I could run around deleting Office prefs and changing settings, but it seems like a waste if it doesn't help.

Am I right to do nothing and just wait for a fix?

Please don't apple bash to much, we don't want this thread deleted. 🙂

What version of OS X do you have installed on the clients?

I have to say for the last day now, logins, and all around speed here have been good 🙂

I would like to also note that the CPU usage has been spiking when there were a lot of logins, but then going back down when the users were logged in.

I would like more users here to please test one lab or group of computers if they are 10.5, and change the

afp wanquantum = 131072
afp wanthreshold = 1000


I need more testers!! I need more feed back.


So if anyone with these crazy AFP issues, please try to change the client configurations.

Today was always a very high usage day, where we would have a lot of computers being used and AFP would die.

from one server

http://farm4.static.flickr.com/3366/31932000394bda7af37ed.jpg

As you can see we have over 70 users. Logins would crawl because of AFP would be taking everything from the CPU

http://farm4.static.flickr.com/3110/31931999893d7698dc39d.jpg

As you can see, we did have a few CPU spikes, people logging in, opening up what ever they had for an application, but it didn't die, AFP never stayed at the max.


The thing is, I need more people testing, please have a look on you clients,

if your clients are running 10.5 please run the following command

defaults read /Library/Preferences/com.apple.AppleShareClient

post results, mine here were

afp wanquantum = 0
afp wanthreshold = 0


Then update them to

afp wanquantum = 131072
afp wanthreshold = 1000

you can either do that with the defaults write command or you can open the plist file with a plist editor , edit the file and then send the plist file via remote desktop and over write any others that are there.

Leave everything else in the plist file alone, just update the afp wanquantum and afp wanthreshold.

Once you have post results eh!

Cheers everyone, things seem to be going well here, but I need more feedback!

If anyone needs any help doing this please post in the forum, I am checking it all the time.

hi there,

there is a new blog only for this issue, please visit

http://www.go2xserve.com

register and discuss the AFP problem in this blog.

we need your help, please keep posting

I see no reason to take this discussion off the official apple boards?

We all need help resolving this and apple are more likely to digest what is written here than on your private blog.

Re. another posters' caution against 'apple bashing' getting this thread deleted, pointing out serious bugs and stating intentions to move to an alternate product are hardly capital offences, even to Apple.

I couldn't agree more. We need help! I assume that they are working on a fix. I hope they can sleep at night because I'm not sleeping at all worried that I'll come in to my job tomorrow and my main Production server will be toast because of this issue. I work for a newspaper. An event like that would put me out of a job (whether it was my fault or not). It would be nice if they'd post some sort of response here is all I'm saying. I am hoping that the next update will address this.