afp server issue - very hign cpu load

hey eric, my problems with AppleFileServer only started with 10.5.6 as well, and for the first week this year it was jumping up to 180% and not releasing. But in my case after trashing users spotlight folder, running cocktail on the server, running diskwarrior on the server boot drive and on the RAID it is now a little more stable, I'm stil getting higher than normal CPU and users are reporting RAID disconnected warning messages which disappear straight away, but they are able to carry on working and I haven't had the AppleFileServer lock up this week.

The disconnection messages are a bit worrying but that only happens for the first few hours some days.

Does anyone know if it would be safe to grab the AppleFileServer from 10.5.5 which was working fine for me and use it in 10.5.6 or is that asking for trouble?

Well I found the corresponding article from Apple explaining that the two 0 settings actually dont mean 0. They mean that AFP should use the standart settings for different OS versions and situations.
Only in a few special situations changing them could be beneficial. Maybe this problem is one of them.

http://support.apple.com/kb/TA24270?viewlocale=en_US

In that artice Apple says that it might help AS if one would allow more than the 200 server threds it has on default.

The funny thing is, that I am able to kill AppleShare on the Server in ActivityMonitor without my users getting disconnected. AppleShare just gets restarted imidiately and is back to normal CPU usage (until next time).

But the excessive CPU usage does not slow down my users or networktraffic much.

MacSpecialistGermany wrote:

But the excessive CPU usage does not slow down my users or networktraffic much.

And this is strange. Some of my customers experience a major slowdown when AFP starts to ramp up (spinning wheel on the clients, everything takes much longer than normal to complete) and some of my customers don't notice any difference, nor is there any difference in performance when I actually test copies over AFP. I've had servers pegged out with 0% cpu available for weeks and performance hasn't suffered, whereas some customers restart almost daily because it becomes unusable.

and what about your subnet's, are clients and server an the same subnet, or not ?

Hello Michael,

are you allowed to distribute this binary ?
and are your server & clients on the same subnet ? or is there some routing in your network ?

tanx in advance for the reply, kind regards, Erik

Erik, we implemented a different AFP binary last Monday. I do not believe this is a new and improved binary, however it is intended to capture detailed information on the AppleFileServer process when it spikes to 100% (or greater). When it did spike again last Thursday, we "sampled" the process and sent to Apple.

Their reply was basically that they confirmed that it is the same problem that they are already investigating. They said that "Currently, there is no work around" and had us go back to the original AFP binary.

As I indicated previously, I strongly believe that you can apply all the corrective measures you want, however none will specifically resolve the AFP issue. It is a problem with the AFP binary itself and is basically out of our hands. I sincerely hope that Apple realizes the severity of this problem and make it top priority to resolve.

Although I shouldn't have to resort to this measure, I am going to cron a job to sample the AppleFileServer process and if > 99% for more that 5 minutes, kill it. Killing the process does not appear affect the end-user or other AFP operations as I have done this many times manually.

Just to let you know, I have had a single core pegged since Friday. My other 2 servers (OD Master and another OD slave) have never had an issue.

Cheers

what are you using for authentication for your share points?

Should be set to standard, not any method or kerberos. Just double check.

All the clients and the server are on the same subnet

An update to our problem.

setting:

afp wanquantum = 131072 and afp wanthreshold = 10000

Seems to not only have solved our CPU spike problems, we are seeing 70% faster throughput from Client -> Server, etc. Previously only our backup server was able to spike to above 80MB/s and now we are seeing clients spiking to that level as well!

Sorry to hear others are still having trouble with this.. We seem to have been spared when making all of those changes.

sullrich wrote:
An update to our problem.

setting:

afp wanquantum = 131072 and afp wanthreshold = 10000

Hope it was a typo afp wanthreshold = 10000 is pretty high, specially when it should be around 1000

Cheers
Rob

so we still have'nt found the common setting what makes our servers fail/spike...
it seems impossible to me that all 10.5.6 server have "our" problems. i can understand that if you don't have problems you dont find/use this apple discussion...

we have xserve2,1 single cpu quad-core, production mid/end-2008, 4x1 Gb RAM, 2 x 80 Gb software-mirror boot.

STANDARD server installed from 10.5.1, combo update to 10.5.6
checked re-checked DNS, perfectly
atto fc-41es (latest firmware&drivers) connected to an easyRAID 5 TB
server ip:172.10.100.251, appletalk disabled (subnetmask 255.255.255.0)
clients-ip 172.10.107.x (so different subnet)
extra sw installed on server:
- chicken vnc
- retrospect client sw 6.2.234
- firefox 3.05
- adobe reader 9
Apple remote managment, enabled for localadmin (so not screensharing)

about 3 "10.5.4" clients, and about 10 "10.4.11" clients, wich ran stable and perfectly to a G5 10.4.x OS X Server
clients boot from local disk, mount a AFP share
clients do a lot of find commands (10.5 & 10.4 spotlight / Houdah & easyfind)
clients open mostly illustrator doc's directly from afp-share, work and save/print, whatever

ran ok first 3 production day's, than load of this system went up-and-up-and-up.
last friday load was up to 15.x and even 16.x this seems skyhigh to me. AppleFileServer over 300% cpu (threads was standard at 200, taking 202)
set AppleFileServer threads to 400, taking 402.

this evening after 1 production day after last reboot, load 4.x (command "w" from terminal, also seen in top), which seems like a lot to me again. AppleFileServer cpu was now about 102%
afp-service could not be stopped, so rebooted xserve again. Hopefully production goes ok tomorrow again.

PLZ compare, and lets find the differances/lookalikes so we can Solve this Discussion.

tanx in advance, kind regards from the NL

Hello Michael, tanx for your clear awnsers, i agree, Apple has problems with the current server release, but you two others wich seem to be stable, why/how, that's the question that's killing me, so therefore at the bottum our server setup.... let's compare !
tanx in advance, kind regards, Erik

mine as been running well again for the last 3 days but is now showing signs of problems.

It is a dual processor and AFP is using 130% and i've just noticed that it is alternating between the processors, so one will be filling all the blue lights then drop down while the other processor fills all the lights. Getting near the end of the day so I will just wait for ppl to log off and then kill AFP

Kevin, you should be able to kill AppleFileServer without any apparent affect to the end-users. We just killed it on our server during production as we had 2 of 8 cores pegged and AFP @ 200%.

Cheers

One thing we notice when we have our spikes is that our memory indicates nearly all memory is consumed. For example, this morning with 2 cores pegged and AFP @ 200%, I only have 40MB "Free" while there is almost 7GB consumed by "Inactive" processes. This memory consumption remains at this level even after killing AppleFileServer.

My other servers, which have no problems and an uptime for weeks, indicate the exact opposite: approx 7GB "Free" and approx 100MB "Inactive".

Will check further into this.

Cheers