afp server issue - very hign cpu load

Dear all,

We have been experiencing this for a lot of time now.

Today we did an experiment: Kill Firefox on all Macs.

The load suddenly went down to 4-5% on AppleFileServer with 40 Users.

Check if your users are using Firefox 3. People on other platforms are getting the same behaviour: https://bugzilla.mozilla.org/show_bug.cgi?id=441481

I instructed all my users to use Safari for the moment. For 10 hours, the load on the server has been fine and everything works as it should.

I hope this solves it for you, because for us, it's been a nightmare...

Replying to myself: Applecare Premium Service and support has been of NO help. They only asked to run EDC and send them the results, replying with a "Will be fixed". That's not acceptable for an Enterprise Service.

I don't think Firefox is the solution to this, 4-5% across 40 users is little difference. With 40 users logged on, we can see 4 cores pinned at 100% if we dont reboot nightly, and nearly all our users are on safari.

It sounds like that 4-5% is just users Firefox cache files going back and forth, I would suggest you redirect caches locally on clients and let the Firefox goodness continue.

This is a bug in Apple's AFP binaries, until they fix it, we're just ******* in the wind here.

If it's still bugged in Snow Leopard I guess we're all off to NFS.

I confirm that getting rid of Firefox solved this issue for us. Even 2 users running firefox gets our server to high cpu usage.

Codeus, read my message: Tt isn't a 4-5% difference: It drops *to* 4-5% load. If my users open firefox again, the load goes up to 300%.

The problem here lies on how Firefox currently handles the urlclassifier3.sqlite and other sqlite files. It seems that it perform some badass filelocking that consequently leads to a thread shortage, that makes less threads available to serve, which, we think, leads to a penalty on accesing the disk. Also, that file is being updated continuously by google servers in an incremental way.

A simple fs_usage AppleFileServer will show you quite some access to that file. Please note that even only 4 users running firefox makes our 4 cores go 100% and our clients to beachball.

We have ~/Library/Caches redirected to the local machine (which appears to be where Firefox stores its cache files) and AFP will still randomly take up full load on all cores

do your users use home directories on the server? If not why would firefox look to the server?

While we do not see the CPU or memory load of the server maxing out on the AFP process, we have witnessed the user experiencing the SBOD when using either Safari or FireFox. When this happens the server shows a high amount of disk activity (read that as IO traffic).

Usually once FireFox starts acting up and they switch to Safari the symptoms carry over.

So as a test we are going to disable the "warn about fraudulent sites" on both browsers to see if that remedies the issue. We calculate we have a very low risk by doing so.

Only network home users are affected on CRT iMacs running 10.4.11 and Safari 3 - 4 or FireFox 3.x.

Deleting the users cache contents will clear it up for a bit as well.

At least for us (10.5 Server Build 9L30). No more shooting the AFP daemon because you find it eating up all your cpu power with no users connected. Late but finally there and real!:)

Good good, have updated to 10.5.8 but is summer break so loads are light at present, anyone else found it helps?

Recently I ran into the same problem with exploding AppleFileServer CPU consumption.

Here are some thoughts and observations regarding the problem.

I don't think AplleFileServer is the cause of the problem but rather a followup. There a two observations I made that support this assumption:
1. When I kill (as root using a ssh connection) AppleFileServer it will or will not be relaunched by launchd, but anyhow the machine still isn't usable nor responsvie. So the problem obviuosly is somewhere in the underlying layers.

2. At at least 2 times i observed an increase of kernel_task (PID#0) miniutes before AppleFileServer exploding: Usually kernel_task uses about less thea 1 % CPU time but at the events I observed, it suddenly took about 100.5% and stayed like this. Depending on user activity some moments or even several minutes later AppleFileServer then gets rampant (>290% CPU usage). So first comes the underlying problem and then AppleFileServer.

Additional paths of investigation:
In another Thread ( http://discussions.apple.com/thread.jspa?messageID=8000668) I found a hint about an incombatibilty with an ATTO SCSI card. Yes we are using such a card but this very server ran fine for more than one year with that particular card. We are still investigating on this.


Maybe its RAM? We have increased RAM form 4 to 6 GB and it was after the upgrade when the problems started. (But then again we also had the OS updated) ... So a brief poll to others experincing the same problems: Do you have more 4 GB RAM ?

And finally: There were some reports on 10.5.8 fixing the problem. Any confirmations out there?

As it turned out for us, the problem actually wasn't AppleFileServer but the low level SCSI driver for the ATTO SCSI card used in our system.
We updated the driver to the lastest version available at ATTO's site and since then we have had uninterrupted uptime of 3 days, 19:07 hours. Seems like our problem has vanished.

The lesson to learn is: Don't only look at the obvious symptoms. From monitoring the system's CPU usage ( top -o+command -s 5 ) we learned that raising CPU usage of AppleFileServer in our case always was precedented by raising kernel_task activity, which lead us to reexamine the I/O subsystem.
Googling the web led us to hints regarding the ATTO card and the (hopefully) final solution.
And yes, now my Sysadmin and reckoned there had been another hardware change just shortly before the problems began: We had replaced our old AIT-3 tape library with a new AIT-5 system. That new tape library obviuosly triggered a misbehaviour in the ATTO card the old tape didn't....

BTW: Still running 10.5.7

We installed 10.5.8 yesterday after suffering from crippling high CPU loads a few times a day. So far, this seems to have resolved the issue.

We're keeping our fingers crossed, and will report if symptoms return.

We are running 10.5.8 on a brand new server(installed this summer and school has been in session for one week) and started experiencing the high Apple File Services CPU spikes Friday(up to 750 and it stays there for several minutes) and users started experiencing the SPOD. We have about 75(3 labs of 25) macs accessing network accounts, all but 6 are aluminum iMacs running ether 10.5.7 or 10.5.8 and the other 6 are eMacs running 10.4.11. Users were only using Word 2008 or Firefox. Seems to spike when I am seeing more than one lab in use at a time. Apple is having us try a couple different things:

1. Disable Spotlight on all sharepoints
2. If the first doesn't work, download the 10.5.8 combo server update, repair permissions, install the combo and then repair permissions again.

I will report back if either of these helps.

I have a Quad Core Xeon Xserve with a SSD for the OS and 3 1TB drives running RAID 5

My server ran fine for two days after the 10.5.8 update - the same issue has come back today as before. Server is very slow to use, even when the activity monitor shows no load, AFP sporadically goes up to 760% - all network home users complain about the dreaded wheels of doom.
Spotlight is disabled on all share points.
Would gladly revert back to 10.4.11 if the hardware supported it.

I will try the above suggestion (repair permissions, apply 10.5.8 v1.1 combo update, repair permissions) this evening and will report back.

Nico

Just to report back - unfortunately, the issue has returned on my server with 10.5.8 too. Worked fine for a couple of days, then AFP usage started ramping up again.

I will try repairing permissions / re-applying combo update / repairing permissions suggestion too.