afp server issue - very hign cpu load

I just had to restart the server, after everything had ground to a halt for network home users. After the restart everything was still very slow - until I asked all network home users to quit Firefox - problem seems to have gone away.
Will delete Firefox off my netboot/netinstall images.

Thanks to lmartinsantos for the suggestion!

After trying Apple's suggestions of disabling Spotlight on sharepoints and fixing permissions and reinstalling Server 10.5.8 combo update we had a few days without high cpu load and corresponding high(790) AppleFileService load. Now we have had times throughout the day when we get high cpu loads and then it will drop and be fine for awhile. It appears to me when we get up to around 60+ users connected to server, we start developing high cpu loads. I am going to try to install NHR on the clients and see if this helps by redirecting Cache and Font folders. I am running out of things to try.

I have been following this issue for over a year now and I know a number of people who have stuck with 10.4.11 server for this reason. It looks like the choices are to either abandon AFP (which I suspect Apple will be doing soon), or to invest again in Apple and hope it is fixed in 10.6 Server.

I guess we will find out in the next few weeks if it is fixed.

Hello,

last week this issue happened to hit me as well.

As reinstalling the whole system did not help, I thought that my problem might be a bit special and I got a hardware problem. (Server is "just" a fileserver. No home folders)

But as it than suddenly worked. I did not change hardware. And I was using the old System again. So everything worked fine with the same setup.

Until today.

What I than did was:

1. Deleting files on the attached raid so that it had more space. (from 80GB free space to 220GB)

2. I shut down a computer of a colleague who was in holidays the last days and came back today.

After that, I had no more problems for the complete day!!

Now, I just hooked the colleagues computer back into the network.. no problems.

So I suggest three things.

1. The AFP service needs a lot of space to be free on the attached storage to work.

2. The computer of a colleague and another computer togehter trigger the problem. As the other computer might be off at the moment, I do not have a problem.

3. I was just lucky, this problem is completly random and can f*ck up my server anytime soon again.

I will see about Point 2. tomorrow when every colleague is back at work.

-----------

Last week, as I had this problem the whole day. I turned of my raid and set up a sharepoint on an internal harddisk. This worked without any problems!

This is so weird...

s.

Has anyone tried the 10.5.8 Server v1.1 Combo update released on the 31st of August ?

Just wondering if this has fixed the AFP issue and I don't want to be an early adopter of 10.6.

Thanks

Hi

We updated one of our home folder servers that has the latest Xserve hardware with the 10.5.8 combo a couple of weeks ago, this seems to have improved things but it is not a fix as we still have AFP causing very high CPU %.

Tonight I shall be installing 10.6, fingers crossed this does the trick I shall post my findings in a day or so.

Bests

10.5.8 Server v1.1 Combo doesn´t fix it. I still got the problem.

We are having a similar issue with high cpu usage multiple times a day on our data servers (Xserves running 10.5.8) which slows our network to a crawl. Is there any correlation between the client OS version and the server OS version which would cause this?

My new Xserves at 2 buildings that were giving me problems with very high cpu loads 0f 90-100 and AppleFileServices loads of 270-290 now seem to be running a lot better with cpu loads under 50 and usually under 20. Here is what we have done:

1. Disabled Spotlight on sharepoints(didn't seem to help)
2. Upgraded servers to 10.5.8 by downloaded the server combo update(didn't seem to help)
3. Increased server threads from 200 to 400(might have helped a little)
4. Changed the afp wanquantum=131072 and afp wanthreshold=1000 on the clients(may have helped a little)
5. Downloaded NHR and installed NHR.pkg on clients(this seemed to have helped the most)
6. But most of all - time seems to have helped the most, it appears that when school first started 3 weeks ago, cpu load and AppleFileServices load was the worst and we would get spinning beach balls under high loads. I am guessing a lot of this was probably from spotlight indexing, files being written to server the first login, cache files, etc. But it appears that after doing all the things above I have been running smoothly for over a week with high cpu loads.

Same issue here...
how many of you are on Xserve (Early 2008) with an Apple Raid card and SATA drives??
there's an interesting thread that someone just started in hardware>xserve that I'm watching due to the eerie similarities that have made me start to wonder whether it's actually software or hardware.
(see the crosspost in http://discussions.apple.com/thread.jspa?threadID=2142070&tstart=0)

Here's what I've got:
* Xserve (Early 2008) (OS 10.5.8) 16GB RAM with Apple Raid card using three 1TB SATA Drives as raid 5
* network homes from iMac g5's and iMac intels all running 10.4.11 averaging about 50 users at peak times, normally about 25-30.


This setup worked perfectly for about 4 months (running server 10.5.4) and then for no explicable reason things started going south. we hadn't installed any updates, software, or made hardware or network changes. Since then, once about 20 users log in, we have SWOD everywhere, taking 5 minutes to log in, AFP eating all 8 cores to about 780%, for about 3 minutes at a time with a few minutes of normal processor use in between. There are times that even when the processor is quiet it takes over 3 minutes to do a local admin login ON CONSOLE!.

I backed everything up, did a CLEAN install of 10.5, patched it to 10.5.8, recreated all the users by hand in case there was something hosed with the OD database, and within 3 hours of normal use everything was back to being slower than death.

I've tried just about everything I could find... all the fixes in the Apple Client Management whitepapers, as well as some other possible fixes from the forums, more disk and permission repairs than I can count, raid verifications, etc.etc.

I did notice however that during the high AFP load, disk I/O drops to nearly nothing. less than 200k/sec PEAK. (that raid card can do peak transfer of 300M/sec) even with nearly no I/O activity, all the drive lights are on nearly solid. Raid utility says everything is good, write cache is on, battery is not conditioning... it should be moving like a speed demon, but it's not.
I replaced the RAID card, in case it was the issue, but no luck.
perhaps one of the drives is failing but not in a way that SMART or the controller can pick up?? I just picked up a spare 1TB drive module to test that theory, but that means swapping the drive and waiting all weekend for the array to rebuild itself before we can check performance. for each of the 3 drives...

right now we're in the process of setting up a modified login (like NHR) that just maps documents, pictures, movies, favorites, and safari bookmarks to their AFP home folder to cut down on the overhead, but I'm not sure that even that will help.

We have an early 2008 XServe, but we don't have Apple Raid Card. I have a dozen users on 10.5.8 and another 20 on 10.4.11. Most have network homes on the server, a few have portable home directories.

I'm starting to think that this could be related to free or contiguous disk space. We've had a bit of a clear out and increased free disk space a little from about 87% used to 82% used on 1TB Mirrored (software) RAID, not a huge difference, but we haven't seen nearly as many instances of high CPU load. We had only one yesterday and none today. We installed the 10.5.8 v1.1 combo update this morning, so will need to wait to see whether it's disk space or the update that's done the trick. If it's neither I'm going to try the NHR option.

We've had a hard drive fail on us a couple of times in the past year, so the high load could be related to some impending failure, but there seems to be too many of use experiencing this issue for that to be a common cause.

Anybody else with full drives?

in our case it's definitely not a drive space issue... server boot/public storage volume (1TB) is only 21% used, data volume (1TB) is only 28% used

I posted the lines below in the discussion "XServe (Early 2008) With RAID card and SATA drives performs slow" a few minutes ago. I am convinced it is directly related to most of the trouble we read about in this discussion as well. Please investigate your setup and give us all feedback.

##########
Just to report I see exactly the same behavior on 8 Xserve (all Early 2008 with Raid Card).

Sporadically I/O speed on the internal RAID 5 drops to kb !! level ... causing disastrous side effects.

Every time this happens, all 3 drives show constant activity lights but Apple's activity monitor reports nearly no I/O! I created a workaround for our users by shifting most data to two external FW800 drives connected to the back of each Xserve. Speed is real 80 MB/sec on the FW 800 ports and independent of server load. All servers perform without issues in this temporary workaround setup.

In my opinion this proves, it is related to the RAID card and/or the used RAID Level 5 on the three internal 1TB drives. I kindly ask everyone with the same hardware (Apple Xserve Early 2008 with RAID card 3x 1 TB) to focus testing on the hardware.

Most of all issues (slowness, AFP hangs etc.) are instantly solved when bypassing the internal drives connected to the RAID card, so it is unlikely an issue of Leopard itself.

For the time being I cannot use the internal drives as planned. My production system stalled several times and I cannot afford this! If someone from Apple reads this please give me some feedback what can be done to solve it once and for all for all users in this discussion and other users in "afp server issue - very hign cpu load" (same reason, I'm 99 % sure)!

All feedback is highly appreciated!
##########

Hello,

we have an Xserve (Quad Core intel Xeon) 3.0GHz-DP, produced August 2008.

We do not have the Apple Raid Card and we do not have an internal Raid.
What we have, is an ATTO FC Card and a Promise Raid attached. (which is Raid5 configured).

Nevertheless we have the same AFP High CPU load.

As this problem still is not fixed, we concider to leave OS X Server und use something else. This problem is a massive slow down on our daily work and we can not afford this.

s.

Please note that this forum is not for official support.

If you feel you need/want official support for a problem you are having, please make certain to call Apple support, starting with the support number you have if you have purchased any specific support-program for your server(s).

High CPU load in and of itself is a red herring. OS X is designed to use the CPU, what else is it there for ? 😉
However, if CPU usage pegs and stays there consistently, it may be indicative of an underlying problem

What is vital is to try to determine why this is happening for some and not at all for others.