afp server issue - very hign cpu load

Maybe I am reading too much into this but unchecking that can be dangerous. IF something were to happen when you uncheck that you could run into corrupted data.

IS everyone running Raid 5 and how many stripes?

Also this should ally affect those with the battery backup add on option. What would be interesting to see is if those people that have this issue have that option or dont.

I am also curious what firmware you guys have on your raids?

I'm seeing the same issue on a MacPro running 10.5.5 Server, and using Apple's hardware RAID card. After several days, APF reports very high CPU usage (300%+) and the server becomes unuseable. The logs don;t show much useful info, but I am getting a battery warning form the RAID controller...any thoughts?

Mine is RAID 5 with 8 Stripes 512KB/Disk.

Firmware 1.5.1

My AFP has been stable since 10.5.5 and redirecting the following for my users:

<key>com.apple.MCXRedirector</key>\
<dict>\
<key>Forced</key>\
<array>\
<dict>\
<key>mcx preferencesettings</key>\
<dict>\
<key>LoginRedirection-raw</key>\
<array>\
<dict>\
<key>action</key>\
<string>deleteAndCreateSymLink</string>\
<key>destPath</key>\
<string>/tmp/%@/Library/Safari</string>\
<key>path</key>\
<string>~/Library/Safari</string>\
</dict>\
<dict>\
<key>action</key>\
<string>deleteAndCreateSymLink</string>\
<key>destPath</key>\
<string>/tmp/%@/Library/Recent Servers</string>\
<key>path</key>\
<string>~/Library/Recent Servers</string>\
</dict>\
<dict>\
<key>action</key>\
<string>deleteAndCreateSymLink</string>\
<key>destPath</key>\
<string>/tmp/%@/Library/Cookies</string>\
<key>path</key>\
<string>~/Library/Cookies</string>\
</dict>\
<dict>\
<key>action</key>\
<string>deleteAndCreateSymLink</string>\
<key>destPath</key>\
<string>/tmp/%@/Library/iTunes</string>\
<key>path</key>\
<string>~/Library/iTunes</string>\
</dict>\
<dict>\
<key>action</key>\
<string>deleteAndCreateSymLink</string>\
<key>destPath</key>\
<string>/tmp/%@/Library/ColorSync</string>\
<key>path</key>\
<string>~/Library/ColorSync</string>\
</dict>\
<dict>\
<key>action</key>\
<string>deleteAndCreateSymLink</string>\
<key>destPath</key>\
<string>/tmp/%@/Library/Autosave Information</string>\
<key>path</key>\
<string>~/Library/Autosave Information</string>\
</dict>\
<dict>\
<key>action</key>\
<string>deleteAndCreateSymLink</string>\
<key>destPath</key>\
<string>/tmp/%@/Library/Adobe</string>\
<key>path</key>\
<string>~/Library/Adobe</string>\
</dict>\
<dict>\
<key>action</key>\
<string>deleteAndCreateSymLink</string>\
<key>destPath</key>\
<string>/tmp/%@/Library/Preferences/Adobe InDesign</string>\
<key>path</key>\
<string>~/Library/Preferences/Adobe InDesign</string>\
</dict>\
<dict>\
<key>action</key>\
<string>deleteAndCreateSymLink</string>\
<key>destPath</key>\
<string>/tmp/%@/Library/Preferences/Adobe</string>\
<key>path</key>\
<string>~/Library/Preferences/Adobe</string>\
</dict>\
<dict>\
<key>action</key>\
<string>deleteAndCreateSymLink</string>\
<key>destPath</key>\
<string>/tmp/%@/Library/Preferences/Adobe Photoshop CS3 Settings</string>\
<key>path</key>\
<string>~/Library/Preferences/Adobe Photoshop CS3 Settings</string>\
</dict>\
<dict>\
<key>action</key>\
<string>deleteAndCreateSymLink</string>\
<key>destPath</key>\
<string>/tmp/%@/Library/Preferences/Adobe Illustrator CS3 Settings</string>\
<key>path</key>\
<string>~/Library/Preferences/Adobe Illustrator CS3 Settings</string>\
</dict>\
<dict>\
<key>action</key>\
<string>deleteAndCreateSymLink</string>\
<key>destPath</key>\
<string>/tmp/%@/Library/Preferences/ByHost</string>\
<key>path</key>\
<string>~/Library/Preferences/ByHost</string>\
</dict>\
<dict>\
<key>action</key>\
<string>deleteAndCreateSymLink</string>\
<key>destPath</key>\
<string>/tmp/%@/Library/Calendars</string>\
<key>path</key>\
<string>~/Library/Calendars</string>\
</dict>\
<dict>\
<key>action</key>\
<string>deleteAndCreateSymLink</string>\
<key>destPath</key>\
<string>/tmp/%@/Library/Sync Services</string>\
<key>path</key>\
<string>~/Library/Sync Services</string>\
</dict>\
<dict>\
<key>action</key>\
<string>deleteAndCreateSymLink</string>\
<key>destPath</key>\
<string>/tmp/%@/Library/PubSub</string>\
<key>path</key>\
<string>~/Library/PubSub</string>\
</dict>\
<dict>\
<key>action</key>\
<string>deleteAndCreateSymLink</string>\
<key>destPath</key>\
<string>/tmp/%@/Library/FontCollections</string>\
<key>path</key>\
<string>~/Library/FontCollections</string>\
</dict>\
<dict>\
<key>action</key>\
<string>deleteAndCreateSymLink</string>\
<key>destPath</key>\
<string>/tmp/%@/Library/Logs</string>\
<key>path</key>\
<string>~/Library/Logs</string>\
</dict>\
<dict>\
<key>action</key>\
<string>deleteAndCreateSymLink</string>\
<key>destPath</key>\
<string>/tmp/%@/Library/Preferences/Microsoft</string>\
<key>path</key>\
<string>~/Library/Preferences/Microsoft</string>\
</dict>\
<dict>\
<key>action</key>\
<string>deleteAndCreateSymLink</string>\
<key>destPath</key>\
<string>/tmp/%@/Library/Fonts</string>\
<key>path</key>\
<string>~/Library/Fonts</string>\
</dict>\
<dict>\
<key>action</key>\
<string>deleteAndCreateSymLink</string>\
<key>destPath</key>\
<string>/tmp/%@/Library/Application Support</string>\
<key>path</key>\
<string>~/Library/Application Support</string>\
</dict>\
<dict>\
<key>action</key>\
<string>deleteAndCreateSymLink</string>\
<key>destPath</key>\
<string>/tmp/%@/Library/Caches</string>\
<key>path</key>\
<string>~/Library/Caches</string>\
</dict>\
<dict>\
<key>action</key>\
<string>deleteAndCreateSymLink</string>\
<key>destPath</key>\
<string>/tmp/%@/Microsoft User Data</string>\
<key>path</key>\
<string>~/Documents/Microsoft User Data</string>\
</dict>\
</array>\
</dict>\


I hope that makes sense.

It may not all be necessary, I had the entire Library folder redirected locally but it caused a bunch of issues to do with managed preferences.

The way I have it now, just about everything within the Library folder is redirected locally, the Preferences folder isn't redirected, but folders within it are.

Also Microsoft Office needs some redirection, I redirect the whole Microsoft User Data folder in user's Documents folder for most users, for users who use Entourage, I redirect selected folders within that folder instead of deleting it altogether.

Turning off "Allow host Cache Flushing" has improved AFPs performance also.

None of this may be wise, but I now have an environment that's working and I'm being shouted at less.

Hi Jeff

Unless there is something seriously wrong the Battery Warning is normal. Every 3 months it re-conditions itself:

http://support.apple.com/kb/HT1346
http://support.apple.com/kb/HT1346#faq10

You might find this useful as well:

http://support.apple.com/kb/TA24883?viewlocale=en_US

There is a slight performance loss whilst the re-conditioning takes place although its difficult to say how much? I've not seen any discernible difference.

What I have seen that might help with the observed cpu load is Spotlight spiralling out of control indexing and re-indexing everything all over again for no discernible reason. You could look there? If Fonts are being stored on the volume I would exclude that from the Spotlight Prefs Pane. I've seen a corrupted or dodgy font cause major cpu leaks resulting in Server instability like you describe.

Apologies if I'm telling you something you already know?

Hope this helps? Tony

sorry, but we are using a Promise Raid!!

Antonio - it has always been my preference not to spotlight big volumes such as the apple raid. I find that this causes some big issues, even though 10.5 was supposed to be better at it.

Thanks Tony:

I had already seen the battery info, and I've turned Spotlight off...but thanks for the replies. There's always a new gotcha...tough to keep up with them.

The problem seems to be related to Leopard Server not dealing well with the caches in home folders stored on the server. We discovered that if you redirect the cache folder back to the local machine the problem goes away.

We seemed to have solved this using a workaround found in this article:

http://www.afp548.com/article.php?story=MCXRedirector

It's been 2 weeks and no troubles since.

Use the renameAndCreateSymLink action.

I12BPhil - i tried out the MCX redirection solution from afp548, using the renameandcreate on login, and the deleteandrestore on logout. unfortunately, i quickly ran into Office 2008, and possibly Safari keychain issues. have you encountered such issues such as permissions errors on save from Office apps? and if so, have you figured out any solutions? i wonder if the new version of NHR works better... http://jochsner.dyndns.org/scripts/NHR.html

Redirecting caches should not cause Keychain problems and Safari shouldnt be affected.

The problems you are having with Office stem from redirecting your ~/Library/Caches/Temporary Items folder, since the redirection causes it to be held on the local computer rather than the same drive as your user files (the server), Office cannot perform atomic writes when it saves out a file.

The solution is to create a folder called .TemporaryItems at the root of your AFP sharepoint. chown it to root:staff (assuming your users are in group staff) and chmod it to 775.

Office will then automatically use this folder for it's temp file needs.

I've been in this boat since the 10.5 upgrade. What I noticed last week was that doing multiple logins with the same account (I was trying for load with minimal typing), the CPU shot up about 50% above the base level per test and then remained about 20-25% above the original base. This is what I'd been seeing in real world use (after 3 or 4 such steps the CPU remained at 100%, even if no users were logged in). I've unchecked the Allow simultaneous logins and will test after Christmas to see if this really is the determining factor. But I do know that students were frustrated by the slow response and logging into the next available computer while logged in to the first. And faculty were logging in in the labs without logging out in their office. I'd love to know if any of you see the same pattern (or does no one else have the box checked?).

Timothy

I have recently noticed a similar AppleFileServer process problems on one of my Xserves (dual quad core Xeons). The process can pin as many as 5 cores indicating a process usage of 500%. The only solution at this time is to reboot.
The services this server is running are minimal and includes the following:
OD Replica (1 of 2 replicas)
AFP (4 volumes and 8 AFP shares)
SMB (single SMB volume)
I upgraded to 10.5.6 last week, however the problem is not resolved.

My situation still remains the same. We actually had all 8 cores pinned on Dec 29. Killing the AppleFileServer process resolved the situation.

Here is what we have attempted so far:
- Disabled auto-disconnect in AFP after idle time (ServerAdmin)
- Removed spotlight indexing on all afp volumes and deleted .SpotLight-V100 directories on AFP volumes
- Set the following default: defaults write com.apple.desktopservices DSDontWriteNetworkStores true (see http://mikemchargue.com/2006/06/finally-defeated-afp-on-tiger-sever-i.html). Set as preference for all groups
- Verified that Host Cache Flushing is disabled on external RAID array.
- rebooting server and attached Promise VTrak array.
- system.log and AppleFileServer related logs do not show any apparent errors.

Although we have noticed that cores can get pinned during times of heavy throughput during the day (we actually had all 8 cores pinned on the afternoon of Dec 29), a core is almost guaranteed of going to 100% in the middle of the night during an archiving process that we have kicking off. No other activity is taking place at this time.

Cores will remain pinned till we either kill the AFS process or we reboot.

I have AFS thread samples taken when 1, 2 and 3 cores are pinned. Will be sending these off to Apple.

As reference, our configuration is as follows:
- 3 XServers identically configured (2 x quad core Xeons, 8GB mem, internal raided OS disk, bonded GB ethernet, OS X 10.5.6)
- one server dedicated OD master, the other 2 are OD replicas
- The server with the CPU/AFS problems is our main AFP fileserver and OD replica with a fibre-attached Promise VTrak Array

Cheers