AFP Service Won't Start Properly

Question

Level 1

94 points

AFP Service Won't Start Properly

On Friday, one of my users complained that she couldn't get to the server's AFP volumes. After some investigation, it turned out this wasn't an issue at her workstation, but an AFP issue at the server.

Saturday, supervisors came in to finish some projects, and we discovered they could no longer connect to the AFP volumes. After the normal sets of things to try, stop/start the AFP service, reboot the server, run disk utility, etc.. I discovered that the AFP Service wasn't starting at boot, and when we tried to start it manually through Server Admin, or the Terminal, it kept coming back with the Service = "STOPPED".

I've checked the /Library/Preferences/com.apple.ByteRangeLocking.plist file, and it looks good, I've deleted/re-created the /var/db/BRLM.db file, and this didn't change things either.

I've done "ps -ax" at the command line and found 2 PID's running with "AppleFileServer" while the service seemed to be stopped. Even after doing "sudo kill -9 <PID>" on them, when I tried to start the service again, they came back, but the service wasn't started.

The only thing I see in the system.log has to do with "configd [81] AppleTalk startup failed, status = 69 (retrying)" but eventually it skips it.

We can access the server volumes via the Windows SMB connection, but that won't help us with our images-app that requires the AFP side for linking images.

Anyone seen this before and know how to sort it out? I've tried things like re-installing the 10.4.11 combo update, and that didn't seem to change any of the symptoms either.

MacBook Pro 15" 2.53 GHz Intel Core 2 Duo, Mac OS X (10.5.6), 4G RAM

Posted on May 9, 2009 12:33 PM

Reply

Answer 1

Lionchild Author

Level 1

94 points

May 9, 2009 6:14 PM in response to Lionchild

I thought I'd drop in a little update.

When the two PIDs of AppleFileServer are running and a user tries to connect, the CPU ramps up to 80 - 90 percent utilization one of the processes and stays there. There is nothing in the AppleFileServieError.log except that the log was created. The AppleFileServiceAccess.log has nothing in it but where volumes are mounted and an entry "DiskArbStart -" 0 6147 0.

The system.log doesn't have anything about an error in the AppleFileServer process, except when I use a kill -9 to crash the process manually. The Terminal command 'sudo serveradmin fullstatus afp" returns the following (with the 2 PIDs active):

afp:guestAccess = "NO"
afp:logging = "NO"
afp:failoverState = "NIFailoverNotConfigured"
afp:currentThroughput = 0
afp:state = "STOPPED"
afp:logPaths:errorLog = "/Library/Logs/AppleFileService/AppleFileServiceError.log"
afp:logPaths:accessLog = "/Library/Logs/AppleFileService/AppleFileServiceAccess.log"
afp:currentConnections = 0
afp:readWriteSettingsVersion = 1
afp:startedTime = ""
afp:setStateVersion = 2

And, of course, in Server Admin, we show that the AFP Service isn't started, and still won't start when you click the wonderful little green "Start Service" button. The only other services running in this box are FTP and Windows.

I'm about to embark on building a new install on an alternate HD and bring it up to speed, then hope I can transfer the one app we use for image organization. However, I would really just like to fix this error and get things back up and running.

Reply

Answer 2

Lionchild Author

Level 1

94 points

May 9, 2009 8:15 PM in response to Lionchild

And one more thing to add that I've discovered in the system.log file:

diskarbitrationd[45]: AppleFileServer[229]:32259 not responding.

So, this "diskarbitrationd" could have something to do with the AFP Service not starting?

Reply

Answer 3

davidh

Level 4

1,890 points

May 10, 2009 4:01 PM in response to Lionchild

Hi.

The diskarbitrationd message should get sorted first.
Since you said the disk in question checks out OK via Disk Utility, I suspect the system is waiting on another volume. Use Disk Utility to verify all attached volumes, and check on the SMART status for each as well.

Past that, see the following from the diskarbitrationd manpage:

diskarbitrationd listens for connections from clients, notifies clients
 of the appearance of disks and filesystems, and governs the mounting of
 filesystems and the claiming of disks amongst clients.
 diskarbitrationd is accessed via the Disk Arbitration framework.
 Options:
 -d Report detailed information in /var/log/diskarbitrationd.log.
 This option forces diskarbitrationd to run in the foreground.

Normally, you'll want to set aside /Library/Preferences/com.apple.ByteRangeLocking.plist
as you did with the BRLM.db file,
and also set aside /Library/Preferences/com.apple.AppleFileServer.plist

Next try making a new folder for each assigned sharepoint, and migrating the data from old folder to new, and recreating the access permissions appropriate for each sharepoint.

Generally, those are some steps that can resolve AFP not starting.
Since you're running 10.4.x , use fs_usage to track what files are being hit as you attempt to start AFP.

Shut off all other services (barring OD of course) and run
sudo fs_usage -f filesys

in the Terminal, just as/prior to trying to start AFP.

Otherwise, you might create a simple bash script to wait a second, grab the PID of AFP,
and run fs_usage -f filesys
specific to that PID.

Best of luck, let us know how it goes.

Reply

Answer 4

Lionchild Author

Level 1

94 points

May 10, 2009 4:24 PM in response to davidh

I've used DiskWarrior against my 3 internal drives, and corrected minor errors, as well as had DW run hardware tests against the drives, all of which respond with 'ok' from SMART controllers. I have one drive, however, that is my 'scratch' drive, and wouldn't let me erase the drive, so I've pulled it and plugged it in via a SATA/USB connector and to get it to format, I'm having do use the Zero Out Data option under Disk Utility.

My additional drives on this server are a 1.7TB XRAID running RAID5 and a 2.2TB RAID0 volume to store a backup copy onto. Those do not show any errors either. I'll soon give your suggestions a try.

Are you suggesting that I turn off all my sharepoints, then create new folders, move the data into those folders and turn the sharepoints on in those new folders, and then rebuild all ACL permissions on those new folders?

When I stay that AFP isn't starting, it's showing up in both the Activity Monitor and in Terminal's PIDs, but in Server Admin it doesn't show as started. When someone tries to connect to the server, the CPU cycles on one of the AFP PIDs jumps up to 80 - 90 percent and stays there, even though Server Admin says the service isn't started, and the client/user doesn't get a connection.

Reply

Answer 5

davidh

Level 4

1,890 points

May 10, 2009 4:29 PM in response to Lionchild

Disk Warrior is a very fine tool for very specific scenarios, but it is no panacea.

You'll want to use fs_usage at this point. Stop guessing, guesswork is not getting you anywhere.

You'll want to kill any afp processes, and then try to start it via

serveradmin start afp

and watch what files are being acessed via

suda fs_usage -f filesys

noting that it could be quite noisy if you don't shut off other services and besides, any file access is going to be noted. Better to target the PID of AFP.

About the folders for sharepoints: yes, especially if they're quite old and were migrated from any prior server(s). But do the above step first.

Always choose methods of gathering info - whenever available - over pursuing a "shotgun" approach which is generally inefficient, frustrating guesswork.

Reply

Answer 6

Lionchild Author

Level 1

94 points

May 11, 2009 11:23 AM in response to davidh

DW is definitely a nice too, but definitely not a cure-all.

My thanks for pointing me down some road to get more info. I was way tired of guessing. (Which, of course, was my whole point in requesting help!)

Currently, we have an active AFP Service, after removing all Sharepoints, going through them and sort of sanity-checking user permissions and inherited properties. I've then restarted the server to see that AFP started, then added sharepoints back, one at a time. I did not add back some of the older sharepoints that weren't in use, or were low-priority, but ones that were key for production.

Thus far, the server has been running and users are in working in stress-test mode. I will be monitoring the server closely for a while longer to insure that AFP stays up. davidh, your input has been invaluable so far, and I'll be looking forward to marking this as 'solved' if things go as planned by the end of the week.

Reply

Answer 7

davidh

Level 4

1,890 points

May 11, 2009 11:30 AM in response to Lionchild

You're welcome, glad to help.

The trick of freshly recreating folders that serve as (the) sharepoints is an old one,
and sometimes is necessary particularly when the original folders have been migrated one or more times from older servers.

I also recommend the Apple Mac OS X Server mailing-list as a resource, however, it can be extremely blunt and sometimes even a bit "rough & tumble" at times. That goes with the territory however (any mailing list really).

see lists.apple.com

Let me know how this goes/holds up.

Regards,

-- david

Reply