Netboot Service Fails

I am going to cross-post this to the MacEnterprise.org mailing list.

I am seeing the same problems with OS X Server 10.3.9 and 10.4.2. The server starts as normal, the Netboot service shows up fine, and the first OS X client can start NetBooting.

Subsequent clients are unable to finish boot--they can find the server and request the image but stop there at the graphical stage that Mike Bombich would describe as "spinning globe turns into indeterminate progress indicator" ( http://www.bombich.com/mactips/netboot.html ). Server Admin shows the NetBoot process as "busy" (light green with elipses) and all the Server Admin detail screens are blank. NetBoot has functionally crashed. NFS services show up in the Server Admin screens as all running fine. The faulty behavior did not appear prior to OS X Server 10.3.9 and I had previously been able to perform 50-client NetBoots in one of the same network environments (I am seeing the problem at three sites).

I did a bit more troubleshooting and started the NetBoot client in verbose mode. It became clear immediately that there were RPC communication problems using both TCP and UDP. Running "rpcinfo -p" on the server showed everything looking fine. A "showmount -e" to the server's IP, however, showed that no mounts at all were being offered by NFS (i.e. no /Library/NetBoot/NetBootSP0 shows up with showmount). One of the log files available through the XServe's Console.app showed that mountd was aware of the requests from the clients but was not able to fulfill them. Killing mountd and bringing it back up in debug mode (mountd -d) does not indicate any strange behavior, and does restore the mounts to "showmount" but does not actually fix NetBoot, which by this point seems hopelessly lost. The only way to restore NetBoot functionality from here is to restart the server.

(The curious will notice that just about all these troubleshooting steps were described either on Bombich's site or on pages 454-455 of Bartosh and Faas's "Essential Mac OS X Panther Server Administration)

The problem is not entirely consistent. Once the system breaks it seems to exhibit the pattern above for a while, even after restarting the server. But within a day it may be able to start working fine again and deploy images to clients. I tried it just now to break it again to try to get more information for this post, but it is actually working again at the current site (although not at two others).

Has anyone else seen this and done any further troubleshooting on it?

It should have been obvious to me earlier what the simplest workaround for this is: Don't use NFS for NetBoot. Or, rather, use HTTP instead.

Make sure that your web service is running. Then go to Server Admin under Netboot-->Settings-->Images and change Protocol for the selected images from NFS to HTTP. Restart NetBoot (or the server) for good luck and everything should work about as well as it should under NFS.

I've got three sites with the problem and it is working fine at one of them. I'm going to test the other two this weekend.

It should have been obvious to me earlier what the simplest workaround for this is: Don't use NFS for NetBoot. Or, rather, use HTTP instead.

Be aware, though, that you may have problems if your NetBoot image (or rather, the image file) is larger than 2 GB. Take a look at KB 107683. (This is a limitation imposed by Apache, not NetBoot per se, though IIRC, the limitation is gone with Apache 2.0.)

David Walton
Lane Community College

It should have been obvious to me earlier what the simplest workaround for this is: Don't use NFS for NetBoot. Or, rather, use HTTP instead.

We're having a similar issue on our newly Tiger-ized XServe (10.4.2). Switching to HTTP worked for one machine - but the rest were slower than molases in winter just to boot the machine up - forget about using NetRestore after it boots, 'cause that just takes too long.

I also note that the root problem still isn't addressed - why is NFS apparently failing or disconnecting from NetBoot? Has anyone managed to get it working? Has Apple acknowledged the problem? I have a feeling that I'll be calling for support later today....

I am also having this problem!

Just wondering what is your hardware configuration for your server? What switches/hubs are you using for your network?

Cheers

Have done a bit of fiddling on this.

My network was a closed environment (ie not connected to the internet).

Today i tried connecting my server to the internet using the 2nd NIC. (xserve -> dumb hub -> router) I made the 2nd NIC the default.

All of a sudden my server stopped having this issue. I thought this was a bit strange and thought that maybe the server just wanted both NICs enabled. So i turned off my router only. And the problem came back! Turned it on again and the problem went away!

This would suggest that my server is looking for something on the internet?
Sounds odd to me although I am able to netboot properly now.

Thoughts anyone?

DNS?
I mean: what's in your server's (and image too) network preferences? The only (big) difference I can think about between being connected to the Internet or not is name resolution.
If the server does have DNS set but they are unreachable, all sort of strange things can happen...

Just a guess, but setting up the DNS server in OSX Server (to resolve local addresses, and forwarding requests to "real" DNS servers for non-local ones) have saved me a LOT of headaches...

I'm having the same exact problem. It's very frustrating.

This didn't help me, but have you seen this:

http://docs.info.apple.com/article.html?artnum=301494

Change the logging on your DNS server to be debug mode

Then tail -f /Library/Logs/named.log

It looked like I had a few weird entries in the logs right at the point where the client makes it's NFS connection to the server.

Nicola is absolutley right. You need to create a DNS zone and record for your server so that you can successfully do forward and reverse lookups on it.

There's an article to this effect out there elsewhere, can't seem to find it. There was an Apple Discussion thread on setting up tiger server that someone posted that had these directions in it as well.

Then in your Network settings for your server, the first DNS server it should look to is itself at 127.0.0.1 . Then add your 'real' DNS servers.

-D

I am having very similar problems in a medium-sized enterprise environment. We have about 10,000 machines on our network and VitalQIP for DNS management. There are about 1,000 or so Macs on our network and I have a 10.4.2 NetBoot server that is used for remote imaging (with ARD) and diagnostics/repairs.

Since the upgrade to Tiger on the server, I am also seeing the darker grey status bubble next to the NetBoot service in Server Admin.app. The service will appear normal shortly after restarting the entire server. I can access logs, review the client list and settings, make configuration settings, and start and stop the service. I can usually boot one or two clients before getting the cursed grey dot, but once it appears, I can no longer do any of the above mentioned tasks from Server Admin.app. NetBoot continues to work for me, I can boot clients until the cows come home, but I cannot make any configuration changes or see who's booted.

I have reinstalled the 10.4.3 server combo update, repaired permissions, created new boot images, and toggled other netboot or netinstall images with no success. I am providing NetBoot via NFS. Turning the DNS service on for this server is not an option.

I am getting an awful lot of these in system.log and console.log:

"Nov 18 10:55:10 netboot DirectoryService[41]: Potential VM growth in DirectoryService since client PID: 46, has 950 open references when the warning limit is 500."

By 'awful lot," I mean 3-5X a minute.

I see plenty of others are having similar problems, but no 'silver bullet' yet. Any more suggestions?

10.4.4 Server update was slated to fix this problem, does not seem so as I am still having the problem with the NetBoot service failing and turing gray in Server Admin.app. The clients still boot, but I cannot edit any settings without a reboot. According to the 10.4.4 update, one of the fixes is:


****
Server Admin

* Addresses a cosmetic issue in Server Admin which caused long image names to truncate incorrectly.
* Addresses an issue causing Server Admin to incorrectly report a running service as stopped or unavailable.
****

But it still happens.

This is very frustrating for all NetBoot users I'm sure, I'm surprised this problem has been allowed to go on for so long without a resolution. 10.3.9 server works just fine, same setup, same images, same clients, etc.