CANNOT_LOAD_BUNDLE_ERR from remote client (same local network).

I'm going to suggest the use of only IP addresses for testing here, and not DNS names. This to remove any DNS issues and any caching from contention. (You're focused on DNS as a potential trigger (which is quite reasonable) but that whole area can be eliminated from consideration by using the IP addresses for testing.)

Are you running both NICs on the Xserve in parallel, for some combination off LOM traffic and for "normal" traffic? (If both are active, then the usual IP routing and subnet rules apply.)

If you are running both NICs, do you have unique addresses for the LOM for each NIC and for the "normal" traffic on the server?

To verify the LOM address, launch Server Monitor on the Xserve box, and (re)configure the LOM. To verify connectivity into the LOM, ping the LOM IP address from a remote server. (The changeip stuff and most of Mac OS X Server itself does not "know" about the LOMs.)

Disconnect the errant Xserve from the network; pull both network cables. Ping the LOM IP address(es) from another host on the LAN and see if some other host elsewhere on the LAN is answering.

And largely for grins, re-download and reinstall the Server Tools package onto the client.

And try aiming Server Monitor from the working Xserve to the non-working Xserve; this to try to isolate potential triggers.

Power down the errant Xserve, unplug the power supply (both supplies, if you have redundant power) for a minute or so, then re-plug it. Having the LOM powered down and unplugged forces it to reload and reinitialize.

NIC: network interface controller

You have 128.178.100.87 for both a NIC and for a LOM? That's incorrect.

For a "simple" configuration for a default two-NIC Xserve box with full LOM access on both LOM ports onto the same LAN or same switch, you would want four IP addresses, in two subnets. The subnets would be co-resident on the LAN, but would allow both NICs to be active and routing in parallel.

Your use of en2 and en3 implies there's a second dual-port NIC in this box, and your reference to the bond device implies there's bonding in play. Use caution here as link aggregation (bonding) is incompatible with LOM; the port(s) used with the LOM can't be bonded. See [HT2773|http://support.apple.com/kb/HT2773] for details.

Your use of a public subnet and a private subnet also implies this box might also be a gateway. That means you get to check the wiring here, as boxes that are gateways tend to have partitioned wiring.

As for the wiring, that 400 ms value is notably slow for what I'd expect with a LAN ping. I'm getting 28 ms first and dropping to around 5 ms for subsequent pings when pinging a server from a client over WiFi connection into a gigabit LAN. Is your network around the XserveBAD box stable and reliable?

When you change LOM settings such as the LOM IP addresses, you must unplug the Xserve box for a minute or so. That's how the LOM itself is rebooted. (The LOM is tiny little computer that's co-resident on the Intel NIC that's used in these boxes, and a computer that's individually IP-addressed, and it's always powered up and running when the box is plugged in.)

AFAIK, Server Tools is resident with and installed with Mac OS X Server. You won't need to add it to an Xserve box running Mac OS X Server.

And FWIW, I'm having to think more when sorting out this Xserve1 and Xserve2 stuff. (Smell that woodsmoke? 🙂 ) The human interface guy inside that's yearning to be free wants me to suggest consistent use of XserveBAD and XserveGOOD or SadLOM and HappyLOM. 🙂 (Not making me go back and figure out if 1 or 2 was the GOOD or BAD Xserve while I'm sorting out the rest of the configuration.)

I'm going to skim over some details of IP and IP routing.

My IP for that NIC on which the LOM is running is 128.178.100.87.
If I'm reading you correctly, I should NOT have an IP for my LOM (if you say 'correct' then I'm really confused).

Your LOM needs its own unique IP address.

This can be a little confusing if you're looking at the hardware, but it's entirely possible to have several separate IP hosts (devices) on one network jack, and each needs its own (unique) address. You're not assigning IP addresses to cables, you're assigning addresses to hosts.

And if I read you to say that yes 128.178.100.87 should be assigned to the LOM but not the NIC,.. then to what is the LOM attached/configured?

Well, if two hosts (or even two network controllers on the same host) have the same address (and the LOM is a host), then (best case) the IP traffic on the second host controller to start detects the address collision and won't allow the host to go online. Worst case, traffic gets snarled.

Finally, it seems that this setup on XserveBAD (an XServe3,1) is the same on XserveGOOD (and Xserve2,1) in terms of IPs NICs and LOMs. The only diff seems to be Xserve versions. 😟

There was a brief kerfuffle with the Xserve3,1 boxes and the tools back when those boxes first arrived, but that all settled down with current tools and firmware.

*2. 2nd dual-port NIC*
YES indeed! 🙂 You're good! From the Preferences>Network panel I see we have a PCI ethernet with two ports (both XserveBAD & XserveGOOD have this) link-aggretated. To that bond, we've assigned the IP address that people use to ssh into the machines. LOM is not there. XserveBAD's LOM is configured to say +LOM uses ethernet Port 1+ and XserveGOOD's LOM is set to +uses Network1+ (the GUI looks different on the two machines).

Two networks (also) means you'll have to consider IP routing; which NIC is used to respond to traffic. Inbound IP packet traffic might well arrive at the box, but if the box doesn't have its routing configured "correctly" then the IP packets "outbound" in response might not be sent out the expected controller; the inbound IP traffic and the IP responses don't automatically use the same controller. Those routing determinations are made separately, based on the local subnet, available local static routes, and the rest of the packets get sent to the default router to deal with.

For both XserveBAD and XserveGOOD, the Preferences>Network panel shows the IP that we configured for the LOM to be the one on 'Ethernet 1" with subnet mask of 255.255.255.0. And in both cases the IP address used for sshing (the one used on the PCI ethernet bonded ports) is on 'Ethernet 0' with subnet mask 255.255.255.0.

The subnet masks tells IP where the IP packet is going to be set. In the most general of terms, that mask tells IP routing where to send the packet, and that decision (and specifically with the 255.255.255.0 mask) says that if the first three bytes are the same, it's a local address and the messages get sent directly. If the first three bytes differ (and it's not an established route), it goes to the designated "gateway" router.

p.s. FWIW, I almost went with SadLOM and HappyLOM but I'm still pretending to be a serious mac admin-wannabe here. 😉

Uh-huh. You'll get over that. 🙂

p.p.s. I didn't catch your drift concerning public/private subnets.. but I'm googling, I'm googling and reading here http://articles.techrepublic.com.com/5100-10878_11-6089187.html

Your network uses IP addresses in the public static IP space, and IP addresses in one of the private address spaces. (The same would apply if you had IP addresses in different subnets in any of the private IP address blocks, though the use of a public space implies there's a "public" LAN and a private LAN.)

That looks reasonable as far as it goes, albeit a little stale, and it unfortunately lacks the associated discussion of routing; of why you'd be interested in subnets, and what happens with the packets. The class stuff discussed there is replaced with what's called "CIDR" - classless routing -- and is long gone. (I've been around long enough that the class A, B and C stuff is second nature and old habit, but that "classful" scheme has been replaced with the far more flexible "classless" CIDR scheme.) And the article lists 17 years for IPv4, and the last of the available IPv4 blocks are likely to be handed out within the next few years; we're all dealing with IPv6 now, and its likely that IPv4 addresses are going to get harder to get and probably also more expensive.

when next you're in Switzerland, I owe you free beers.

Danke. Der Ratskeller in Zürich war ziemlich gut. Quite a pretty walking area around there too, and with a beautiful view of the river. 🙂