Incorrect server monitor password

Can you confirm the system is running Mac OS X Server 10.5.8 and not Mac OS X 10.5.8?

With Mac OS X Server 10.6 and earlier, the server distribution was an entire and separate DVD distribution from that of Mac OS X. The server DVD distributions were either the purchased distributions (usually either black DVDs, or a white DVD with a colored image in the case of Mac OS X Server 10.6, costing US$500 for limited users or US$1000 for unlimited users) or the recovery and support and upgrade DVDs for specific models of systems (usually grey DVDs).

If it's Mac OS X Server 10.5.8 and is tossing bundle-related errors, wipe and reinstall and try again.

If setting the unique IP address and user and passwords are established via Server Monitor running directly on the local Xserve system or maybe via ipmi and that doesn't address this access, then there's probably a hardware or networking problem.

Don't bond/team/gang the Xserve NICs together. That doesn't work when the LOM is required.

Also check that the firmware is current for whichever Xserve box is in use here.

Versions prior to macOS 10.9 are going to be having increasing problems with certificate and connection errors, too; the support for connections prior to TLS1.2 found in 10.9 and later is being deprecated around the 'net.