remote desktop keeps disconnecting

I have a mac running 10.4.11 that I use Remote Desktop 2.2 to control an Xserve G5. Now in the last few weeks I can only connect to it for a few seconds at a time before it loses connection if I am even able to connect at all. The Xserve shows up in the list I can connect to, but sometimes it shows as offline, sometimes it shows as being able to connect to with VNC, and sometimes shows as having ARD version 3.2.2. Most of the time I get a failed to connect error or sometimes it says it has old software. There is no update for it when I do a Software Update check. It says it is offline if I try to upgrade the client software in the Remote Desktop menu. I've gone online to download updated versions and it looks like they install just fine, but then when I try to open Remote Desktop I get a failed to open message saying it may not have been installed properly or may need the computer to restart. Restarting the computer doesn't open it either. So now I can't even open Remote Desktop. Does anyone know how to fix this problem? The Xserve does not have a video card either to control it at the Xserve itself.