when the issue happens, can you connect using other proticals / programs? ie VNC via connect to computer in the go menu of finder? OR ssh in the terminal to connect to the server?
if you can ssh into the server, you could try the kickstart command to restart ARD.
if you can't connect to the server at all. Can you ping the server? Ether way, if you go to the server , will it browse the internet?
if you can't ping the server, are you using the Fully qualified DNS name to ping? or the IP to ping? or the server.local name to ping?