Delays and related can be caused by DNS errors, and by network errors.
OS X Server is dependent on correctly-configured DNS services.
To verify DNS, launch Terminal.app from the Applications > Utilities folder on the server, and issue the following diagnostic command:
sudo changeip -checkhostname
That command won't make any changes, but will report whether your host name and your DNS are correct on your server. You'll get some output and a message that no changes are required, or some output and a message indicating something about whatever error was detected.