I've just updated to the latest version of Maverick Server (3.01) and have been able to get it working without downgrading to Mountain Lion Racoon.
However, I did need to turn off UPNP on my router. As soon as I turn this back on, I cannot connect again.
I do not have (never had) Back To My Mac on, so this isn't an issue for me.
I'm port forwarding ports 500 (UDP), 1701 (TCP/UDP), 4500 (UDP).
It seems to be a short-term fix for me. Would like to turn UPNP back on my router though. C'mon Apple.
Edit:
Sometimes requires 2 or 3 attempts to connect before it finally does.
Message was edited by: MarkeMark