This has happened to me twice now, both times in exactly the same sequence of events I will describe below, since I installed Mavericks and OS X Server 3.0 on my Mac Mini server:
After running fine for a while (up to a day or two), I will attempt to do a share screen of the server from my laptop (something I do regularly with success), but this time it just says "connecting" forever and never gets any further. So I quit the Screen Sharing app and go to the console of the server to attempt to login there instead, but every time I key the correct password, the password field shakes from side to side as if the password was keyed incorrectly (which I know it wasn't ... and no, caps lock wasn't on or anything of that sort). Finally, out of some desperation I go to click on the "Restart Computer" button on the bottom of the Login page, which then clears the server's screen to a flat gray background as if it is rebooting ... ... and proceeds to stay that way forever until I finally have to forcibly shut down the hardware by holding down the physical power button. Then I restart the server using the power button, and everything seems to be fine again ... I can once again login, share the screen remotely, etc.
I have little info to go on thus far, since 10.9 has been out for such a short time, but it is at least possible that this behavior began after I enabled the Xcode Server functionality in the Server app (at least I observed that the first time it happened was only a few hours after I initially enabled Xcode Server). The other reason I am suspicious is that, during the time that the server is unresponsive and before I forcibly shut it down, just about all of the messages in the system log appear to be from Xcode Server. Either that, or it may have something to do with the failed screen share attempt, I suppose.
Is anyone else having the same problem? If so, do you have Xcode Server enabled? Or are you perhaps seeing a relationship to screen share attempts?