Why does Screen Sharing require a Mac Pro reboot for subsequent connections?

I reboot my Mac Pro and can access it via Screen Sharing just fine. Once I'm done, if I quit Screen Sharing and then want to go back in, it always fails to connect. Reboot and I'm back in just fine. What is the issue? I don't want to Log Out, as I have converted the Mac Pro to run as a headless server for Plex and VirtualBox. I have installed AnyDesk and it works (Every time), but it's a little slower than Screen Sharing. This has got to be a simple one, ideas?

[Re-Titled by Moderator]