Have you contacted the author of Jolly about your problems ? Seems to be a nice guy.
AFAIK, you already have tried all possibilities there are for VNC on a Mac.
My main gripe with Jolly is actually the UI, I just find it a bit unintuitive.
There are other clients e.g. Jaadu VNC - but that requires a Jaadu VNC server.
I just did a bit of performance timing and I found that when dragging a Finder window an inch across the screen Jolly took 9 seconds, but Apple's Screensharing took 3 seconds. Perhaps I'll stick with Screensharing and add some of my hosts as Textexpander snippets.
The other thing about Screensharing is that the response, although a little sluggish compared with real time is pretty immediate, but with the others you can wait 5 or 10 seconds before anything visually happens at all - this can lead an impatient person (like me) to start randomly moving the mouse around and getting in a right tangle.