You could force remote clients to tunnel their inbound VNC connection through ssh, in order to restrict access. I've got a couple of posts in one or more of these forums that explain how I set up my computer to do that.
Additionally, in Sys Prefs | Sharing | Services, when you check ARD checkbox then click on Access Privileges, you can restrict what individual user accounts (on computer that is acting as local VNC server) can do, by highlighting user account, then clicking the appropriate privileges checkboxes. Would that give you enough control over other users?
Plus, don't leave the VNC server computer in a logged-in state. You can login once you've made the VNC connection.
On the Mac, the freeware client CotVNC has a checkbox on the connection page that allows a user a choice as to whether to permit simultaneous connection to an active session by another user, or to be exclusive user of a session. Hopefully, tightVNC does, too.
So, limit other accounts' VNC privileges in Sys Prefs | Sharing | Services | ARD | Access Privs, tunnel connections through ssh, be exclusive user in an active session, and then login to your account on the VNC server computer's login window. That might be enough protection to meet your needs.