Do you mean you System Preferences -> Sharing -> Screen Sharing -> Computer Settings -> VNC viewers may control screen with password ???
If you are going to use another Mac, then you do not need to set the VNC password, since other Macs can use your account's username and password.
The VNC password is for 3rd party VNC clients, and VNC clients coming from non-Mac systems.
If your Mac stays behind a home router,
AND you do
NOT open "Port Forward" port 5900 from the internet to your Mac, then ONLY your home systems can use screen sharing.
If you take your Mac out of the house and into public WiFi access points, then a potential stranger can attempt to connect to your Mac's screen sharing. So you should have a very good password.
Mac to Mac screen sharing, using the username/password, is done in a secure manor.
Using a VNC client and the "VNC viewers may control screen with password" is not secure. If done at home behind your home router, no problem.
If you make a screen sharing connection in a Public WiFi location using the "VNC viewers may control screen with password" then someone could sniff your connection attempt and obtain your "VNC viewers may control screen with password" password.
As for turning off screen sharing when not using it, as long as your have a strong password for both your "VNC viewers may control screen with password" value and for your Account/Password,
AND you do not make a connection where a Public WiFi could steal your "VNC viewers may control screen with password" value, then it is unlikely someone will guess the needed password information so they could get into your system.