Try rebooting the remote Mac.
Is "Remote Login" enabled or disabled?
I actually was having trouble connecting to a Mac using VNC myself. I had to try multiple times before I was able to connect. After disconnecting from the VNC session I was unable to access the remote Mac via VNC for a while. When I was having trouble I was able to enter my VNC password, but a VNC window never opened. Even when I connected to the remote Mac the graphics in the session were very slow to load and both computers are on the same LAN. I don't have these issues when connecting to a Linux remote system. Plus different versions of macOS seem to behave differently.
On Linux I use the following command to connect to a Mac using VNC (xtightvncviewer is being used -- I believe you can just use the tightvncviewer command as well):
vncviewer <ip-address-DNS-name-of-remote-Mac>
or
vncviewer <ip-address-DNS-name-of-remote-Mac>:5900
On the Mac I have enabled a VNC password (click on "Computer Settings" button under Remote Management). I have also enabled "Observe" & "Control" found on the "Options" button for the authorized user. When using the command line I am prompted for the Mac's VNC password before it launches a window for the remote GUI session.
If Remote Login is enabled, then you may first need to establish an SSH tunnel before being able to connect using VNC.
I don't recall if I ever was able to enable a logging session (If I did I don't recall how I enabled logging the connection attempt).
Do you have any security software running on either computer that may be interfering?