Why is the resolution through VNC sometimes not right?

Know the base resolution of the host. If the host is a different resolution ratio than the client, resolutions get adapted.

It has been a while since I used anything other than the macOS built-in VNC viewer (ARD), but I seem to recall years ago I could configure a resolution to use with VNC when using the command line to connect because I had a similar issue where screen differences were just too great. Unfortunately I don't recall how I did this. Does the RealVNC viewer have any configuration options for setting the display resolution to use?

Actually a quick search gave me this forum post where it shows being able to set the display resolution when connecting to a server using TigerVNC. I know I've used TigerVNC on my Linux systems so this is probably what I did. According to this post, only TigerVNC is capable of auto-resolution as opposed to auto-scaling utilized by other VNC apps which (auto-scaling is basically a zoom in & out like feature with poor quality).

https://superuser.com/a/1485201

Otherwise you would have to configure the macOS VNC server to set a default display resolution, but I don't know if this is possible or how it would even be done.