". . . I'm not sure how to enable RDP on my Mac . . ."
AFAIK you can't.
RDP is a proprietary protocol developed by Microsoft which I can't see being any "faster" than what you're currently using? Apple have their own offering known as ARD (Apple Remote Desktop). Yes it uses the same 5900 VNC port to establish connectivity but thereafter uses port 3283 for most of the control, observe and command features supported by ARD.
The client part of ARD is part of macOS. The Admin part (what this Forum is mainly concerned with) is a separate program/app better described here:
Remote Desktop - Apple.
If you've already established VNC connectivity from Linux to macOS and unless anyone else knows any different, then, I think, you've done as much as you can and, possibly, suffering poor connection speeds is something you may have to live with? Comparing your experience to my own regarding connection speeds, I've found Mac to Mac, when on the same network, is pretty much reliable and fairly quick depending on what I needed to do. There are oddities that pop up every now and then but that's nothing unusual.
In my experience when using ARD remotely, it's never been 'fast'. Mostly clunky, sluggish and occasionally unreliable. Other similar offerings (eg: TeamViewer etc) are like that (again in my experience) anyway.
What about SUSH? Depending on what you want to do it may be more suited to your needs?
Good luck