I too am struggling with the SAME situation! I've also had to rely on using TeamViewer which is night and day in performance.
My situation seems to be the most noticeable when using a heavy application on the system in question. The app is ProPresenter 6 which is a presentation software. I have two external displays connected to a new iMac also. When PP6 isn't running ARD works but still a little laggy. Launch PP6 and a complete stand still. Launch TeamViewer and smooth as silk. The iMac is running macOS High Sierra 10.13.2. Connecting system is macOS Sierra 10.12.6.