Microsoft Remote Desktop and iPad Sidecar lag

I have a MacBook Pro 16'' with the M1 Max 32-core GPU and I'm using the iPad Air (4th Gen) as the sidecar. When using the Microsoft Remote Desktop app it will lag pretty bad. This lag will even happen if I choose to use a single screen for the remote connection and have the sidecar being used for nothing. As soon as I turn sidecar off and restart the remote connection the lag is gone. The lag isn't coming from the server as I have tested it out with dual monitors on another computer and it works fine. I also had a MacBook Air with the M1 and the same exact issue was happening on that.