Brand new Magic Mouse has problems clicking

I just received my new iMac with the fancy M1 chip and the Magic Mouse has issues clicking on stuff. Every so often whenever I click on something, the page moves a bit? I even have problems clicking on the right spot because the page shifts while I'm clicking. Then after I finally can click on something, the page shifts a few inches from where I clicked. I even connected my wired mouse to this iMac and it doesn't happen as often, but still moves the page while clicking.

Anyone having these issues?