Hi kai.dean,
Thank you for posting in Apple Support Communities.
It sounds like your mouse lags when holding down any keys on your M1 Mac mini. We'll be glad to assist you.
Try testing the mouse while in safe mode. Safe mode prevents your Mac from loading certain software as it starts up, including login items, system extensions not required by macOS, completes a basic check of your startup disk, and clears cache. How to use safe mode on your Mac - Apple Support
If you're continuing to experience the same thing in safe mode, set up a new administrator account on your Mac and test. Set up users, guests, and groups on Mac - Apple Support
Have a great day!