Slow Keyboard Response

Almost every day the input on my keyboard slows to the point where it takes more than a second to record each of the keys I am typing. The mouse response and all other computer response is normal. If I reboot the problem goes away, but comes back the next day. I have a Mac Mini (M1, 2020) and I am running macOS Big Sur with a Magic Keyboard, model A1644 EMC 2815.

My keyboard preferences are set to normal response time. I have also tried the bluetooth factory reset fix, but that only works temporarily, just like the reboot.

This issue only started when I replaced my iMac with this new Mac Mini in March. I have also swapped magic keyboards, but the issue has not gone away.