Same with my M1 Max Mac Studio. I tried turning off/on BT and using a different mouse but the same problem persist. The same mouse would work fine on my 2014 Intel iMac. I do note that the M1 Mac Mini and M1 24 inch iMac in my house both have a ~ 3 second lag upon initial connection. I note similar issue with the Mac mini and 24 inch iMac at the local Apple Store. So, M1 Macs do have a delay in BT connection.
Back to my M1 Max Mac Studio. I reinstalled the OS but the same problem persist. I do note that if you click the mouse, the mouse would then connect in about 3-4 seconds. It seems the computer is waiting for a BT signal before it wakes up. I worked with Apple's senior advisors several times over the phone but they can't fix it either and send the info to the engineers higher up. By that point, I was fed up with it and returned the Mac Studio to the local Apple Store.
I bought another M1 Max Mac Studio just now, hoping that it was an isolated issue. Guess what!! Same problem! Without stimulating the computer with a click, the connection lag can be over one minute!! That's when I realized that I better do a different search online to see if others share the same issue. It seems this is inherent with Mac Studio.