Universal Control 2 Mac Mini's stability

I have an M1 and M2 Mac mini, with a single screen (I accept that input switching is probably the only way for now). Universal control from experience, doesn't treat the keyboard and trackpad as a pair, as the keyboard is connected to M1 bluetooth, and trackpad on M2 bluetooth, so whilst I write this on the M1, the cursor has switched back over to the M2 immediatly and is sat idly in the centre of the screen. To get the cursor back again to the M1, I have to move the cursor to the screen edge. Sometimes the devices are connected to the same device on bluetooth, and when controlling the other device where the input is shared via universal control, they just switch back to the main device randomly, forcing me to drag the cursor back over again

If I reboot one of the Macs, I have to plug the cable into the trackpad/keyboard (both Apple magic devices), to be able to get past the login screen, so universal control seems to have somewhat control.

I'm 3 feet away from the WIFI router running 5Ghz(both Macs are vertically stacked), so I don't believe it could be network related, and I'm seeing 600Mbps+ running tests. CPU is mostly idle.

Has anyone else experienced a really flakey universal control experience, or is is universal control really just a really unpolished feature.

I have tried all combinations between Macs on how reconnections are handled, currently both set to automatically reconnect to nearest Mac. Nothing seems to improve the situation.

Appreciate any insight people may have