Universal Control mouse and keyboard switching computer focus when using bluetooth keyboards

As I have started using Universal Control, I have noticed that sometimes the "focus" of which computer is active with the mouse / trackpad and keyboard unexpectedly. Has anyone else noticed this behavior, and found a fix for it?

Setup:

I use a keychron K1 keyboard connected to a 2019 MBP along with a Magic Trackpad 2 and a Logitech MX Anywhere 2S. I then am using Universal Control to also control a M1 Mac Mini sitting on my desk.

Details:

Sometimes this happens while I am generally typing (such as when writing this question here), while at other times it might be as I am navigating between browsers or tabs or things like that. In my testing so far it looks like sometimes when I press the Command, Control, Shift, or some of those other keys, the focus goes from my Mac Mini to the MBP. The strange thing is that it doesn't affect every device. I might be typing out a message and suddenly find the keyboard writing to the MBP while my Magic Trackpad is still showing that the focus is on my Mac Mini.

It also appears to just be focused around when I use the external K1 keyboard, as I haven't experienced any issues with the MBP keyboard as of yet. Unfortunately I generally work in such a way where the MBP is placed more in line with my other monitors and not necessarily within easy reach.