Universal Control - mouse & keyboard stop working when safari tab sharing shows up in Dock

I have two Mac minis, both on the latest Sonoma (14.4.1) and I'm using Universal Control between the two to share the Mouse and Keyboard from one system to the other. It's magic and I love it.

However on the "shared to" system I'll be working along and the mouse and keyboard will just drop the connection. Over time I have come to realize that this is happening at the moment that the secondary Mac is getting a Safari tab suggestion from the other system (you can see it pop up in the Dock and then the mouse and keyboard go dead on that Mac).

I have dug around and wouldn't mind disabling the secondary Mini's ability to receive those Safari tab notifications altogether, but I can't find a place where that is doable. It seems like Handoff itself is either on or off and with it comes all the features or not.

Is there a place that lets you disable Safari specific window/tab handoff? has anyone else seen this? It's a prolific issue for me and happens DOZENS of times a day. TIA.