AirPod Pros connect to my phone/MacMini when in the case

I have a pair of AirPod pro's connected to all my devices. While I'm on my MacMini and the AirPods are safely charging in the case, my Mac will all of a sudden switch audio from the computer speaker to the 'found' AirPods nearby. The case is firmly closed. Both AirPods are in the case charging. The case hasn't moved for an hour, but the Mac (and sometimes my iphone) will suddenly change audio sources. There is no specific pattern, it just happens sometimes. Is this a bug? I've rebooted, disconnected, reconnected.....HELP???