How can my Airpods be connected to my phone but NOT my iPad or AppleWatch?

It seems like my AirPods are always trying to figure out which device I want them to connect to, but the reality is the answer is always "my phone".

For example, if I'm walking along and listening to music on my iPhone and want to use my watch to make a reminder for myself, I want to talk to my watch, but when I bring up Siri on my watch it does this slow switch to my AirPods and not only turns off my music, but also doesn't work right to make a reminder because the sound is delayed.

Or when my kid is trying to watch a video on my iPad it will turn off my music or audio book on my AirPods and I'll start hearing his video.

I don't think I've intentionally used an AirPods with my watch or my iPad in 10 months. I've tried disconnecting the AirPods from both devices, but when I do it also disconnects it from my phone.

Is there any way around this? Can I have the AirPods *not* sync in the cloud? or turn off specific devices for it?