Can I auto-connect two pairs of AirPods (Pro) to Apple TV 4K?

We've successfully been able to use two pairs of AirPods (Pro) with our Apple TV 4K so my wife I and and I can watch TV at night without disturbing our children in nearby bedrooms.

However, I haven't been able to get both pairs of AirPods Pro to connect automatically at the start of the Apple TV session. The Apple TV 4K auto-connects to the first pair of AirPods (whomever puts them in their ears first), then disconnects that pair and auto-connects to the second pair when they are inserted by the other person. Then we have to go to the Apple TV AirPlay menu (not AirPods!?) and then manually connect the second pair.

I even went so far as to add my wife's AirPods Pro to my Apple ID (after removing them completely from her account), but that didn't improve the situation.

Apple often handles these types of scenarios well, so I'm hoping I'm missing something.

Any solutions/suggestions appreciated!