I generally only wear one air pod pro 2 at a time. I keep having trouble when I go to switch to the other one that it isnt working. I

I generally only wear one air pod 2 at a time. I keep having trouble that when I go to use the other one that has been in the charger it doesnt recognize it is in my ear. I have to do forget the my device and re pair itto get it to work. I have to do this multiple time a day. Any thoughts?