Airpods Pro Spatial Audio Re-centers Automatically

While working, I often have my iPad off to my right side playing a movie, TV show, or random videos.

When I first start it up, it works fine. The audio sounds like it's coming from the device off to my right. It sounds amazing. When I turn my head and look at the screen, it does a great job of keeping the positioning in alignment to the screen.

However, after working for a few minutes and not looking at the screen, the "alignment" slowly shifts and automatically re-adjusts. I guess it thinks it's not in alignment since my head is constantly facing away from the screen. So, then the sound is coming from in front of me (even though the iPad is off to the side), and when I look at the screen it's as if I've turned AWAY from the screen (since it's recalibrated it's alignment).

This is super annoying. Is there a way to only calibrate when starting a video, and not have it recalibrate the positioning automatically??