How can I prevent AirPods from switching devices?

I have an iPhone 15 Pro Max and an iPad Pro 12.9" 2018 and both are up to date with the latest software. The problem I am having is that I will be listening to music on my phone with my Airpods Pro while working on my iPad with a pencil and the audio will stop on my iPhone and the Airpods Pro will connect to the iPad. This is so very annoying. I have no idea if there are settings I can change to stop this from happening. I cannot turn off bluetooth on my iPad because I need it for the pencil I am using. Any suggestions would be great.

[Re-Titled by Moderator]