how to get AirPods to resume play on watch not phone

Im having an annoying issue with my air pods.

I like to film my workouts - but I like to listen to music at the same time. So I have been filming on my iPhone (X) and play music from my watch which works fine.

however - if I double tap the pods to pause, then double tap to play again, instead of resuming what I was listening to on my watch, they play from my iPhone, which stops any recording I’m doing and also plays a completely different song/ playlist!

any tips on how to solve this? Except I guess press play on the watch face, but I got AirPods for a reason and I’d like to just be able to pause and resume from there.