Podcasts fast forwarding or rewinding when I tilt or move phone

When I wear headphones, the Apple podcasts app is suddenly responding to how my phone moves. It fast forwards or rewinds very quickly if the phone tilts left or right, and it will stop or start playing if I raise or lower my phone, and the Siri circle will pop up (even though I have Siri turned off for everything). So now if I go for a walk and try to listen to a podcast, having my phone in my pocket at any angle causes complete chaos.

It doesn't happen when I am listening on a Bluetooth speaker, even when my phone is in my pocket and I'm moving around. It only seems to be with headphones, and it'll do it whether the phone is locked or unlocked.

I can't find anything in the podcast app settings or my phone settings to turn this off. It started about a week ago and it's making me crazy. I've tried Googling but haven't found a solution. Is anyone having a similar issue?