Space_21,
When the Automatic Ear Detection is turned off, you'll need to be sure your AirPods are showing connected on your iPhone in Settings > Bluetooth.
If they are showing connected, and the issue continues be sure the motion is a press of the Force Sensor on the stem and not a tap.
You can also test a reset of your AirPods, and testing the controls when paired to a different device. How to reset your AirPods and AirPods Pro - Apple Support
Should the issue remain, reach out to Apple Support directly to further assist.
Contact - Official Apple Support
Contact Apple for support and service - Apple Support
Cheers.