How to disable Hey Siri on AirPods Pro only

More often than not when I have my AirPod Pros in my ear, and I do not want to answer a phone call, I simple say nothing. However, if I am in the middle of a conversation, now and then the phone answers itself. Somehow Hey Siri thinks I asked it to answer, but I never used the words Hey Siri. Worse yet, if I am silent, but I am in a noisy environment, the phone answers itself. It must hear something in the ambient noise that tells it to answer the call. Drives me absolutely nuts. Especially when I do not want to talk to the person calling me.

Is there a way to disable Hey Siri on my Airpods only. I don't mind it on my phone as I never have this problem with it. This problem is only when I am wearing the pods.