Disable Siri calling anyone while keeping Siri enabled

I find that controlling airpods with iphone SE is very difficult.

Siri misunderstands my command to change volume levels constantly. Lately she's taken to calling colleagues in my contacts when I ask to turn up the volume. This is unacceptable. Is there a way to keep siri enabled (it's the only way, apparently, to control airpod volume without taking out my phone...) but disable her calling anyone?