Airpods mic picking up background noise

When I use my Airpod Pros while taking a call, I often get complaints from the person on the other end that they can hear every little background noise, sometimes even more so than my voice. I can't do anything other than sit in silence to take a call on them, or else the person on the other end of the line asks me to switch back to just using my phone. I bought them to be hands free while doing other tasks and it's impossible. I have been trying to find settings to adjust the mic pickup and can't. Is there a way to reduce background noise?

To specify - this has nothing to do with transparency mode vs. noise cancellation. The complaint is not about anything I am hearing, it's the person on the other end of the call.