AirPod mic not registering immediately

I noticed that when I am recording myself or talking on the phone, it takes about 2-3 seconds for my AirPods to register that I am talking and to record the sound. Each time I fall silent and start talking again, it will not pick up the first 2-3 seconds of what I am saying.

I have tried resetting my AirPods, changing the mic to either side, resetting other settings on my phone, and nothing seems to help fix the issue.