Airpods 4 microphone poor on calls

I have the AirPods 4 with ANC and the sound is close to amazing for music etc. However, when I am making calls the person at the other end complains I sound like I am far away from the phone and the clarity of my voice isn't clear. I then use the phone mic/speaker and they can hear me fine.

I have tested it in the voice memo apps and have the same issue. Using the phone mic the recording is loud and crystal clear, using the AirPods, it is faint, robotic sounding, and speech clarity is poor.

Does anyone else have this issue and know how to fix it? I have tried unpairing and restarting my iPhone (14 pro max running iOS 18).