When you’re using the built in microphone, if you step too far away, the noise reduction system will begin to filter you out as background noise. And the noisier it is around you, the closer you’ll have to be to the microphone for it not to filter you out. The built in microphone and noise reduction system is designed to focus the microphone pickup on the source right near it, and tries to filter out all other noise.
You’ll need to use a wireless headset or microphone if you want to move around and use your iPhone from a fixed spot.