You may want to work out how to hold the phone for the best hearing, I find that the speaker doesn't sit anywhere near where my ear hole is. The speaker is so weak that if it is misaligned then the caller can't be heard. If you have a case try without it, they can cause more issues with alignment too.
The mic is also very easy to cover with a finger, frankly I think you are better of jamming the earbuds into your ears to use the mic on the remote.
If you think it is a hardware fault take it to an Apple store for them to check but it may just be how it is.