Piercen, I'm experiencing the same problem with my Black iPhone 4S 16Gb.
When I make a call I can see that the call has connected by the call timer. There is no audible ring through my speaker and I can't hear the person I've called speaking. When I tap the Speaker Phone, I can hear the person, tap it again and the phone returns to normal audio through the speaker. The next time I dial the same number no problem, normal audio.
I spoke with AppleCare and backed-up the phone through iTunes and returned the phone to Factory Settings and installed from back-up. Still no joy, same problem. Then I thought I'd Reset the Network (Orange UK), still the same problem.
I phoned AppleCare and they initially asked me to visit an Apple Shop, which is a 1 hour car journey from home and not at all convenient, so I asked for an Express Exchange and will receive a replacement iPhone 4S in 2-3 working days. Hopefully the new handset will be problem free!