I had this same problem with the original iPhone. The only way I was able to resolve this was to continually plug & unplug the headphones until it worked, sometimes slowly & sometimes rapidly. I've had a 3G for sometime now & have not had it happen at all.
I'm having the same problem. I have read on a number of forums that plugging and unplugging headphones rapidly solves the issue. I'm yet to be convinced. A friend of mine tells me that if the phone gets too hot then it trips the headphone mode on. There experience was that the fault was current when they visited their local genius bar for help. The advisor took the phone into the back for about 20 mins. He re-imerged saying that there was some debris stuck in the socket. He took the phone away and within 3 hours the same problem happened again. I guess this problem is one that Apple won't own up to and have no fix for. I really wished I had opted for the 3GS as this is my 3rd iPhone 3G since November 2009!!
My iPhone is stuck in the headphone mode as well. I've reset the settings, plugged a headphone in and out slowly and quickly and upgraded to the new software. There doesn't appear to be any debris in the jack.
I brought it into an Apple Store and of course it started to work. This after being told I would have to wait 30-45 minutes past my scheduled reservation.
Ive only had this happen a couple of times with my iPhone 3G, and never as severe as what you're describing.
For me, shutting the phone off (holding the sleep/wake button on top and then slide to shut off) and then turning it back on worked.
Also, you could try connecting a bluetooth headset. This might switch it from headphones to bluetooth mode. That way, maybe when you disconnect the bluetooth, it might go back to normal mode.