Skip to last paragraph for my fix.
I had this same problem today. My phone was charging ok, and even in the car the phone would charge and also play on the car speakers while plugged into the usb. However, when I tried to plug in headphones or plug into a non-charging external audio device, the music would only play on the phone speakers. The first answer didn't make sense to me because if there was debris in the port, it most likely wouldn't be charging either. I still tried it though, just for the sake of checking it off the fix list. It didn't work. I tried to restart the phone, which seems to be a more logical fix, but it didn't work either. I tried this a few times.
The official apple support page "If your iPhone is stuck in headphone mode - Apple Support" indicates that if "After you plug any headphones into your iOS device, sound plays from your device's speakers instead of the headphones" then you should just go ahead take your phone into the store.
However, I noticed that a new update was available. After installing the update, the headphones are now working.