This just continues to be very strange.
After all the restarts I did, it has changed it's behavior, but still doesn't work properly, just differently.
When plugged into the dock it came with I get nothing. No output, no recognition from the thing that it was docked at all.
When plugged into an iPod video dock (with the little plastic piece that makes it fit nice removed so that the phone will fit in it) it works. I get the "Do you want to turn on airport mode" message and then whatever I select, on or off, the audio output on the dock works! (oh and the wireless remote control works too)
When plugged into my Belkin car charger/lineout device I now get the airplane mode question, but whatever I select no audio goes out.
So the only dock it will work in currently is my old iPod video dock. Even the dock it came with doesn't manage to make line out.
Makes no sense to me yet. I think I'll call apple support later on and see if they have a suggestion. Be a shame to have to send it back in. I hope I can put the SIM card in my old phone to keep having a phone while they fix it.
The fact that the various resets have changed the behavior now makes me think that it's software anyway.