Ok I think I found what I was doing different in the settings.
When I exported to a .MOV file, I unticked Frame Reordering.
When I exported to a .MP4 file, there was no option to choose frame reordering, but an option to choose to restrict profiles to either Main, Baseline or both. I left it on Main.
Could this be why the .MP4 file looked better?
Thanks for your reply. I think it was because I had deselected Frame Reordering in the QuickTime Movie export. I did another test with Frame Reordering selected and I couldn't tell the difference between the two.
Which makes me think using Main Profile is like using frame reordering. The video is going straight to YouTube and it is recommended by YouTube to keep Frame Reordering unchecked. So I need to be sure before I upload it.
I did a Google search and found this page:
What are B slices? Are these the bidirectional frames that is used when frame reordering?
I think I found what you're after, trying to export an mp4, with h264 encoding (iMovie '11)
Share - Export using quicktime
Export: Movie to MPEG-4
File Format: MP4 (as opposed to "MP4 (ISMA)" which will be selected by default)
Video Format: H.264 is now available
Seems like a lot of steps, for what I would have expected to be a common use-case ....