FYI - the mp2 sample clip you provided will not play either on my MacBook. Apparently the QT 10 player does not have the mp2 codec included (under Snow Leopard) although the Apple support site hints that is should 😐.
Then the problem is with your computer. Snow Leopard Install includes support for muxed MPEG-2/MP2 files as specified above. The file I posted is natively compatible with QT X v10.0. Further it is also compatible with the QT 7 Player is you have the QT 7 Playback component installed. If the QT X component is missing from your system then it must have been removed. If still present, then you likely have some sort of a software conflict. The OS installed component only supports the QT X player and the add-on Apple component only supports the QT 7 "classic" apps. Here is a screen capture of the posted file playing on my MacBook Pro under Snow Leopard. Excuse the artifacts, low frame rate, and broken audio but remember this old platform was both playing the full DH recording and simultaneously capturing the entire 1920x1200 display screen.
ScreenFlov.mov
The mp4 shows as H264, 1920x1080, millions, AAC, 2-ch, 48KHz, 29.97fps
QT X v10.0 should play all "standard" H.264/AAC supported Profile and Level combinations. It does not support proprietary and/or hybrid codecs which exceed these specifications. At this point I must ask, are you really specifically interested in MPEG-2 files or simply want to create files compatible with the QT X v10.0 player?
The mov shows as DV 720x480 (853x480) millions, 16-bit (little endian), stereo, 48KHz, 29.97fps
This is a standard format. It's main advantage is it is easily edited. Its main disadvantage is that it is a relatively low compression format making for somwhat large files (e.g., on the order of 13 GB per hour for DV25 SD content whether in 4:3 or 16:9 aspect).
QT player opens the mp4 in a full-screen window (on my HD display), whereas it initially opens the mov in a much smaller 853x480 window.
Not shure what point you are trying to make here. Are you referring to an "HD display" connected to tour computer as opposed to the built in display or an extenal device which automatically applies scaling to the content? Files can contain embedded settings to open at "full screen" just as some players can be set to open content as a "full screen" display or be manually sized to "full", "normal" or "fit" screen modes.
When enlarging it to comparable full-screen size, the resolution looks similarly bad to the mp4 (as far as I can discern).
Your comparison work flow is reversed. If you wish to compare the video quality, shrink the larger file down to the size of the DV file and then make your evaluation. Doubling the size of your DV file merely means it takes four pixels on the display device to represent each pixel at the original resolution—i.e., introducing a pixelation effect. Basically, you are trying to compare apples and oranges here. To make some sort of a valid comparison here, we need to know things like the data rate, profile, level, entropy coding mode, etc. to compare the H.264 coded file against the DV format which has a more or less fixed or standardized encoding format. For instance, the DV NTSC codec only encodes to a 720x480 encode matrix which can play as either a widescreen (853x480) or fullscreen (640x480) file. If you want to export DV to HD dimensions, you would need to export using the HDV or one of the DVCPRO HD codecs. As to the H.264 codec, it is the best codec for you to use here but you must use the proper settings. At this point it does not sound like you know how to select these settings if, in fact, they are available. I am not an Adobe Premier user, so I don't know if you are using a predefined setting or manually entering your settings. For instance, H.264 is very scaleable—suitible for data rates compatible with older 3G cell phones up to and including BD content. I can, for instance encode a 1080p H.264 at 2.5 Mbps, 12.0 Mbps, or anything in between. If two files are encoded from the same source file and one is encoded at 2.5 Mbps and the other at 12.0 Mbps, then the file with the higher video data rate would obviously have the potential for better quality.
Again, both files were rendered by Adobe from a 1080i (1440x1080) clip that was extracted from my Sony camcorder (which is another issue altogether).
Are your camcorder files and/or the Premier exported files really MPEG-2 or HDV? If HDV, that would explain their incompatibility with the QT X Player. HDV is a "Pro" codec installed by iMovie HD, Final Cut Express, or Final Cut Pro installations but not supported natively by the basic OS codec installation. Current versions of iMovie automatically convert HDV to AIC/PCM MOV files for video editing.