Have you read this article before?
http://www.apple.com/quicktime/tutorials/h264.html
There are some guidelines there for frame sizes and bitrates, that you might want to have a look at.
Keep in mind that the frame rate of your video also affects bitrate requirements. Robert Reinhardt (the guy I linked to in my last paragraph in my previous post) has made a great bitrate calculator for both Flash and AVC/H.264:
http://www.adobe.com/devnet/flash/apps/flvbitratecalculator/ (remember to change the video codec setting to H.264, if that is your output format)
Have you tried activating frame controls in Compressor? That should give you better quality downscaling. If you would like to save some time, you could try this tip (same principle for downscaling):
http://discussions.apple.com/message.jspa?messageID=7309534#7309534
Video compression, especially for web, is all about testing to find the optimal settings for best possible quality/filesize ratio. Remember that it will save you a lot of time doing some short 10-20 second tests (just set in/out points in the Compressor Preview window), before encoding everything.
Try changing your resoultion to either 1024x576 or 768x432 (or any other resolution divisible by 16, 8 or 4), activate frame controls, adjust the video bitrate according to frame size and frame rate, and see if the result looks better.
It might be that the version of of
x264 that VisualHub uses, is a bit better than the H.264 codec Apple provides in QuickTime/Compressor (at the moment). I really do not know.
Have you tried
DV Kitchen from DVcreators? That application have something they call
SampleLab, which makes it very easy to test different video bitrates. You can also use x264:
http://www.dvcreators.net/how-do-i-install-the-x264-codec-for-dv-kitchen/
I guess I would some what agree with Josh, that the human eye might not notice the difference. I would still recommend choosing a "safe" frame size though, just to make sure you do not get any strange artifacts (like green lines at the bottom or right side of your frame), and to make sure your video plays as smoothly as possible.