I don't have a PS3 to test, but here is a copy of a very old post from Dan Hawkins. In iMovie, you would use
SHARE/EXPORT USING QUICKTIME. Choose Movie to MPEG4. Then choose h.264 codec under video options.
Keep in mind that the PS3 may have changed since then, but this should get you on the right track.
==================
from Dan Hawkins:
I am using a PS3 to preview HD video edited in FCE-3.5.1HD with the following QT-PRO export settings:
1) export from FCE using Quicktime conversion
2) select format MPEG-4
3) select options:
a) under file format select MP4 (not MP4 ISMA)
b) under video tab select H.264
data rate in the range of about 10000 to 18000 kbits/sec depending on what filesize/quality you are shooting for
image size---select 1280X720HD or 1920X1080HD
frame rate---select 29.97 or "current" if already on a editing timeline of that type
key frame---select either automatic or every 24 frames
video options---select "MAIN" profile and single pass encoding (when I use "MULTI-PASS" encoding I get a "FILE UNREADABLE" error message on my PS3)
c) under audio tab select default AAC-LC
d) under streaming tab select default "not enabled"
I transfer the resulting high definition .MP4 file to the PS3 by one of the following methods:
ethernet network connection
iPod/thumb drive
SD-card
I have to hit the "alternate menu" under the PS3 video section and select "display all files" to see the video file and then save it to the local PS3 HDD (some dumb programming there!).
I have heard there might be an upper limit of 4Gb on the file size (if so, another dumb move), but have not come across that limitation yet (I just got the PS3 about a week ago and am working through some of the same issues you seem to be struggling with).
I think that some of the confusion regarding the PS3 relative to playing back HD from regular DVD-R is that the PS3 will hardware decode a properly encoded .MP4 HD video file from whatever media it is presented on and as such does not require a HD "authoring" container with menus and such. These so called "HD authoring tools" are currently in a pretty confused state at the moment and are the source of much incompatibility with many HD playback devices, but the PS3 bypasses all of that and plays the file directly through hardware decoding.