I'm pretty sure it has to be h.264 but i thought the appletv preset on handbrake defaulted to that. in my opinion h.264 is the best of the codecs anyway as far as quality/compression goes. i really have no desire to use any others.
be sure to let us know because i'd be very curious if it didn't work after encoding using h.264