I'm pretty certain AppleTV does one of the following with audio:
- converts a known supported codec (WAV/AIFF/AAC/MP3 etc), to PCM to output via the optical digital link
- in some circumstances outputs a 'raw bitstream' which an AV amp will recognise as either Dolby Digital (2 or 5.1 channel) or DTS
I'm also pretty certain that it does not output the 'raw bistream' for the other formats above as most AV amps will not know how to decode them - hence it converts to a generic PCM 2 channel bitstream which most amps recognise.
My Yamaha processor only handles PCM/Dobly digital/DTS. It has no support for raw mp3/AAC bitstreams but plays back via optical as these formats are decoded by AppleTV and output to optical as standard PCM bitstreams.
If AppleTV recognises a format it will do the conversion to PCM - if not it may in certain circumstance just passthrough the raw bitstream, so getting it to do that is the trick to AC3/DTS playback.
See:
http://web.mac.com/ragboy/iWeb/TheGarcias/Dad's%20Blog/9E500240-D005-499D-9E44-74B50E96FE4E.html
AC