based on your troubleshooting... you hooked up another optical source and had success, it sounds like the optical output is bad on the Apple TV. no matter what bitrate the audio was encoded at, it the original source was PCM (standard stereo audio), then the output on the Apple TV will be "redbook" PCM 16-bit 44.1kHz, 100% compatible with your Yamaha or any other digital input equipped audio receiver.
1 thing to try, off the top of my head, is to reboot the Apple TV (put into standby, pull plug, restart) with the optical cable in the jack. perhaps it detects the presence of the cable at start up and otherwise mutes the output?