FWIW. iTunes uses a QuickTime CODEC to convert audio files to Apple Lossless, and then, uses AirTunes to send them to the AX. In turn, the AX uses built-in software that converts the Apple Lossless to an Encoded Digital Audio format. From there, digital audio is sent to a optical transceiver to convert the electrical signal to an optical one before sending it to the innermost part of the audio port. For analog, the AX has a built-in DAC to convert the Encoded Digital Audio to Analog which is sent to the same audio port.
Since, you are getting a good analog output, you know at least the DAC is working properly. Unfortunately, it does still sound like there is something amiss with the optical transceiver in the AX ... or the calibration of the digital signal between the AX and your stereo. If your stereo has multiple digital inputs, I would suggest (if you haven't tried already) is to use a different port.
... and you may still want to try another cable. I had a similar issue between my MacMini and Logitech speaker system that perplexed me until I tried another TOSlink cable. Trust me, I triple-checked the original cable, which did work with other equipment, but just not between these two.