I want to add my thanks for your post: it solved my problems too.
The only thing I find now is that I need to leave the MIDI app open. When I want to play video in iTunes or DVD Player app, I need to ensure I have 48 kHz selected. I often find the channels will automatically go to encoded digital audio. I need to also select the surround sound option from the speech-bubble on the bottom menu bar in the iTunes playback window. Failing to do both has produce a few crashes in iTunes.
If I want to play music, I need to select 44 kHz.
(I wish Apple would fix this once and for all. I can work this okay, but it is still not ready for mainstream home use in my view. Neither my wife or kids can work this. Until then, the PS3 with its ability to just put a disc in and press 'play' will continue to win out.)
But at least I can use iTunes for playback. It's now beginning to be a worthwhile media management tool. Cheers!
I want to share an update in case it helps anyone else.
I had downloaded and installed the Perian plug-in. (It installs in the preferences pane in the Mac's system settings.)
In previous versions of OS X, Perian was a sometimes essential to getting 5.1 surround from a Mac. I know I used it with great success in OS X 10.5 and 10.6.
However, I was having real troubles — some of them as I discussed above — with my sound preferences in 10.8.2.
After much mucking around, I stumbled upon the solution.
I removed Perian.
I know have no trouble at all. I use either Handbrake, and more recently iVi Pro, to convert movies for iTunes. With Perian removed, the only change I need to make in some movies is to select the surround sound track. Mac OS X will automatically change the settings if necessary in the MIDI player, and my Marantz NR 1603 does the rest.
If I play music, OS X changes the settings back to stereo, with no intervention.
The only setup that might be necessary for anyone else is to ensure that your MIDI player is set for your HDMI or optical audio (TOSLINK) device to use 48kHz.
I have gotten the 5.1 surround sound through HDMI connected to my Pioneer AVR. However, there is audible noise, continuous hissing sound, through my speakers. Did you face the same problem?
I have read here that the reason is because because of the difference between audio signals from a computer and other consumer electronic products. One of the suggestion is to connect the HDMI from Mac Mini direct to the TV/projector and toslink cable between mac mini and AVR.
I have not tried the suggested solutions. But I would like to know if anyone else is getting the noise out of their speakers when connecting the mac mini to the AVR?