I don't have any of those connections, or the ability to. Just don't have the cables. For me, the Sound selection is separated...
Since I have a headphone plugged into the port. I can not choose Display Audio via USB, which belongs to the Apple Cinema display. If I were to try to select it, the blue highlight will bounce back to the headphones. However, I still can select either the AppleTV or VSX-53 avr. I rarely do though, no point for me.
However like I've said I've used the VSX-53 to stream my iTunes music. (But this isn't your problem, right?)
In iTunes I can select just one, in this case the Mac Mini, or all three devices. When I select the VSX-53, if it's off it'll switch ON and tune to the proper input. If ON, it'll still tune into the proper input. If I were to enable only the VSX-53, I still have the ability to turn the AVR off when I am done and what not. iTunes pauses the music and no more action is taken. When I come back home or whatever, I press play. The VSX-53 turns on and tunes to the proper input. (This is what I meant by my previous post).
With that said. I just went to my Sound settings and select the VSX-53 as my main audio producer for the entire Mac Mini, not just iTunes. I am able to control the volume up/down as well as mute via my keyboard, just as if nothing changed. My AVR shows the volume change as well as the mute.
Lets say nothing is playing, no music, video or whatever. I change inputs on the AVR or turn it off. The Mac Mini doesn't change to my headphones. It'll stay on the Airplay, so as soon as I press play for iTunes. My AVR will automatically tune in. If however something is playing, like iTunes, and I switch off the AVR or switch inputs. The Mac Mini will default back to the headphone jack.
Hopefully that gave you some ideas on what is going on. It also wouldn't be a bad idea to switch over to using the headphone jack. Rather than the USB to digital to DA or the digital output to the device that accepts the ditial input. At least try that for testing to see if the problem goes away.
KOT