No, I don't think there's any problem with your setup. I think what you're seeing is just ordinary limitations of your electronics. It sounds like you are using external, powered speakers? If so, then what you've really got are two amplifiers in a row. First, your computer has a sound amplifier built in, and you use the iTunes volume control (and/or the computer volume control) to adjust the level of the sound from that amplifier. Then, you feed that into the input of your external speakers, which have their own amplifier. When you have the iTunes/computer volume set up too high, you're 'over-loading' the amplifier circuit on your external speakers, and the 'muddy' sound results. This is just the way the electronics behave -- an amplifier circuit expects the input signals to be within a certain range of 'strength', and don't respond well when the signals are either too strong or too weak. You'll get bad sound quality as well if you do the opposite -- have the computer volume turned way down near the bottom of its range, and then try to amplify that tiny signal way up with the speaker's amplifier. Often in this case, you might get some annoying noise - either a hiss, or a hum, because in addition to amplifying the quiet music, you're also boosting up some noise that is found in all circuitry and wiring, but is ordinarily much quieter than the music.
The usual recommendation for when you have multiple amplifiers in series, and it sounds like that's what you have, is pretty much what you have found out with your own experimenting. Keep each amplifier (volume control) in the middle range, and do the adjusting with the final volume control in the chain, which would be the speakers, in your case. Then, the amplifiers are handling the signal levels that they are best designed for. The exact best settings may take some experimentation; it may be more like 1/4 or 3/4 setting, but it sounds like you've figured out.
(This was my best attempt to explain some signal and amplifier theory of electical engineering without using too much jargon.)
G4 Dual 1 GHz MDD, PowerBook G4 Mac OS X (10.3.9)
