Apple's "optical" audio port is standard Toslink which you need a Toslink stereo mini to regular Toslink adapter and optical cable. It provides a cleaner, interference free (unaffected by magnetic waves) transfer method of high quality audio.
All channels pass through the single combination stereo mini/optical port and designed for a high end home theater system etc., that accepts Toslink optical inputs.
Mac's don't do 4.1, 5.1 or 7.1 processing by themselves, only pass the channels through, perhaps on the MacPro one can buy a internal audio card and your 4.1 device could work correctly. What your doing with your analog splitter is basically playing two channels to four speakers, it's not true "surround sound."
With the right cables and adapter, if your playing a DVD movie that so happens to have extra features like 5.1 surround sound, you can pass this through your Mac and to your home theater system. You need to match the format of the 5.1 on the DVD to the format the home theater receiver can decode. There is Dolby 5.1 and other 5.1 formats...so picking a quality receiver with a lot of surround sound decoding formats is important.
When you play 3D video games, most have some kind of pass through to a sound card on a PC which then decodes and passes the signal to something like what your describing.
What you need to experience quality sound and true surround sound is a quality receiver that does it's own 5.1 - 7.1 decoding. A quality receiver can also take 2.1 stereo recorded from iTunes Store music, passed through your Mac's optical audio port and then perform some magic on it for a 5.1 system. For instance taking the bass and sending those frequencies to the sub-woofer, taking the two stereo channels and sending to the rear speakers and the mid range, add "theater" effects and so forth.
You sound to me like a budding audiophile, you got one part already accomplished, the Mac and high quality iTunes music, you just need the quality audio system and the Toslink adapter and cable. 😉
http://www.harmankardon.com/EN-US/Products/Pages/MarketGroup.aspx?MID=HOM
http://www.amazon.com/Toslink-To-Optical-Mini-Adapter/dp/B0002MQGRM
http://www.amazon.com/Toslink-Digital-Optical-Audio-Cable/dp/B0002MQGOK/ref=pd_b xgy_e_text_c
(No comp)