Stereo has 2 channels, 5.1 surround sound has six channels total.
The Mac passes 5.1 through the optical cable/port.
The audio has to have a 5.1 encoding of some sort, Dolby, Logic etc.,
Your receiver has to be set to decode the 5.1 format being sent to it.
You can test this with a DVD that has extra sound capabilities in the audio menu, then match the encoding on the receiver.
Some 3D games provide surround sound on a Mac, but usually it's to a PC sound card that handles the processing. Since Mac doesn't do that, it just passes the surround sound through, the surround sound must be in a encoding format your receiver can understand.
Some good receivers can take Stereo 2 channel and separate out some of the frequencies and send them to the appropriate speaker, like the sub woofer and mid range. Then take the left and right channels and duplicate them for the front and rear left and right channels to fill a room with sound. However this isn't true surround sound as you won't hear objects moving up behind you or flying past over your head like in a good movie or video game with true surround sound abilities.
Your lucky I was here, not many people mess with surround sound on a Mac, or really don't even know it's there.
But it's really worth the effort, even iTunes music sounds great. A iPod sounds like crap in comparison.
