TOSLink is the name for the optical cable, there is a adapter to convert the stereo mini optical to a regular TOSlink which then connects to your receiver which can process 5.1 sound.
The Mac only passes 5.1 through, it doesn't process it.
So to get it you must have something or play something encoded in a surround sound format, like Dolby on a DVD movie for instance, which the Mac passes through to your receiver for decoding and playback on the 5.1 system (or 7.1)
Naturally your receiver has to be able to decode the surround sound format.
What's nice if you have a good receiver, that it will convert the 2.0 of iTunes music into surround "sounding-like" sound (not true 5.1), with the sub-woofer taking those bass channels for a much richer, full room sounding playback experience.
You need to enable the optical output in the Sound System Preferences.