Are you sure the problem isn't in the source files that you're playing? What player application are you using?
Try connecting headphones to the computer and see if the sound is clear. If the problem remains, you now know that the issue is inside the computer.
Try a pair of headphones in the headphone jack of the Mini. If it is clean, then try another sound source connected to the stereo via that same cable (such as an ipod). Is that clean also? If it is, then the problem is specifically happening when the stereo and the computer are joined together.
This could be due to interference of some sort. Make sure the cable is not running near other cables, or near wireless/wifi sources. You should have wireless devices like wireless routers, cordless phones, cell phones, 2-way radios, etc. at least 6 feet away.
When connected to the stereo, make sure the MacMini's sound volume is at maximum at all times. Always adjust the listening level on the stereo. This will help the audio signal to overcome noise and interference.