Just want to report that I too, running 10.6.5 on a MacBook Pro, had the mysterious " Audio Out Stuck On Optical Digital-Out Port" problem whereby I was unable to adjust the volume on my built-in speakers and they were permanently "muted" because the operating system was sending the audio signal to a non-existent device that it falsely thought was plugged in to my unused headphone jack hole. I thought at first the problem must necessarily be software-based because the built-in speakers worked fine when playing the start-up chimes. So I uninstalled something called "Soundflower" which does some mysterious shenanigans with commandeering the internal audio signal so that one can make audio recordings of one's computers' sound effects. But uninstalling Soundflower had no effect. So I uninstalled some other audio-related software, also to no avail. Then I zapped the PRAM, to no avail.
Then I decided to try what some people call "the toothpick trick," i.e. poking around in the headphone jack hole to see if something was stuck in there shortcircuiting it, and initiating the false signal to the OS.
Actually, I did a hyper-sophisticated version of the toothpick trick:
First, I own a home "otoscope," one of those doohickeys that doctors use to look into your ear. One can now buy amateur versions of otoscopes for home use, to looks for kids' ear infections, etc. (They only cost $30 or so.) Anyway, using the smallest attachment designed for babies' ears, I was able to look (with illumination) into the headphone jack hole and yessiree there was a big bunch of dust stuck way deep in there.
So then I took not a toothpick but instead a slender dental prong-pick thingie made of stainless steel, and very delicately was able to snag the clot of dust deep in the hole at the bottom on the first try. Quite a wad!
Then I took one of those rubber ear-bulbs, used for cleaning out your ears, and used it as a micro-vacuum-cleaner: Squeeze the rubber bulb til it is as empty as can be, then place the tip of the bulb at the entrance to the headphone jack hole, then suddenly release my grip: it ***** air out of the tiniest spaces! After doing this a few times, I then did the reverse and blew blasts of air into the hole.
After all this, I looked back into the hole with the otoscope, and it was sparkling clean! Not even the tiniest speck of dust in an area that just three minutes earlier was thick with dust.
I then turned my computer back on and...problem solved! The audio worked again like normal, and it was no longer detecting an imaginary "Optical Digital-Out Port" device. The internal speakers once again worked as normal, and the volume could be adjusted.
I guess the problem for me really was dust causing a short-circuit. It might very well be the same for you -- so try cleaning out the headphone jack hole as I did.
I'm writing this detailed description to help anyone else in the future with a similar problem.