no optical audio when HDMI tv is attached

Mac mini -> HDMI -> TV

Mac mini -> toslink -> ???

Can you clarify what the other device is there — is it the same TV, or a separate audio system? Is there also an HDMI connection to the audio system (some more complex home theatre systems have HDMI inputs as well)?

Did you change/update something on your Mac recently? If not, I would be inclined to place blame on your other equipment, either a loose connector, a changed input selection on your TV/audio system, etc.

You're right, that sounds kind of like the Mac is switching its audio input to HDMI as soon as you plug in HDMI, and then back to optical as soon as HDMI is unplugged. Does the TV have built-in speakers you could turn up to verify that hypothesis?

If it's still behaving strangely, I would be curious to see if the Mac does the same thing on a new user account. You can easily create a guest account by going to System Preferences > Users & Groups. Click the Guest User account in the left-hand sidebar, and check off the box beside "Allow guests to log in to this computer." To switch to the guest accout, just log out and click the guest account on the login screen. Note that when you log out of the guest account, it will warn you that anything you did in that account will be deleted. It won't affect your personal account.

Go to: Applications > Utilities > open Audio MIDI Setup and try selecting the desired audio output from there.

You're welcome, glad to hear that you found the problem.