Same problem as the OP, except that I have a separate cable for each input. Also, I sometimes VNC into the mac mini from my iPad.
I want to leave the computer on, and just switch the TV input when I need to use it. However, if I switch the TV input while the mini is on, it no longer recognizes the display and switches to the "internal display". Then when I change the TV display back to the (still plugged-in) HDMI cable that is connected to the mini, it says "No signal detected".
It seems to me that this is a problem with OS X (Yosemite) cdmi handshake protocol that requires two fixes. One, it should be able to maintain the tv profile even when the TV is using a different input and send a handshake request on mouse/keyboard activity. Two, It also is not responding to handshake requests from the TV, making it practically irrecoverable (at least from the sofa).
I was unable to sleep the computer from VNC, so I restarted the computer and the restart activated the cdmi port on the mac mini. Since the TV was utilizing the mini's hdmi input, the mini got a good handshake response. However, the sound output was still set to the internal speaker 😟
This issue makes the mini pretty lame as a HTPC, since you have to manually sleep it before changing inputs and ensure that the correct input is selected before waking it. This is very impractical for using the computer during commercial breaks, for instance.
Matt