iMac Speakers

I have an iMac and a secondary display with external speakers, I have my iMac hooked up to my secondary display via hdmi and when my computer goes to sleep the audio automatically switches to the secondary display and I have to keep switching it back; is there a way to prevent this so that my iMac speakers are the main ones being used all the time because it tends to get very annoying having to switch it all the time.