Mac Mini - is it possible to use different audio out options for different apps?

I have my mini and monitor in one corner of a room and have also connected up my mini to my TV and amp (other side of the room) using the HDMI and optical out and the sound plays ok through the amp. How can I set this up so that when I am not using Plex the sound comes through the built-in speaker?