I have had the same problem for well over a year with both Mountain Lion and Mavericks. When I try to select Apple TV for audio output, the selection jumps back to Internal Speakers. I use "killall coreaudiod" to get it going again. I have to do this almost every day.
I noticed some differences in the System Log, which might give someone a clue. When it works properly, selecting the Apple TV results in 7 lines in the System Log:
25/05/14 11:51:52,512 SystemUIServer[257]: 2014-05-25 11:51:52.511516 AM [AirPlayAVSys] Picking route 'Apple TV' (XXXX)
25/05/14 11:51:52,513 SystemUIServer[257]: 2014-05-25 11:51:52.512780 AM [AirPlayAVSys] Selecting endpoint XXXX, item 302 for (audio only)
25/05/14 11:51:52,513 coreaudiod[263]: 2014-05-25 11:51:52.513399 AM [AirPlay] Selecting endpoint XXXX "Apple TV" for A/V
25/05/14 11:51:52,528 AirPlayUIAgent[5171]: 2014-05-25 11:51:52.528189 AM [AirPlayAVSys] AirPlay became the default audio device
25/05/14 11:51:52,529 SystemUIServer[257]: 2014-05-25 11:51:52.528712 AM [AirPlayAVSys] AirPlay became the default audio device
25/05/14 11:51:52,534 iTunes[2941]: 2014-05-25 11:51:52.533805 AM [AirPlayAVSys] AirPlay became the default audio device
25/05/14 11:51:52,818 coreaudiod[263]: 2014-05-25 11:51:52.818224 AM [AirPlay] Set up for stream GeneralAudio, PCM/44100/16/2 -> ALAC/44100/16/2
But when the selection jumps back, it only logs 3 lines:
25/05/14 11:37:48,593 SystemUIServer[257]: 2014-05-25 11:37:48.593088 AM [AirPlayAVSys] Picking route 'Apple TV' (XXXX)
25/05/14 11:37:48,594 SystemUIServer[257]: 2014-05-25 11:37:48.594412 AM [AirPlayAVSys] Selecting endpoint XXXX, item 300 for (audio only)
25/05/14 11:37:48,595 coreaudiod[263]: 2014-05-25 11:37:48.595407 AM [AirPlay] Selecting endpoint XXXX "Apple TV" for A/V
(In the above "XXXX" is the MAC address of my Apple TV.)