This may be one of the drawbacks of HDMI-CEC (EasyLink in Philips-speak), when the HDMI connection switches back to Apple TV, as it may be perceived as the active device at that moment. Unfortunately, HDMI-CEC only has on/off settings without further fine-tuning to control its behavior.
To prevent this, turn off HDMI-CEC on the Apple TV. Having it off will also prevent the Apple TV from waking or sleeping other devices, and switching to the correct HDMI port automatically. To turn off HDMI-CEC on the Apple TV, go to Settings﹥Remotes and Devices﹥Control TVs and Receivers﹥On|Off.
Control your TV or receiver with your Apple Remote - Apple Support
Turn on your TV or receiver and switch to the correct HDMI
[HDMI-CEC on] To turn on your TV or receiver and automatically switch to the correct HDMI input, just wake your Apple TV: Press any button on your Siri Remote or Apple TV Remote to wake your Apple TV and turn on your home-theater equipment.