Unable to add to up Next via Siri remote

Whenever I press and old the button on the Siri remote to tell Siri to add something to UP Next ti get the following reponse, "I can't add to your up next queue while you're listening to this." The problem is I'm not listening to anything, nothing is playing on the AppleTV. I've restarted the AppleTV and closed all the Apps but is persists. My AppleTV is 12.3 16M153 model A1842 64GB.