Greetings Cyberacc,
Thanks for visiting the Apple Support Communities! I understand you'd like to clear the items in the Now Playing section of Apple Music on your Apple TV.
The Now Playing section will automatically display songs that are adjacent to the song you've selected to play, and there is not a setting to clear this section: Now Playing: Control music playback on Apple TV - Apple Support
"Songs adjacent to the currently playing one appear in the Now Playing queue. A timeline also appears showing elapsed and remaining time. When the timeline is active, you can also press the Touch surface to play or pause the song."
The Now Playing section will clear after you quit the Music app. You can see how to force an app to quit on your Apple TV from the following article: Switch quickly between Apple TV apps - Apple Support
Kind regards.