When I take out Airpods music begins playing on Apple TV

I usually use AirPods when watching the Apple TV and sometimes when I take them out of my ears, music begins to play. What causes that? And is there anything I can do to keep it from happening? It can happen when I'm on the home screen or even when I'm in an app.