So I've been having this issue for a few months. I remember last year I was able to Airplay music from my iPhone to Apple TV and be able to control the volume.
Today I Airplayed music from Apple Music on iPhone to Apple TV and the handoff problem didn't occur. It worked as it used to and was able to turn volume up or down with iPhone physical buttons.
Then I paused the music, and when resumed, the handoff to Apple TV happened again and I wasn't able to control volume.
Now I've tried to stream music from my iPad Mini running iOS 12.4.7 and the Airplay function happens as expected, I am able to control volume and everything and now my iPhone's Music app has an Apple TV logo on the album cover.
I also tried disabling WiFi on iPhone and tried to Airplay and it worked fine, handoff didn't happen and I was able to control the volume from my phone. But then, when I tried to turn WiFi back on, it wouldn't work and had to go to settings, turn it off again and back on.
It might be an iOS 13 issue as I remember controlling the volume last year for a while.




