How can I stop the HomePod from taking over my playlist from my iPhone?

When I broadcast music from my iPhone to my HomePod, the HomePod often takes over the playlist, meaning I can't control it from my phone anymore. I have to use the Home app, which is annoying and less useful. I've turned Handoff off, so that's not it. How can I stop this?