HomePod Siri fails to shuffle Apple Music with 'problem with Apple Music' error

When I ask my Living Room HomePod to "hey siri, shuffle my music." it replies "here is some music just for you" then there is a silence before it times out and replies 'Sorry there was a problem with apple music'.

Once I stop Apple TV using the HomePods to use the TV speakers instead the problem is temporarily resolved.

I don't want to have to keep changing the settings.

Any suggestions?