HomePod Handover only works on Default account

Two HomePods, updated to iOS 13.3

Two iPhones (X and Xs) updated to 13.3

My phone works fine with the HomePod., I can do handover no problem. I used my phone to set up both HomePods. I added my wife to the home app

My wife’s phone will not do a automatic music transfer at all. She can only play through connecting via airplay.

I’ve tried all the language, location sharing checks. I’ve restarted and reset the HomePods, I’ve restarted the iPhone’s.

It seems the second user (wife) is just not connecting properly.

Any suggestions?