There should really be an option to turn off the icloud 'follow me' feature because a convenient feature for some, is an annoying bug for others. Or at least a pop up option to reject another device trying to connect, and keep the APMs connected.
There are a couple of options under 'Connect to this phone', or something, which I don't know if they help or not?
The 'follow me' feature works fine for me though. No one else shares my apple account and I never take phone calls through headphones. I like the way the APMs automatically connect to my Mac, Apple TVs etc as I roam around the house. It's far quicker and smoother than having to manually connect/disconnect every time.