Yep -also having problems with this. When using my MacBook Pro outside of my home and work networks I can use back to my mac to conect to my iMac at work, however when at home and trying to make the same connection it fails or responds very slowly.
I'm on a BT homehub 3.0/infinity. Haven't found a solution yet.
Replace the Home Hub with a timecapsule. It plugs directly into the openreach modem, and works like a charm. You need to configure it for "Connect using PPPoE"
Set username to be firstname.lastname@example.org,
Set the password to BT or anything (don't leave it blank, even if you use just a space)
Set "Connection" to Always On,
Set "Disconnect if idle" to never.
I found the homehub to work fine with PC's but even my connection was flaky with the macs at home. And Back to my mac works on a timecapsule!