To save you a bit of time...or perhaps a lot of time.....regarding Back To My Mac (BTMM), Apple assumes that you will be using a simple modem on your network......something like this, for example:
Most Internet Service Providers (ISP) supply a modem/router or gateway device to their customers. Something like the example pictured below that combines the functions of a separate modem and a separate router in one package.
When you connect an Apple router to a modem/router or gateway, BTMM will not work correctly, since the modem/router is acting as the main router for the network and the Apple device cannot do this, which is needs to do for BTMM to work properly.
So first check and confirm that you have the correct type of equipment to work with BTMM.
Sometimes, it is possible to reconfigure a modem/router or gateway to have it operate only as a simple modem. If you have this type of "modem", then check with your ISP about what options you might have.