Hi....I found the solution to this on my MacBook Pro running OS X Yosemite and now everything is working perfectly and reliably.
In the days after installing Yosemite on my MacBook, and before I even went to set up Continuity features and iPhone dialing from the Mac, I received a few pop-ups asking me to allow incoming connections for several system services. I was not doing anything that would have caused these pop-ups to show up at that time so I denied incoming connections for all of them because I didn't know what they were for.
Fast forward a few weeks and I was trying to set up Screen Sharing so that I could help my father with his Mac remotely, and we could not get it to work for the life of me. Kept getting connection errors, which eventually led me to the thought that all of those services I denied access to in the Mac firewall a few weeks earlier might be needed for Screen Sharing in the Messages app.
So I went into the Mac firewall and Allowed incoming connection on the following services:
Screen Sharing (enabled in Sharing in System Preferences)
awacsd
coreaudiod
FaceTime.app
identityservicesd.app
kdc
launchd
Messages.app
NETserver.app
sharingd
Not only did this get Screen Sharing working, but the iPhone calls from the Mac started working too. And it has stayed working since. Some of these are clearly related to the Screen Sharing feature so I can't identify which ones specifically got the iPhone/Mac calling working....but one or more of them certainly did because it only worked after I did this.
I did not change any setting on my modem or router.
I hope this helps someone out there because iPhone/Mac calling is pretty convenient....