I had this exact problem and had to call Apple Support to fix it. According to the tech, this is some kind of "feature" to make sure that a bot isn't trying to hack my Apple ID. (OK, sounds reasonable, but wouldn't it be nice if FaceTime actually generated an error message under this condition rather than simply bringing you back to the login screen?) When you call Apple (see below for details), they "unlock" your Apple ID and it magically starts working on the Mac.
If your problem is like mine, here's the details:
1. Open iMessage and go into the preferences section. Try to sign in with your Apple ID. If your problem is like mine, you'll get an error message an a link to click on to go to an Apple web site.
2. The web page, titled, "iMessage Registration", will read, "Apple needs more information to complete your iMessage registration. Please call AppleCare in your country (listed below) and provide them with the following validation code". A 12-digit number is listed.
3. Call Apple Support and hope that they won't insist on charging you if your hardware is out of warranty. (I didn't have a problem, but emphasized and re-read exactly what the web page was telling me to do.) Seems pretty reasonable that this shouldn't be a warranty issue.
4. You may be shuffled around a few times (I was), but you should eventually get to someone who knows what's going on. Give the validation code from the web page to the Apple tech. He/she will enter it into their system and then your ID gets unlocked.
5. If all goes well, your Apple ID will now work in iMessage as well as FaceTime.
What's frustrating about this is that I never even use iMessage. I just stumbled upon it under a hunch that the problem was with Apple's servers and that iMessage it might generate a useful error message that FaceTime did not.
Hint to Apple's programmers: please insert the same code into FaceTime that iMessage uses to direct people to call Apple Support and validate their Apple ID account.