Facetime/Messages Audio does not work in Mavericks

So I got this new feature in Messages. I can click the little phone icon and speak to the person I was text chatting with, or using Facetime video. Sometimes you just want to speak and not see, so this is good. Thing is, I go to call the other person on a computer in the same house, same network. We can Message chat and Facetime video, but when I call using Audio only, it rings on my computer, but nothing on the other computer. If the other computer calls me, then we connect, but no sound. Yet there is sound with Facetime video. Any idea what is going on?