Echo in headphones recording podcast through Skype/GarageBand

I've been following instructions for using garageband to record podcasts via Skype. I connect with the other person on Skype and then use Soundflower and Linein to shift the audio to Garageband and garageband records it fine.

The problem I am having is that I hear my own voice in my headphones as a very annoying echo. Is there a way to get rid of this? I am guessing that there is a way for me to set this up so that only the other person's voice comes back at me through my headphones, but I can't figure out how?

Thanks for the help.