iPhone Allows Caller to Hear Conversations I’m Having and Caller is Speaking Before I Have Answered the iPhone!

iPhone rings and before I get to the phone to answer it, people are actually talking and and when I finally answer the phone they tell me that they can hear me and conversations on my end of the phone. This is a real problem and invasion of my privacy !! How do I stop this from happening??