Call switches to earbuds (not ipods)

This is a problem I've had for years but I've eventually got sick of it so would like some help as Apple first line support havent helped. I am of the older generation that like to hold their phone up to their ear :). so the problem I have is that if I'm listening to music on my sony earbuds and a call comes in, I press the speaker button to transfer away from the earbuds to the phone. The problem is it keeps switching back to the buds and I'm talking having no idea that it's back to the buds. The only way around when a call comes in, is to also disable bluetooth on the phone (as well as press the speaker button on the iphone). But that then means by earbuds wont then reconnect to the iphone unless I have the case with me!

So - any way of stopping the call repeatedly connecting back to my earbuds!? Thanks :)