iPhone 5 and earpods siri?

When I hold the button on the earpods for siri, I have the siri coming up, I start talking and suddenly it stops taking the question. It's like it didn't even recognize my sound. Sometimes I just hear the siri coming up, say my question to it, and when I start waiting for the answer I pick the phone up from my pocket and see that siri hasn't even been working on the question. Any help?

It's getting annoying.