How to properly clean iPhone 5 Earpods?

I'm afraid to clean them with any sort of solution on a q-tip, I'm not sure if it would damage the insides of the speakers. I've used a dry q-tip but all that seems to do is push the dirt and junk into the outer edges!