How to make silicone cases stop picking up lint, dust, and everything else in your pocket?

I have th griffin survivor slim case, and I love it except for the fact that if I put it into my pocket it picks up all the lint and everything else and takes forever to rub it all off. Has anyone figured out a way to get this to stop happenng? It's rather annoying not being able to put my phone in my pocket.