iPhone 5 random slow to wake problem?

Sometimes on my iPhone 5 it randomly is slow to wake up, can take anywhere from 5-20 seconds to wake. Like when the phone is sleep, and I press either the home or power button on top, it won't wake or anything, and also when it's not waking and still at a black screen, ill try to plug it into the charger to see if it wakes and still it stays black screen. It's like the phone is like dead for 5-20 secs and wakes back up. I have a feeling it's not a hardware problem because my older iPhone 5 had this problem so I replaced it at an Apple Store and the new one still gets it.

If I wait a bit then it'll wake up and turn on. I have already replaced the phone with a new one at the Apple Store for this reason, and the newer one has the same exact problem, so I'm guessing it's something I did.....

My phone is a factory unlocked one and my SIM card currently is a T-Mobile and I cut it myself and I did not cut it so great and it's smaller than the SIM tray but it fits and works, maybe that's the problem?

Has anyone ever had this wierd wake issue with the new iPhone 5?

I am on iOS 6.0.1 by the way.