iPhone 5 Passcode Problem

I just bought the iPhone 5 and enabled the passcode lock. My problem is that the passcode is sometimes rejected and other times it is accepted. I am sure that I am entering the correct passcode. I even filmed myself doing so to double-check and have allowed family members to try and enter my passcode to make sure that it is correct. I would say 95% of the time my iPhone accepts my passcode and unlocks my phone and then the other 5% of the time the iPhone does not recognize the passcode and locks me out (then I must wait 1, 5, or 15 minutes until it accepts it). Does anyone have any ideas as to what is wrong? Is this a software bug?