iPhone getting wrong time from time server after brief idle period

After leaving my iPhone idle in sleep mode for an hour or so, I notice that the clock is almost at the same time as when I put it away. This is the first time I had ever had a problem with the clock and no matter what I do to solve the problem, it is only temporary, as the problem persists after it is fixed. I have tried syncing it with my computer's clock (which also syncs with Apple's time server), resetting the phone's settings, and just changing the clock manually, but the problem always happens. How do I fix this?