First keystroke after idle has a slight delay

So when I don't touch the keyboard or trackpad for about 5 seconds the first keystroke I make is delayed for about .5 secs which is annoying when recording in Ableton with the MacBook keyboard or when gaming. After the initial key press everything works fine until I don't touch the machine for 5 seconds again.

I'm on the 2012 i7 quad core MacBook so performance isn't the issue. I'm guessing this is some power management thing?