Magic Trackpad and Wireless keyboard can't click or key to login

I have this issue intermittently at the login screen. Where my Magic Trackpad pointer works but you can't click on anything. Using the wireless keyboard doesn't help when the Mac Mini is in this state.

When the Mac Mini is not allowing trackpad click or wireless keyboard input. I can use VNC to login on the Mac Mini and then logout. After that the system behaves normally.