IR remote return function stops working, reboot required

I have universal remote that controls all of my other devices, sets the correct inputs, etc. This system worked great with the previous generation Apple TVs.

I purchased a new Apple TV 4 and everything was working great. However, after a random period of time the return/enter button stops working. If I reboot the Apple TV, everything works for a period of time and then stops again after a while.

Is anyone else experiencing this? Are there any known fixes?