The solution turned out to be far simpler, but I still don't understand why.
Resetting the internet connection on the ATV fixed the issue. I just unplugged the ethernet cable and then plugged it back in. I did notice later that after letting the ATV sit for 20 minutes I had the same problem. So for some reason once the connection is idle for awhile it loses the connection to the library and cannot reconnect.
Interestingly enough, power cycling the ATV and various other resets didn't fix this, which leads me to believe it might not be the ATV. I am using powerline adapters to get internet in my basement. Maybe somewhere between the adapters and the router something goes wrong.