Thanks for using Apple Support Communities.
TCP port 123 is used to communicate with a network time server.
If your router is blocking this port it will never set the time.
For more information on this, take a look at this article:
Apple TV: TCP and UDP ports and protocols used
Also it needs an active internet connection.
Apple TV (2nd and 3rd generation): Troubleshooting Wi-Fi networks and connections
Best of luck,