I've had this same problem. Talking through the various levels of Apple support, it sounds like this might be an issue that arises if your network is blocking access to port 123 which this document ( http://support.apple.com/kb/TS1629 ) shows is used by Apple TV for Network Time Server Synchronization. As to why it didn't appear to be an issue until 4.4.4 (3330) I couldn't get an answer.
Chucks correct and for good measure I have configured mine with additional known ports as well, especially port 3689 to UDP (not TCP as Apple suggests). Attached is a link with the configuration changes I've made and since then I have not had any issues.