To all of those who say this problem sprung up after the 4.2 update - I have had this problem since the beginning, so I doubt that is the cause.
It happened again this weekend to me. It started crapping out right around 7pm last night. And once it started (or stopped, as it were), it continued to do so for a while, even to the point where it would not continue playing. When this happened, I decided to test my memory theory by cutting power to the Apple TV and then restarting it and resuming the video. It turns out, the problem persisted through the reboot. This tells me that it's not specifically a memory issue.
The interesting thing is, the reboot didn't fix it, but a little later Springboard crashed and restarted and then everything worked fine. Bizarre, eh? (NOTE: Recall that Springboard is the app that runs by default on the Apple TV. It is the user interface we all use to navigate our content.)
So, this is a bit confusing, right? I am now thinking that it's possible that the Apple TV memory had previously been serialized to disk (which is what happens on the iPhone/iPad to support multitasking) and rebooting it just reloaded that memory from disk. And that the fix didn't occur until Springboard crashed, which caused it to start up without any serialized memory store.
MY DIAGNOSIS: This is a bug with the way that the Springboard app manages its memory when dealing with Netflix streams. It rears its ugly head when traffic is high and has to rebuffer frequently.
Please write to Apple and tell them to fix it!
http://www.apple.com/feedback/appletv.html