I agree with you - this is frustrating!
I've spent hours trying different things an have the same issue.
iTunes I garbage and I can't even stream my own movies locally as they don't buffer enough. Videos keep pausing. Wifi or gigabit Ethernet. I gave up on trailers long ago but my own content???
Just a few months ago a few GB video would completely load in minutes while watching. Since then I have not updated anything on atv2, the only update I made was iTunes on my computer. The same video will only load for a bit (maybe 2-3%) then stops buffering. Then while watching the video will choke and then buffer just a bit more ... Then choke, buffer again and again.
This is unacceptable. Network transfers are fast (I've been checking it obsessively) and since I haven't made a chane to atv2 I suspect iTunes to be the problem. That or my atv2 is dying.