Find out how fast your internet it at speedtest.net.
It is probably too slow or variable.
AppleTv ideally needs:
2.5 Mbps for SD, 6 Mbps for 720p HD and 8 Mbps for 1080p HD
If you have those speeds for the given content quality playback should start virtually instantly.
Any slower however and AppleTV buffers (partially downloads) the stream.
Say a movie is 60 mins long, 4GB in size and your internet can provide 2GB/hour then it would take an hour to buffer half the movie. AppleTV calculates how long it will take to buffer (to enable playback) without stopping to re-buffer. Once it starts it shoule continue to the end, unless the internet speed drops and can't keep up (or someone else is using the internet too effectively slowing the speed for AppleTV).
If you have slow/intermittent speeds, one option is to rent via iTunes on the computer - once FULLY downloaded - you will be able to play on AppleTV using the Computers icon which gets a new Rentals row added if there are rentals in iTunes. Computer/iTunes of course need to be running to send the data to AppleTV to play.