Same problem here. Using AirPlay streaming from an iPhone 4S, iPhone 5, and iPad 2 all result in very slow buffering on even short (30sec) videos. Takes a long while to buffer enough to start video play, and in the middle it pauses because the buffer ran out.
Same AppleTV streams OK from internet sources (eg. Netflix) even on my slow 5Mbps DSL connection. Using AirPort Extreme. Again, streams fine from internet sources.
Expect the device-device AirPlay performance to be much better.