AppleTV (3) only uses same 25 photos for screensaver

If I have an album, event, etc with a bunch of photos in it, the screensaver should cycle through all of them, shouldn't it? I don't understand why I am watching the same 25ish photos (I haven't counted) from the "top" of my iPhoto album. Choosing another one and switching back results in the same photos being used. How can I make the screensaver do what it should?

(Note: I want it to do this when I idle)