There is only a limited amount of space on the flash drive for this purpose. Most of it is used to cache the streamed content. The actual amount can vary depending on photo resolution etc. but it tends to be around that mark.
I agree - I also see just the 100 or so photos on Apple TV screensaver and it is disappointing because I have over 4,000 marked with 4 stars that I would like to see on it. I really hope they fix this on the next Apple TV.