The screen saver using your photos from iCloud has been an utter mess for years.
Sometimes Screen Saver Settings doesn't even see the photos or folders; sometimes it allows you to set what you want but it won't work under the preview and then doesn't work as a screen saver, and when this happens sometimes the screen saver never turns on, sometimes it goes black, sometimes it's black with a spinning icon, sometimes it reverts to Aerial; and on the rare occasion when it does work, I often end up seeing just the same a few photos repeated over and over and over and over again. I've seen all this on my Apple TV 4, but since changing to a 4K the other day, all I see is near exactly the behavior the original poster described - despite my setting it to My Photos, using Preview just shows a black screen with a never-ending circular status indicator, and when left to drop into Screen Saver on its own... there's Aerial, showing the same short clip of downtown LA for the hundredth time.
I've tried myriad options to make it work, and make it stable, and it's just this ongoing mess... now continuing for me over 3 generations of Apple TV devices from Apple TV 3 to Apple TV4K . I keep expecting Apple to sort it out (and maybe, just maybe, add new features or flexibility), but instead they amazingly just make it worse.
I've got around 70,000 photos in iCloud Photos; seeing them rotating on the Apple TV while music plays is my #1 use of the Apple TV, by far. So this fail in the most basic functionality of the product is extremely disappointing and frustrating.