Precisely. At least on iOS and on Mac OS, the Photos view (not the All Photos default album; confusing, I know) allows a manageable approach to an the full stream of photos. You can zoom out, by collections and years, so there's some chance of navigation.
The All Photos album is pretty much useless everywhere (with the exception, as you mentioned, on having the pictures ordered by add date, which comes in handy at times), but on the TV, since you can't do anything with the photos except look at them, it's particularly idiotic.
I really don't know why they bothered adding iCloud Photo Library at this state...