Yes, that is a drawback, that iCloud Photos is "All or Nothing". I am keeping my iCloud Photos Library small and are archiving all photos on my Mac on external drives. Only the photos I want to see on all my devices are in my iCloud Photos Library.
But otherwise I am really enjoying iCloud Photos - it is offering many new features compared to My Photo Stream. Previously I had always to remember to transfer the videos manually, as My Photo Stream did not transfer them. The Live Photos arrived without the animation. my Photo Stream had continually gaps, when I have been traveling abroad without a decent internet and could not sync the Photo Stream within the thirty days the new photos have been cached. My Photo Stream created a mess of downsized duplicates, because the photos have been downloaded on the mobile devices at a smaller resolution.
All this is now easier with iCloud Photos. The best new feature is, that the adjustments I apply on my iPad or iPhone will sync back to my Macs - my Photos Libraries on all devices are always identical; can work with my photos on any Apple Device I am having just with me.