I know this won't help you, but I'm facing the same problem. I suspect it's a problem with the way iCloud handles very large files.
I wanted to see how long I could record video with a full battery on 2 devices; this created a 10gb file on one device and a 20gb file on another.
I immediately deleted them (literally as soon as I finished recording), and removed them from the "recently deleted" album - but a few days later, I noticed that the 2 files had been synced on ALL my devices. But when I deleted them (again), and removed them from "recently deleted" (again), the icloud library photo size still counted the videos.
I can prove it's a bug/issue by doing one simple test. On my MacBook, I open photos.app and go into 'moments'. I select every item and right-click and select "get info". This calculates the size of every item in my library. It comes to 14gb (this is the figure I get when I go into Finder and click 'get info' on the "Masters" folder in the photo library bundle). Then when I deselect all, the "get info" box remains open and reports.... 48gb used. This proves that there's a problem somewhere - the files *do not exist* on my computer, on my iPads or iPhone, and they are not shown on icloud.com. Of course, the difference is 38gb, when my two video files are only 30gb. So there's another file somewhere causing my space to fill up.
I've searched for several weeks to see if I can find a solution. I'm about to do the one thing I hate doing: I'm gonna call support and see if I can get them to sort it.
If this doesn't do it, then I'm going to turn off iCloud photo library and wait 30 days - after that, every stored photo will be deleted from Apple, and I can start again.
There's a problem with how iCloud decides to sync large files. I suspect that it "decides" to sync as soon as a file is created, and perhaps syncs bits of data while the video is being recorded, which is why it synced the video files even though they were deleted immediately. And then, somehow it either keeps those files in the cloud, hidden, or it has a database somewhere that has entries for those files even though they don't exist, making it believe the space is used.
If I get any joy from my call to Apple support I'll post again here.