I have the same issue so moved a 2.5 gb file to Google drive. So I only use 5 gb icloud - documents and desktop. After removing said files Documents has 2.42 Gb and Desktop 164 Mb. I have no photos. The only other things are quite small eg calendar, contacts. When I go to the icloud info via Finder it says I have 4.94 Gb used for Documents !! This is the same figure I had before I removed the big folder (actually it was closer to 5.I don't use Photos.
It's really annoying you can't just delete a file from the icloud Drive without it syncing across as others have described.
One thing I have also noticed is that if I go to icloud drive on my computer it shows the Desktop and Documents folders and contents but not the sizes.