How do I actually see what books are stored in my iCloud?

Go to SystemPreferences, AppleID, iCloud.

Is iCloud Drive checked?

If so, Click on Options, next to iCloud Drive.

Is Books checked? If yes, all your books are stored in iCloud

Login at iCloud.com. Click Photos and later Click iCloud Drive.

If you are syncing iBooks, you can't see what is stored in iCloud. That data isn't accessible at iClod.com.