It sounds as if you may have Optimized Battery Charging enabled. On the Watch, go to Settings>Battery>Battery Health>Optimized Battery Charging. Is it on? If so, the watch will try to "learn" your usage patterns and only charge your battery to 80% if it thinks you're not going to be taking it off the charger for a while. It will charge it the last 20% so that it will be ready when it believes you will be taking it off charge.
While I find optimized battery charging works very well on my iPhone, I'm less pleased with it on my Watch and have it turned off. I charge my watch for about 90 minutes twice a day and the algorithm just doesn't handle that.
While you're in those settings, take a look at your battery health. If you've had the Watch 4 since it was first released, your battery may be reaching the end of its useful life. It it has dropped below 80% capacity, it's time for a replacement/