Same exact thing happened to me starting April 11th. Same watch with the same settings but if I look at the battery app you can see the steep downward line showing the draining at a very fast rate. I even went to the Apple Store and they did some diagnostics and said there was nothing wrong with the battery itself. She suggested turning off background app refresh and turning the watch off and back on. I had already done the latter, and (reluctantly) did the former (i.e., turned off background app refresh) since I had never had a battery problem even with background app refresh turned on. Anyway, none of this has helped.
I should mention that the battery drain started happening before the most recent Watch OS update. I've since installed the new update and it has not changed anything for the better (although it is not necessarily any worse).
Anyone know of the reason for the problem and the fix?