How to keep Apple Watch screen brightness always low?

The default screen brightness of Apple Watch for me is too bright.

So I have to lower it via Apple Watch settings to value as low as possible.

I expect that it will keep to be low brightness until I change that setting.

But few times per month, brightness become high again, and when I visit settings it shows that brightness setting value set to low even the screen bright in fact is high.

To make brightness low again I have to change it to higher value, then lower again.

It is really annoying that this setting not being saved.

I had this issue in Apple Watch Series 2, and it remains in Apple Watch Series 4

Any way to fix that?