Apple Watch won't swipe up or down

I've asked the same question and this is the only solution anyone can come up with. This happens to my watch multiple times daily and I am looking for a fix without having to restart the watch every time.

Don't want to setup as new. I'd like to know why its happening or a proper fix. Setting up as new or resetting it isn't really a fix. Appreciate the help though.