Swipe up/down stops working

I have the indicator that there are alerts but swipe down does not work.

Swipe up works more often but sometimes it stops working as well.

I can reset the watch and it works again for a while but after a day or 2, it stops responding to swipe down again.

I am running Watch OS 7.6.1 but this has been going on for a while.