Can't swipe up or down to check notifications on Apple Watch

Why does my watch freeze sometimes when I get notifications? It seems to only happen now and again but sometimes it means that I can't swipe down to check my notifications or swipe up to check control centre. I have to keep switching it off and on again for it to work. My watch is only weeks old so not sure if it's an issue with my watch or it's an issue with all watches.