FYI, I bought the apple series 6 40mm 2 weeks ago, first time using an apple watch. I am using iPhone 11 pro max and have the same problem since day 1 with both swipe up and down. After a week of daily power cycle on the watch, I've contacted the apple care on Monday and suggested to update my phone iOS from 14.3 to 14.4. Works for 3 days in a row after the update and today on day 4 started again. Contacted them again and suggested me to do factory reset on the watch and book a appointment with their service centre?!
Found the suggested fix of turning off the Zoom accessibility from the forum and seems to work last time before iOS update.
Looks like it's very well known issue from Apple end but refuses to acknowledge that there is actually a software bug on their product! It's a band-aid to turn off a function to make the watch work. Not cool!