Apple watch unresponsive touch with Wrist Detection

I have recently purchased an apple watch series 5 44 mm. The real issue that I am facing is that when wear the watch on my wrist with wrist detection enabled, the touch screen is not as responsive or fluid as it is on removing it. I have to make multiple taps on the watch in order to get this to work. Additionally, I noticed that if I remove the watch, it works just fine and also disabling the wrist detection feature also helps. However, this is not a solution and I really wants the Apple Genius folks to take this concern into account. This is a very serious problem for me.