Issue with ending or resuming an Apple Watch fitness activity / workout

I cannot end a workout or resume a workout after pausing the workout without pressing the screen / button multiple times. When I press down on the watches screen the buttons appear to be pressed but the activity either doesn't stop or won't start again it takes multiple presses to get it to work. This is very frustrating.