Step tracking with apple watch series 4

I crochet. Yesterday when crocheting in a sitting position in my chair (I am right handed and watch is on left arm) I got a notification stating they noticed I started a swim? Did i want to start my activity tracker? LOL

I did a review of my steps over the past week while I have been crocheting and it appears that it is counting my arm movement with the crochet hook as steps. That movement is not any more drastic that typing on a keyboard? Thoughts? Do I have to take my watch off while I am crocheting?