Disabling Siri during workouts?

I currently have my Apple Watch set up to only fire up Siri when I press the watch crown. This works perfectly, other than when I'm working out.

Because wearing the watch low on my wrist is the most comfortable for me (and because my sleeve tattoos above the wrist mess with the sensors), the crown often gets inadvertently pressed when I'm doing pushups, high planks, racking/unracking a bar from the squat rack --pretty much anything that has my hands at a 90 degree angle to my forearms - and will activate siri (who invariably will be confused as to the music being played on the gym loudspeakers).

Since I'm using Apple's workout app on the phone, is there a way to simply disable Siri while that app is active? The other workarounds I've considered (flipping the phone around, completely disabling siri) seem to be a pain in the butt to have to do them right before the workout, as well as then setting them back afterwards.

Thoughts / Comments?