Activity sharing when off

I recently had someone NOT in my contacts congratulate me on my workout. My workout showed on his watch as a contact of his, when I replied bc I didn’t know who it was texting me is when my name and number came up. I do not want to share my workouts with anyone and am considering getting rid of the watch due to privacy concerns.