Is it not a bad thing that water gets into your. speaker.

Apple Watch Series 2 and later are rated for water resistance up to 50 meters, or about 160 feet, deep. You can remove water from your Apple Watch speaker using Water Lock. Swipe up from the bottom of the display on the watch face and tap the water droplet icon, then turn off Water Lock by pressing and holding the Digital Crown.

Swim with your Apple Watch - Apple Support

About Apple Watch water resistance - Apple Support