Podcast playback speed, watch, speaker interaction

When I start a podcast where the speed has previously been set to 1 1/4 (and I want it to stay there) and it is playing on a bluetooth speaker, there seems to be 30 seconds or more where the speed starts at 1 1/2 or faster and jumps around a bit before settling back in at where it is supposed to be. - like the phone and watch are negotiating who controls the speed even though they are both set the same. Does anyone else see this, and is there a way to avoid it?