Apple CarPlay changing from regular audio to voice control

My music randomly will change to play out of the other speaker for my cars voice control speaker which would be used for Siri or Navigation Voice Prompts. It always happens half a mile after I leave home, and randomly while driving. I have tried in a newer Honda Accord and this issue did not happen. I have previously had problems with the sound randomly dropping and going back up, and dropping again because it was going in and out of the voice control speaker. I have upgraded my phone and tried everything. Has anyone else experienced this problem?