MacBook Pro Keep switching sound settings to External speaker

I don't have any external speaker connected to MacBook Pro. While I listen to any audio in either built-in sound systems or Airpod Pro, sometimes it randomly switches to external speaker. When I switch back to my settings, it instantly switches to external speakers again. it's quite annoying that I cannot find any settings to disable external speaker selection. I have been facing this issue for the last couple of months. Even after updating to the latest OS Sonoma it didn't fix the issue.