Me too and it drives me absolutely crazy. I can't work without music, and due to this I am forced to wear my cans all day (which thankfully have their own dongle).
From what I am reading I think the issue is clearly when people are using mouse, keyboard, and speaker/headset via bluetooth.
It seems like its the combo two or more devices (when one of those devices is BT Audio) causes it.
Using the command: "sudo pkill bluetoothd" does provide some relief but not entirely, it always comes back.
I switched back to using both dongles for mouse and keyboard but I have so many usb devices that causes other issues. I read someone to turn off the airplay receiver in the settings, I did that and it didnt help. Since others have had hardware replaced and it didn't fix it, its clearly a software issue Apple needs to fix at a driver level or in the bluetoothd process.
I am using a MBP 14 2021 M1 Max 32GB Ram on 13.3, totally unacceptable.