Macbook Pro Bluetooth audio issue

I've been having the most annoying bug and couldn't find a single solution. It was happening on 13.3.1 and persists on 13.4.

When I connect any type of audio device via bluetooth, anything sound related has to wait for few seconds to actually play. For example when I click on a video, it waits 1-2 seconds and then starts playing. After it starts playing, it plays without a delay and in perfect sync. So its not a common bluetooth delay issue. Note that this issue only happens over bluetooth, wired connections or internal speakers work perfectly fine.

I noticed that it happens after few seconds of audio inactivity. If I start playing spotify or anything in the background and mute it just to keep audio active, everything works as it supposed to be, videos start instantly, but again if there is inactivity everything waits for few seconds, even screenshot shutter sound comes late. Does anybody has a fix to this?

Tried:

• Safe Mode
• Creating a new user
• sudo pkill bluetoothd
• Restarting the device several times
• Forget all the bluetooth devices and then reconnect
• Deleting bluetooth plist.
• Updating to latest software.