iPhone SE stuck in headphone-mode when connected to Bluetooth speaker

I've tried to Google this issue, but I can't seem to find a solution to this exact problem. I've seen alot of posts regarding iPhones stuck in headphone-mode in general.

So the iPhone SE is stuck in headphone-mode _only_ when the phone is connected to a Bluetooth speaker. I've tried soft reset, plugging in and out the headphones, tried reconnecting the device etc..

Does anyone know if there is a solution to this problem without having to do a factory reset?