This just happened with my wife's phone (an iPhone 15 Pro running iOS 17.3.1). Searching Apple discussions here and elsewhere unearths lots of questions about this issue and few solutions. I tried everything suggested: restarting the phone, updating iOS, swapping adapters and headsets (she is using the USB-C to headphone adapter with a beloved wired headset) – and more – but nothing worked. She had to switch to another phone to be able to use her headset. The odd thing was that, as described above, music played in the headset just fine.
This told me it had to be an obscure iOS system setting, so I started methodically trying anything that might work. I eventually found this:
Settings -> Sounds & Haptics -> Headphone Safety -> Lightning Adapters
The setting screen here contains a toggle labeled "Connected To Headphones," but only when the headphone adapter is plugged in and there is a headset plugged into that. On my wife's phone with this brand new adapter, this toggle was off. Switching it on fixed the problem.