Wired audio heads with mic not working on iPhone 12 mini?

I'm using one of those lightning cable to 3.5mm jack adapter + a normal wired hands free. Now, listening to music, vids etc works as expected. The problem is when someone calls and I pick up, its not working.

I dont hear them. FaceTime video audio was actually external instead of using the connected handsfree.

Are there any settings or something to fix this issue? I don't want wireless headset so im trying to figure out how to make the wireless work.