iPhone 5 headphone jack not detecting headphones

It's been several months now that my iPhone hasn't been able to play music through headphones. It just wouldn't detect my headphones or any for that matter. I thought it was a hardware issue so I decided to replace the charging port which the headphone jack is attached to. Same problem. I basically had given up on that when I read somewhere that it might be a software bug and a simple wipe and restore would fix it. Tried that, didn't work again. I also tried to see if it would solve the problem.. didn't either. After a while I tried putting a metallic pin inside the jack and tried to remove any dirt that way, I realized the music I was playing pause and then came back. Tried a few more times and figured out that the pin was still able to trigger the headphones feature. I would see 'Headphones' and the volume would increase instead of the usual 'Ringer'.. but still no luck with actual headphones.

So now I know my headphone jack is working but not entirely.. Does anyone know a fix for this?

This is without the pin.

With pin and touching a specific part inside headphone jack.