I had the same problem where neither my headphones or my charger would stay in place. I read about lint in the port but when I looked in the port I didn't see anything and my first attempt to clean it out didn't yield much lint or dust.
Then, I used a push pin which was sharp enough to grab the packed lint and pull out a wad. Couldn't believe it. It was at the very back and just dragging the push pin across the very back is what finally broke it free. Works great now!