My iPhone ringer rings out loud even when my headphones in

Whenever I plug my headphones in my phone calls still ring on the phone itself instead of the headphones. I am at work typically when this occurs so I would like a solution that stops the phone from ringing on the phone itself and instead on the headphones. I know you can change the silence to out loud with the switch on the side but I specifically put my headphones in to politely hear my notifications without annoying others. Is there a solution ?