iPhone 6 volume changes when I use headphones

I've set the volume of my phone to less than half for the ringtone, but when I use headphones (bluetooth or wired) I increase the volume to listen to my music, and when I disconnect the headphones, the volume stays at the higher level.

I thought the iPhone would detect this change and reset the volume accordingly, in the same way I can set the volume control for each speaker/headphone device on my iMac.

Is this me being a bit optimstic, or have I missed something?

Thank you.