Q: Cant go to previous song

This has been happening for years now, goes back to my iPhone 5 (I have 6 now). It tries to go to a previous song, but immediately skips it forward back to the next song. I have to rapidly hit the button and it might actually go back, but typically results in restarting the current song as many times I hit the button.

This happens on the Music app, wether controlling via the slide up, the music app, or any other means (earbud controls, car infotainment, etc.).

No, restarting the phone doesn't help.

iOS 9.3.1 (I think).