Hey homecat4,
Try enabling Reduce Loud Sounds at Settings > Sounds & Haptics > Headphone Safety. You should be able to specify the volume parameters used with headphones there.
We also wanted to note:
When you’re on the phone or listening to songs, movies, or other media on iPhone, you can use the buttons on the side of your device to adjust the audio volume. Otherwise, the buttons control the volume for the ringer, alerts, and other sound effects.
Adjust the volume on iPhone - Apple Support
In other words, you have to be actively listening to media or on a phone call when you adjust the volume or it will only adjust alerts and sound effects.
Finally, make sure you don't have any Accessibility features set to amplify sound. You'll find these at Settings > Accessibility > Audio/Visual > Headphone Accommodations. Adjust audio settings on iPhone explains each option in detail.
Kindest Regards.