Hey there, CoachHarry,
Thanks so much for using Apple Support Communities. We want to help you enjoy music on your device seamlessly. Without more information, however, that may be difficult. Based on your post, we're not sure if you're using an iOS device or a macOS device, which can make troubleshooting tricky.
If you are using Bluetooth headphones, then there may be a slight delay depending on the strength of that Bluetooth connection. This would be considered expected behavior for wireless headphones, simply due to the nature of wireless connections.
However, if the issue is occurring with wired headphones, then you may want to consider implementing a volume limit to prevent accidentally exceeding a maximum volume higher than you are comfortable with. This is how you would do that: Headphone notifications on your iPhone, iPod touch, or Apple Watch - Apple Support
Last but not least, the best course of action may simply be pressing the button a little more slowly as you wait for the volume adjustments to take effect.
We hope that this information is helpful.
Have a great day!