Homepod can’t control Volume with “Hey Siri”

I can only control the volume of my 2 HomePods from the device transmitting the sound/music. In this case, I’m playing music from my phone iTunes library, but the only way I can change the volume of the homepods is by either going directly to it and tapping on its top or getting my phone. Problem is 25% of the time I can simply go into my iPad for example and in the control center adjust the volume of the music when my phone is in another room, but the other 75% of the time the volume slider isn’t there and it just allows me to pause/play and skip the songs, and I can’t figure out why that happens or how to fix it. On top of that, I try to tell “Hey Siri adjust the volume in all the homepods X amount/percentage” and it just tells me there was a problem doing it and doesn’t adjust it. Most of the time it only works when telling it to adjust it’s own volume, but I want to adjust all the homepods volume since it clearly has the ability to do so. How can I fix this?? Is there some setting I need to change or something? Most of the time I play music at home is cuz Im doing stuff around the house and my hands are full, so being able to grab the nearest device to control all the volume or simply telling Siri would be really Helpful…