I ended up figuring this out, so I am posting the solution in case anyone else gets stuck in this situation.
I was originally trying to do this in Hulu, by swiping down on the remote to bring up the menu with options "Info / Subtitles / Audio". Under the "Audio" tab, there are three sub-options: "Language / Sound / Speaker". Under "Speaker" the Apple TV was selected, and I would select one or more Airplay speakers available. Audio would play to the Airplay speaker(s), but volume could not be adjusted. The only option available on this menu is to enable/disable speakers, but no volume control. Same behavior in other apps (Netflix, Prime Video, etc).
The volume control for Airplay speakers on Apple TV 4 can be found by holding down the "Home" button (next to Menu) on the remote. In the menu that comes up, select the Airplay icon (three circles and a triangle). Here you can enable/disable Airplay speakers, and a volume bar shows up for each speaker that can be adjusted. Works just as I was hoping!