So it's not just me, then. Yes, very, very irritating!
Worth noting too: it's controlled from iOS. If you get the AirPods configured the way you want, then switch to Mac, you will find that this does not happen; your configuration remains as programmed. Something, somewhere, deep in the bowels of iOS, is spontaneously switching this off, for some reason.
I do notice that this tends to happen more often when I move for some reason, especially when I go to the bathroom and the ambience changes noticeably. Of course, from there, it's irritatingly hard to change it back, because you need to do this from the phone directly and not the pods, so if you're fully occupied and it switches off, you're stuck, temporarily.
I also couldn't help noticing that sometimes the controls showed custom transparency as being on and the amplification set correctly, even though it clearly wasn't actually amplifying, but moving the slider even very slightly would instantly restore the correct behaviour. In other words the state of the pods wasn't actually reflected properly in the UI. Sigh.
Absent contacting Apple Feedback (which you should do), or the Apple Accessibility team directly, anyone have other solutions? Could a Siri shortcut do it?