I've had this issue with my AirPods on and off ever since I bought them a couple years ago, through 3 phones and at least a couple iOS upgrades. The ONLY thing I've found that helps is to go to "Settings -> Sound&Haptics -> Headphone Safety -> Reduce Loud Sounds -> Lower to 75 decibels."
This keeps the AirPods from destroying your eardrums when someone calls/texts (even with the ringer set to low) and lowers the volume on calls to a more reasonable, albeit still loud level (again, even when the volume is set to the lowest level possible).
Downside here is that when I use my bluetooth speaker to play music in the house, I have to manually disable this feature, as it affects that bluetooth device as well.
Really wish Apple would do something about this! :(