I doubt sincerely that the extra HomeKit functionality of the HomePod has anything at all to do with your issue. The amount of traffic that all generates is quite small, and zero if the features are not actively engaged.
I think the key here is that you are using 3rd party audio sources (e.g., Spotify) and AirPlay. As an experiment, try using ONLY the Apple Music app for a while and see what results you get. My hunch: the Apple Music app works perfectly.
Why do I have this hunch? Because I have observed similar erratic behavior with various 3rd party audio apps and AirPlay, with or without HomePod. I cannot be certain, as I am not an Apple engineer; but I'm an audio guy - and I think that some apps don't play nice with AirPlay's buffering and latency demands.
If that hunch plays out, then the issue is with the implementation of AirPlay on the iPhone/iPad and how it interfaces with 3rd party apps, not your network. Small consolation as it offers no cure, but at least you might know what the problem is.