I am hopeful this will get fixed. Either not enough people have reported this or it is a nasty one it seems. I have seen reports of similar behaviour on models even prior to the M1 release, so I am also hoping it is not an issue edged into the silicon, but a software one. Out of interest, which model are you running on?
With the support, they made me partition my disk and reinstall macOS there, essentially making a minimal set-up where nothing else could interfere. There is also a special way to do the SMC resets on the M1 lineup. Although I am noticing it getting temporarily better after a restart, this was and is one of the major reasons why I got a mac -> to avoid being forced into restarts. I should add that I was able to reproduce in the browser Spotify/youtube, as well as in the desktop Spotify app.
I am playing music 24/7 pretty much, this is very annoying, sort of an Achilles heel on an otherwise perfect device.