HomePod minis say they don't have internet connection

HomePod mini constantly disconnects and says "Entschuldige, Ich hab ein problem mit der Internetverbindung" (Sorry, I have a problem with the internet connection).

I have 2 HomePod minis and 2 Alexas, I will buy more Alexas as they do not have this problem at all. I have a Unifi network with a separate WiFi and VLAN for my IoT devices, Apple HomePods and Alexas are on that WiFi. I had to block MAC address on my router for them to enter my main WiFi where the iPhones are (for Alexa devices not needed, they stay on their configured wifi network).

Now I constantly have HomePods disconnect from wifi or saying they don't have internet connection. This caused me to go away from Apple Home towards using Home assistant as its faster (!) and much more reliable.

Apple used to be up there where everything "just works" now I would say it's "just not working".

Any Ideas how to fix that?