Slow Response Smart Lights with Multiple HomePod Minis

I have 3x HomePod Minis (1 in the Kitchen and 1 in each bedroom). I also have multiple Nanoleaf Essentials & Philips Hue bulbs throughout my house.

I use a few Philips hue dimmer switches (configured through home app only and NOT the Hue app) to control them for each room.

I have noticed that all the lights (Nanoleaf & Philips Hue) respond VERY slow whenever the “main” (connected) Home Hub switches to either of the bedroom HomePod Minis and the Kitchen is in “Standby”.

When the Kitchen HomePod mini is marked as “Connected” everything works perfectly.

From what I’ve seen/read, there is no way to mark/choose which device is the default main home hub. The only way I’ve been able to get the kitchen HomePod back as the connected one is to unplug the others for a minute before powering them back on.

This works, but just about everyday at some point it switches back to one of the bedroom HomePods and the issue keeps coming back.

Any thoughts or ideas?