OK, I'm starting to understand why my "complete" installation from scratch was a solution, but from what I'm reading it looks like many don't need to go that far.
I see a lot of comments that talk about signing off and on again with iCloud account on the Apple TV which should rebuild the config from iCloud on the Apple TV.
What I forgot to mention in my previous reply, is that also my old installation was on another iCloud user account (my wife's account) on the Apple TV, which I have renewed to my own iCloud account while settings things up again. SO, in fact everything is installed on a "fresh and unused iCloud account".
So, I'm really starting to believe the rumors that it might be caused by corrupted data on the Apple TV which requires a reset to rebuild the config again.
I can also confirm what many others are noticing, that with my newly installed home I have the feeling that all devices are more responsive than before.
This is the case for Shelly devices, devices linked to bridges via Home Assistant, even RF and IR devices that are running via a Broadlink plugin on a Homebridge that is running on docker in Synology. I really don't notice any delays in any of the manual tasks or automations anymore.
I would still try to play around with the accounts on the AppleTV to get it solved.