OK, I think I have a working solution.
After searching a lot on the web the past few months I found talk about kernel panics and incremental MacOS updates not always replacing everything they should replace with new versions (especxially the kernel for instance) and not always cleaning up old mess.
The advice was to install a MacOS Combo Update. A combo update can only be installed over the previous or current version of the same OS (10.15.3 combo update over 10.15.3, 10.15.2, 10.15.1 or 10.15 but always Catalina over Cataline, not over Mojave for instance).
A combo update makes sure everything is replaced, cleans all caches etc ...
So what I did is:
- make a backup to be sure.
- closed all applications
- rebooted
- closed all TSR apps that could be closed in the taskbar
- ran the 10.15.3 combo update over 10.15.3
It's been about 30 hours now, the computer went to sleep numerous times, I rebooted it a bunch of times, shut it down and powered it up again a few times, all three situations that always resulted in the kernel panic on my machine with a couple of TB3 drives, a USB-C drive, displayport over TB3 and heaps of USB devices connected.
Result: not even 1 kernel panic or crash when sleeping, restarting or rebooting.
As a bonus my Time Machine backup to a Synology NAS works without problems again as well (it used to ask to do a complete new backup every hour).
Here's the link to the combo update for 10.15.3: https://support.apple.com/kb/DL2030?locale=en_US
I hope it helps for others as well!