I returned my first one and got a new one. The new one does not seem to be affected. Here are my observations:
My first one was a spacegray base model and the error occurred on both a time machine restore as well as a complete fresh macOS installation on 10.15.4. If I waited long enough (maybe 2-3 hours, definitively over night) the kernel panic happened every time. I've also had problems when I first tried to restore my time machine backup. The mac out of the box had a lower version than 10.15.4 so it offered to update first on the initial setup. This update always failed and I wasn't able to restore my backup until I manually erased the disk and did a clean install. After restoring my time machine backup I again performed a complete fresh install but the error was still there.
So I returned this macbook.
I've ordered a new one in silver this time, also base model. The update to 10.15.4 during the initial setup worked without a problem on the first try. I've restored the same time machine backup and it didn't crash again. I've tested it 2 times after 2-3 hours and over night. The previous one always crashed over night. This one did not crash yet.