I have found the cause. It turns out, indeed, that all my problems of the last years (since upgrading from Mojave to Monterey late 2022) stem from faulty hardware. In fact, the Mac mini was almost certainly already faulty when I bought it new. I bought it for one use late 2018, that did not work out so I only started using it a year later (late 2019). As soon as I upgraded it to Monterey, my problems started. Mostly the system would freeze/panic. I blamed the software I had installed, uses I made (e.g. using sockets, pf, sensor readers, etc.), being used to Mac hardware being expensive but very good. Boy, was I wrong. No wonder every attempt I did to isolate the software problem didn't work. It was hardware. From day one. And around the time Monterey was released, hundreds of people found out I can now see in these forums and elsewhere.
Recently, next to freezing I got the "Volume Hash Mismatch" messages and that has led to finally finding out what is the case. It turns out the memory on these systems is poor and while it passes the Apple Hardware Test, a decent test (memtest86) shows this clearly. While my 2019 iMac without issues passes that memtest86 test, my 2018 Mac mini does not. A single pass results in hundreds of errors.
So, all the effort I put into this, all the time wasted on it, all the grief, it all was because Apple sold me broken hardware, something that came to light from macOS Monterey on. You can go back into e.g. MacPorts archives and see me asking if other people had the same problems. No, because they did not have a faulty system. 4 years I have used this unreliable system, 4 years of problems. Before Monterey, the system was only rebooted unpatches and such. After Monterey, I was lucky if it ran for a few weeks.
My trust in Apple was already low because of these unreliability issues, but the fact that Apple hasn't alerted buyers and provided a decent hardware test, really, really stinks. The fact that I sent in reports to Apple's bugreport and Apple wasn't decent enough to tell me I probably had a broken machine really ****** me off. Why buy expensive hardware when the basic quality is so poor?
Had I tested this machine when I bought it (why would I have, Apple was reliable, right), I could have exchanged it for a working system. Now, the system has been in use for 4 years, and there is of course no warranty. But the way it feels is more like I should get damages compensation.