I would suggest the following:
**Make sure (whatever you do), that you have a full and known-good, locally available back up of your Mac that is recent.**
1. Hold-off on upgrading to Monterey for at least one if not two point updates (unless you need to update).
2. Delete the installer app for Mac OS Monterey.
3. Update your Mac mini to the most current non-Mac OS Monterey version of your OS (so, from your post, I believe that you need to upgrade to Mac OS Big Sur 11.6.1)
4. Reboot your Mac mini, then download a fresh copy (make sure your Mac is using an ethernet connection instead of WiFi, as this will give you a better and more stable connection to your ISP).
5. Once the new installer is downloaded, wait for awhile and reboot your Mac again. The reason here is to make sure the package is fully downloaded and you have a clean cache, etc., before starting the install attempt again.
Though not likely, also make sure you are using your Admin account (please make sure you have one and that your daily User account is just a normal user, the reason here is for both safety and security) to do the OS upgrade, as you may have set things on your User account that are causing this as well.
Good luck.