Those large negative error codes indicate a network issue of some sort when booting into Internet Recovery Mode. It is best to connect the Mac directly to the router using an Ethernet cable for a faster more reliable connection that bypasses any hard to troubleshoot WiFi issues. You can also try power cycling your router (wait at least five minutes for your router to finish booting). You can also try a PRAM Reset in case the Mac is stuck booting to a previous location regardless of your selection of startup keys (hold the PRAM Reset for at least two chimes). Sometimes your router settings or your ISP can interfere with being able to boot into Internet Recovery Mode.
As for the Mini booting into Internet Recovery Mode when booting from the USB installer, this is likely due to the fact you probably never modified the Mac's security settings to allow booting from USB. When you attempt to boot from USB without first configuring USB booting, the 2018+ Mac will boot into Internet Recovery Mode so that you can authorized the Mac to boot from USB. See this Apple article for details:
About Startup Security Utility on a Mac with the Apple T2 Security Chip - Apple Support
Or it may mean the USB installer stick you created has a problem. Macs can be very picky about the drives used for booting plus the quality of USB sticks is extremely poor (even from reputable brands). You can try using another USB stick (especially a different brand), and also try using another USB port and/or adapter as well. Connect the USB stick directly to the Mac and even disconnect all other external devices which are uneccessary for this process.
Also, make sure to use the instructions in this Apple article for creating a bootable macOS USB installer:
How to create a bootable installer for macOS - Apple Support
Make sure to download the installer directly from Apple. Using any other installer or using any other method for creating the bootable macOS USB will most likely fail (from posts I've seen on these forums).