In Intel days lots of mini owners (including me) experienced connectivity problems due to the placement of the BT and WiFi antennas which are in the back near the power button. Third party hard drives and other devices aren’t always well shielded and electronic ‘noise’ could interfere with connectivity. This was often intermittent because if the hard drive wasn’t spinning, for example, the noise wasn’t enough to interfere. The easiest fix was often simply to move peripherals to the left side of the mini.
I don’t have an Apple Silicon mini so I’ve not tested this and can’t, but it should be a simple test to move peripherals to the other side (or further away) from the mini and see if that helps.