Hi I had the exact problem as you do, when trying to install Big Sur 11.4 on my Mac mini external portable SSD (SanDisk extreme Pro v2).
But on my previous Mac mini m1 I had successfully install Big Sur 11.2.3 on external portable SSD (Samsung T7) some time ago. After that I did not touch internal Mac OS (did not update Big Sur) and used installed one on Samsung T7 and later updated to Big Sur 11.4 when it come out.
So I connect SanDisk to Mac Min that was running Big Sur 11.2.3, download "Install macOS Big Sur" and was able to successfully install Big Sur 11.4 under running Big Sur 11.2.3. Then I just connect SanDisk SSD to the newest Mac mini and set SSD as first bootable. And now I have two Mac mini running 11.4 with no issues.
So my assumption here is that there is some problem with installed 11.4 version as all other parameters where the same (both mac mini were base version 8/256). Also I saw that there are 11.5 Release candidate 13.07.2021 so you can try to update BigSur.