Hi domcaz,
Yes, you can currently downgrade an M1 iMac to macOS Big Sur 11.3 or newer - even if it shipped with a newer macOS. There are two primary ways to do this, and both of them involve an erase.
If you have another Mac running macOS Monterey 12.4 or newer, you can use Apple Configurator on the other Mac to downgrade both the firmware and macOS on the iMac. Download the restore file for your desired macOS version, put the iMac into DFU mode, connect the two Macs, and drag the restore file on top of the DFU icon that appears in Apple Configurator. Choose to restore.
(The IPSW restore file site linked above exclusively uses genuine Apple server links and downloads all restore files straight from Apple. It doesn't act as an intermediary.)
Otherwise, you can prepare a bootable installer for macOS Big Sur, and then use the bootable installer to erase the iMac and install macOS Big Sur. Please note that Apple only provides recent versions of the Big Sur installer, unlike the other option above. Also, some system firmware (and probably Recovery) will remain on newer versions instead of being downgraded.
EDIT: If you're going with the restore option using Apple Configurator, some cables work better than others for macOS Big Sur in particular. I've found the most success using the USB-C to USB-C charge cable that came with my M1 MacBook Air. Apple's 0.8m Thunderbolt 3 cable fails for some reason.