While I don't do the sort of work you do, I am doing some embedded Linux development with Fedora Linux as host with the Gnome desktop environment (sometimes XFCE) using VMware Fusion on an M1 MacBook Air with 8GB RAM/512 GB SSD. I have it set up with 4GB and 2 cores and seems to be okay. I used Buildroot which basically builds the entire build environment including C/C++ cross compilers and Boot and the Linux kernel and BusyBox. It takes a while to do the full build, but with all that is being done, that is not a surprise.
I don't know if what you are doing would be that intensive, but I just throw that out as an example. I will also note, that I don't do that much on the macOS side when doing that except maybe for a Safari window open to a site for reference info.
The challenge may be whether the required apps are aarch64 native or whether the will be needed to be built from source as only aarch64 apps can be virtualized on an M1.