Fusion drives can be slow because they rely mostly on a mechanical drive, especially for large audio and video files. That access is very slow. In addition, with audio and video software, one needs typically to page to disk and that is very slow when the mechanical drive is pulled in.
My experience with a 2008 iMac with just 4 GB RAM and an internal HDD was that when the HDD was replaced with an SSD, it was like a new machine. I think the SSD is much more important than the memory.
I suggest you have a look at this: Use an external SSD as your startup disk … - Apple Community
With that ancient 2008 iMac, and more recently with a 2015 iMac (just 8 GB memory), the internal HDD was replaced with an internal SSD. The 2015 iMac can run Monterey and it is very fast now. I did not use external SSD as the boot drive but I think that would have been almost as fast.
I think you should have a look at the link above and also ponder putting in a new SSD internal drive. Even if you decide to go with the internal drive, you can start with the external to see how fast it runs. If you end up with the internal SSD, the external one can become your backup drive. One nice thing about the external boot drive is that you can continue to use your internal fusion drive for misc. storage, which could be convenient.