I've given up on Big Sur until there's absolute certainty that a new release addresses Fusion Drive issues.
I've wasted weeks of my life dealing with this. Things would slowly devolve to being spinning balls waiting on anything to happen, lag on clicking on items in Finder or Apps, slow bus speeds of copying files, long boot times, and finder/files taking even longer to show up.
I've done many of the aforementioned processes. Reinstalled Big Sur over the existing, and it would work better for a bit, but then back to issues a day or two later.
While I had a backup of my computer already, I was concerned about a couple things (including a VMWare Virtual Machine) that may not have been backed up fully or as recent as I needed it. The ability to backup from this troubled Fusion drive to a USB, a WebDav, or a Thunderbolt Target Drive to another Mac was moving at a snails pace... it would have copied a few gigs in days!
I finally decided to just format the hard drive/resetFusion, and install Big Sur 11.3 clean from an external drive. Did not install many apps for a few days, and just used it. Let it sit on Desktop several times and reboot like it's been suggested too. Install a couple nominal apps and moved files from backup (not any system/library files at all). It all came right back eventually.
What is obvious is that something in Big Sur is affecting the Fusion partitions such that even acting as a Target Drive (not running OS, but simply a connected drive), is a problem to use it. Combinations of Safe Mode, Recovery mode, reinstall OS over existing, will at least make it usable for a while so you can pull files off it.
I formatted again, clean installed Catalina, including all my files and apps. I have NONE of these issues with Catalina. Days now... quick boot, no spinners, all is good.
I can't keep wasting my time on this and will stick with Catalina until I learn something new about a release that Apple acknowledges.
iMac (Retina 5K, 27-inch, 2017)
3.8 GHz Quad-Core Intel Core i5
40 GB Ram
Radeon Pro 580 8 GB
1TB Fusion