Status for BigSur 11.4 (20F71) still broken.
Good for you that your workflow is not corrupted. But to us it feels broken and your workflow is slower (go to top, select the space and then waaaait a very long second) than switching spaces with a 3 or 4 finger swipe.
But try this. Grab a file with "click and hold". 3/4 finger swipe up (left right down, does not matter) and see how broken it feels. You still hold the file but you cannot move it anywhere, while the animation that you expect with the 3/4-finger swipe stopps in the middle.
It only works for me (on my 2013 pro) when the finger "holding" the file (pressing the trackpad) is on the very bottom of the trackpad. Then the swipes work as expected.
In fact i switched to my gaming PC for working now (this issue not being the only reason). I use a mouse there and Windows 10 now also has usable spaces. The MacBook is back to being a couch PC.