I've had this problem ever since installing Big Sur on my Mac Mini (Late 2014), 3 GHz Dual-Core Intel Core i7, 16 GB 1600 MHz DDR3 Memory, with 1TB Solid State Internal Hard Drive. Currently at Big Sur 11.4, and problem continued, computer becoming very hot during use and while asleep. In addition, simple warm restarts not requested immediately after restarting often require five minutes or so and indicate computer was restarted because of a problem. Also, Finder 11.4 frequently crashes and sometimes restarts on its own.
Recently I discovered that deselecting the Ken Burns screensaver appears to cure most of these problems. I switched to the Word of the Day screensaver, but others also appear to help. My computer is definitely running cooler, also much cooler when in sleep mode with the screen saver active, and a lot fewer (almost none) Finder crashes or unsolicited restarts during use.