I'm having similar issues, sluggish, mouse and scroll bar not responding, etc. I clear cache & inactive memory often, same with history. However, I've been running into even more issues when I re-open Safari, and try to restore my previous session, by using the re-open all windows option: OVER HALF MY WINDOWS DISAPPEAR, or all that's left are the titles in the tabs!
They also are no longer in my history, despite my having not deleted them, so ***????
To be using Macs for many, many years, and see the things that used to work much better, now all buggy & difficult to work with, is quite disappointing, frustrating, and aggravating....
I sure shouldn't have to spend a lot of time reconstructing manually & laboriously, all the windows Safari didn't re-open. If I didn't have to keep closing & opening it just to drop it's memory usage down, this wouldn't even have to ba an issue.....smdh