Safari (MacOS) loads extremely slow

Safari sometimes start loading new pages extremely slow (like taking >10 seconds) Closing and re-opening Safari fixes it, but only for a while. I have had this issue for 2 years now. Any tips or ideas?

Notes

• It is not network related, Chrome works like normal
• Updating to Tahoe 26 did not solve it, it was present on previous versions as well
• I have tried deleting all history, cache and cookies
• Turning private relay on or off doesn't change anything
• Had this issue on M1 mac mini, but also on M3 macbook pro. Never on iPhone.