Randomly cannot click on websites

Safari version: 17.4.1

macOS: 14.4.1

This problem is driving me crazy. Randomly, on any website (there is no specific site) I cannot click on anything on the page. Safari is not frozen, I can change tabs and such settings come up. If I restart Safari and go back to the site it works just fine. I have a few extensions, I can swear I've disabled them all and it happens again. I've cleared the cache, rebooted, etc, etc and it happens again. Sometimes not for a few hours, sometimes ten minutes later.