This may have been answered somewhere but I didn't see it. So I'm answering myself in case it helps someone else.
Safari doesn't reload upon hitting back, it only reloads when you use the two finger swipe back on the trackpad. So I've jsut gone back to hitting the back button which doesn't do this. Kind of dumb to have them do different behaviors but whatever, anything that fixes the problem I guess.