When I use finger swipe to go back, Safari freezes

This has just started happening in the past couple of days -

I have an iMac, updated to Lion, magic mouse -

If I use the one finger swipe on my mouse to go back a page in Safari, it will take me back to the previous page, but then I can't do anything on that page or in Safari at all sometimes. If I try to scroll up or down, the page gets scrambled and doesn't really move but around the edges...

If I hit the back button on the tool bar, everything is fine (so far) - but this one of the most-used features and I'd like it to work.

Any ideas?