I was having the same problem. This is a percieved delay.
By default, Safari uses it's cached version of the previous page to refresh the window when navigating back, and when it's pulling from its cache, the refresh happens quickly enough that we don't really notice. When Safari refreshes the page from the server, we see a longer delay while it does so. We're used to both of these scenarios from pre-swipe-gesture days.
Under normal circumstances, Safari will only refresh the page from the server either because it's been told to do so by the webpage, or an extension requires it. For example:
Many web pages have a javascript onLoad call to refresh the page every time it's loaded (this includes when it's loaded from the cache). So when Safari reloads the page from its cache, the javascript in that page forces safari to do a refresh from the server. This is why disabling Java fixed it for some people.
Other web pages use HTML code to achieve the same onLoad server refresh or to prevent a web browser from caching the page in the first place. Since it's coded in HTML, nothing will disable this behavior, this may be why nothing works for some people.
Some individual extensions or plugins require a server refresh on page load, this is why disabling some or all extensions fixed it for some and disabling plugins fixed it for others.
FYI, (for testing) Gawker sites use one of these refresh-on-page-load strategies. In either of these cases, Safari is forced to refresh the page from the server.
When you use the swipe-back navigation, Safari performs the animation you see where the page slides off to the right, revealing the previous page below it. Safari does this by substituting the "live" webpage with a snapshot of both the current and the previous pages to perform the animation. Once the animation is complete, Safari then reloads the page from its cache, and if instructed by the webpage, refreshes it from the server. If the page included the HTML meta tag "NO-CACHE", then there is no page in Safari's cache to draw from and it has no choice but to refresh it from the server.
You can see this snapshot substitution happen via the Debug menu. (look back on this thread or google for how to enable it.) Select Debug > Miscillaneous Flags > Discolor Fluid Swipe Snapshots
Now navigate to a page and swipe back. You'll see the snapshot that Safari is using to show you both the page you're on and the previous page during the animation because it'll be grayed over with a dull brown-ish color. The discoloration shows you that you're not seeing the live webpage itself, but Safari's snapshot of it. Once the animation completes, you'll see the discoloration go away and the previous page is fully displayed. Safari then reloads it as explained above.
Now go to System Preferences > Trackpad > More Gestures, and change "Swipe between pages" to "Swipe with two or three fingers". Swiping with three fingers bypasses Safari's swipe-back animation. Now navigate using a three finger swipe and you'll see what Safari is doing without the animation, and it feels normal.
The percieved part:
The swipe-back animation makes you take notice of the fact that it's refreshing. Because you're seeing the page displayed underneath the one that's sliding out, you expect it to remain static once the animation finishes. Then it refreshes and you're like, "What the ****? it was just there, why did it have to refresh?" Without the animation, it feels normal.
To fix it, perhaps Apple could code Safari to keep recent history pages open and active in seperate processes (like it currently does with open tabs and windows) and display that at the end of the snapshot animation instead of pulilng from its cache. It might be a worthy memory trade-off. PSST - you listening, Apple? 🙂
As it sits, the animation is a cool idea, with a no-win implimentation.
Unfortunately, I have not found a way to disable the animation on a 2-finger swipe. So when I browse Gawker sites and other regulars such as Digg and Reddit, which I know force a refresh, I'll use a 3-finger swipe to navigate.
Hope this clears up some confusion.