Ok, I have everything working except swiping between fullscreen apps.
1. Go to System Preferences --> Trackpad --> More Gestures
2. Uncheck Swipe between full-screen apps, Mission Control, App Expose, Launchpad, Show Desktop.
3. Set Swipe between pages to Swipe left or right with three fingers or Swipe with two or three fingers. Notice that you should be able to swipe back and forth with three fingers in Finder, but you can't drag with three fingers.
4. Go to System Preferences --> Trackpad --> Point & Click
5. Three finger drag should be disabled. Enable it. Notice now that four finger side swiping will go back and forth in Finder. Safari still retains scrolling left and right to go back and forth in history. But now, three finger dragging will work.
6. You can go back to System Preferences --> Trackpad --> More Gestures and enable Mission Control, App Expose, Launchpad, and Show Desktop. Don't enable Swipe between full-screen apps or else you'll undo everything you just did.
It's a small trade off, but I use Mission Control to see all my apps anyway.