Hi,
I had exactly the same issues as everyone. The main issue I found is lack of DRAM. On OS 3.1.3 I could free up around 45MB using Memory Info (before apple removed that feature from the Apps Store - what's THAT about?). On IOS 4 that dropped to a max of 16MB, but regularly was in single figures.
I persevered for about 2 weeks but the bad performance definitely outweighed the good additions from IOS 4. There are loads of tutorials on YouTube on how to downgrade to OS 3.x.x
The one I followed was on
http://www.wonderhowto.com/how-to-downgrade-your-iphone-from-ios-4-back-3-1-3-37 7414/
The prerequisites are:
1. A downloaded OS 3.1.3 (or 3.1.2 or whatever) - just Google it again...
2. Download RecBoot to force your iPhone out of Recovery Mode once finished
The whole process took less than 30 minutes and I now have a zippy iPhone 3G running OS 3.1.2 again. I didn't have to downdrage my iTunes either, I'm still running 9.2 Great!
Good luck!