I recently installed Lion on my mac mini (2GB RAM, 320GB HD) only because I had to, since mobileme is EOL by June. No other feature of Lion, beyond iCloud, would make me upgrade out of fear of losing the decent speed of Snow Leopard.
However, Lion runs just as fast and I may get the beachball rarely but the overall experience is not different than SL. Lion was installed on top of my SL setup, so I suppose a clean installation would make things even better, but I didn't have the time, and I also wanted to test this way of upgrading.. Lion alos runs on my early 2008 MBP after a clean install and there are no speed issues there as well. Now, I am not a developer and I don't edit movies, but for the everyday tasks I use my Macs, Lion is on par with SL on terms of speed. Oh, and I could live without the constant animations of app windows in Lion...
For a relatively recent switcher as me (I came on board in 2005), running decently the latest OS in 5 yrs hardware is more than enough to make me stick to the Mac. Apple respects customers in a way other PC users can only dream of...
So, trust your mac and upgrade it!