Your not alone. Safari has always eaten up way too much memory and eventually slows to a crawl and sometimes crashes which Firefox never does. At least for me anyway, after it hits >1.75GB I'm on borrowed time. I want to use it as my default browser, always start surfing the web with Safari, but then switch to something else after I notice the memory use piling up. Quiting and restarting fixes the issue but how is that a solution?
I haven't even bothered with 5.1 because I've heard it's buggy. Will try with 5.2 or whatever when that comes out. I can try the 32-bit mode but that's is just silly, its a 64-bit os and I'm not keen on losing the benefits of runnig safari in 64-bit mode. I'll try it though just to see what it does memory wise.