Updating the RAM and upgrading/updating the OS aren't that difficult for the end user to do.
Before upgrading the OS, you would need to upgrade the RAM.
This is the only user upgradeable/installable hardware in an iMac.
It is, also, not very hard to do.
First you need to purchase the additional RAM.
Your iMac can take 4 GBs of RAM max.
There are two reliable online sources for Mac RAM. Crucial memory or OWC.
RAM for iMacs is fairly cheap to buy, now.
Here's the specs of the RAM your iMac takes
3.0 GB (4.0 GB can be installed, only 3.0 GB will be addressed)
Memory Slots 2 - 200-pin PC2-5300 (667MHz) DDR2 SO-DIMM
And here's the procedure for replacing and installing the RAM in your iMac.
http://support.apple.com/kb/HT1423
As you can see from the diagrams, this is not too hard to do.
Next, you will need to purchase a retail copy of OS X 10.6 Snow Leopard. This disc can be directly purchased from Apple by calling 1-809-MYAPPLE. Ask for retail software sales. They will have OS X 10.6 for $19.99 plus shipping.
Once you receive this disc, before installing OS X 10.6, if your iMac has a lot of data on it that is important to you that you wish to save, the easiest and most direct way to so this is to purchase an external FireWire 400 hard drive and copy all of the data from your iMac to your new external hard drive. This is a backup or safety for your data Innthe event something does go wrong with upgrading/installing your new OS.
Search online for FireWire 400 external Hard drives. Should give you results where to find and purchase.
You need to download a program that will exactly copy of the contents of your iMac's hard drive to the new drive you just purchased. There are two programs that do this. One is called CarbonCopyCloner. The other is called SuperDuper.
Either one of these programs is capable of copying/duplicating the contents on the hard drive of your iMac.
Once you have done this, you can now insert the install disc, you purchased, and follow the instructions to begin the installation process of OS X 10.6 Snow Leopard.
There are install options I am not sure which options retains all of your data and all the various settings already setup in your iMac.
It's either going to be an Update Install or an Archive Install.
Someone else maybe able to tell you this for sure.
Once the installation is all completed and your iMac is back up and running and everything appears to be functioning OK.
Under OS X system preferences panel click on the Software Update icon.
Click on find updates to update your system to OS X 10.6.8. And, also, to update any other software on your iMac.
Once allof this is all complete, locate your name home folder, Applications folder, Utilities folder and look for the Disk Utility app. Dbl. click this app to launch it . Do a permissions repair on your iMac's main drive.
Once your iMac is up to date, your dock should have an icon for Apple Mac Apo store in it now.
If you choose to upgrade the OS further, you need to click on the Mac App Store icon in your OS X dock and search to see if OS X 10.7 Lion is available for purchase and download.
If OS X 10.7 Lion isn't available for purchase and download , you may have to contact Apple software sales, again to see if it can be purchased in a different manner.
You could also do another software update in the system preferences panel to see if the option to upgrade to OS X 10.7 Lion appears, also. It may not, but won't hurt to try this.
My advice, since your iMac is fairly old, I would stay on OS X 10.6.8 and NOT upgrade to OS X10.7 Lion.
This is the best I can do to walk through what is a fairly involved , but very user doable OS upgrade.
I know there is a chance I missed something so, hopefully, other really experienced users here will fill in any gaps I may have missed.
It seems like a lot of steps until you've tried this once.
Doing OS upgrades isn't quite as bad the more times it's done.
If you choose to,try this yourself, then good luck!
Every one here in these discussion forums is here to help you if you need it.