You need to find out what model of iMac you have (what year). This may be visible is "About This Mac" under the Apple menu, or the System Information app that you can click from there.
From this, you can find out whether your Mac can upgrade to Mavericks 10.9, or Lion 10.7 (if it's older).
Next, you need to buy more RAM. Many web shop selling RAM, like Crucial.com let you select your Mac model, and then they teill you what RAM you need.
Installing the RAM is pretty easy: unscrew a plate on the bottom edge of the iMac, remove the old RAM and add the new. There are videos on the web showing you how to do this.