new operating system

Welcome to the Apple Support Communities

First, check that your Mac is supported > http://www.apple.com/osx/specs If your computer is supported, open the App Store (you can find it on the Dock) and purchase OS X Mountain Lion, so it will start being downloaded.

While OS X Mountain Lion is being downloaded, make a backup of your files with Time Machine and/or Carbon Copy Cloner, and check that your apps are compatible > http://www.roaringapps.com It's so important to do this, because PowerPC applications will stop working after upgrading to Mountain Lion.

When the download finishes, the OS X installer will appear, so follow its steps to upgrade to Mountain Lion. After upgrading, you will have OS X 10.8.3

First be sure your Mac meets the system requirements.

http://www.apple.com/osx/specs/

Also be sure your apps are compatible

http://roaringapps.com/apps:table

Then go to AppStore and purchase ML.

You also might want to check the model number to see when your iMac was produced:

Model Number - iMac

Model Numbers

If your computer isn't compatible, you might be able to upgrade to Lion. Computer Compatibility - Lion