HT1338: OS X: Updating OS X and Mac App Store appsLearn about OS X: Updating OS X and Mac App Store apps
Choose About this Mac from the Apple menu and check the processor.
If it's a PowerPC Mac, it's already running the newest OS it can.
If it's a Core Duo Mac, click here, install the DVD, and run Software Update.
If it's a Core 2 Duo or better Mac, follow the Core Duo instructions, open the Mac App Store, and try purchasing Mountain Lion. If you get told it's incompatible, phone the online Apple Store and order a download code for Lion.
Further updates/upgrades will require you to purchase the upgrade. See below for info on Snow Leopard (10.6), which will be required for further upgrades to 10.8.
There are two reguirement to get beyond 10.5.8. One is your Mac must be able to run versions after 10.5.8. The second is that you must purchase the newer version starting with the Snow Leopard install DVD from Apple to get you up to 10.6