If yours is a PPC (G4 or G5) Mac 10.5.8 is as far as you can go.
If it is an Intel mac:
Start by checking if you can run Snow Leopard:
Requirements for OS X 10.6 'Snow Leopard'
The OS 10.6 Snow Leopard install DVD is available for $19.99 from the Apple Store:
and in the UK:
When you have installed it, run Software Update to download and install the latest updates for Snow Leopard to bring it up to 10.6.8, or download the combo update from here:
Check via Software Update whether any further updates are required, particularly to iTunes.
You should now see the App Store icon in iTunes, and you now need to set up your account:
You can now upgrade to Mavericks OS 10.9 for free IF you have one of the following Macs, with not less than 2GB of RAM, and at least 8GB of available space on your hard drive:
iMac (Mid-2007 or later)
MacBook (13-inch Aluminum, Late 2008), (13-inch, Early 2009 or later)
MacBook Pro (13-inch, Mid-2009 or later), (15-inch, Mid/Late 2007 or later), (17-inch, Late 2007 or later)
MacBook Air (Late 2008 or later)
Mac Mini (Early 2009 or later)
Mac Pro (Early 2008 or later)
Xserve (Early 2009)
If you cannot run Mavericks you can purchase the code to use to download Lion from the App Store (Lion requires an Intel-based Mac with a Core 2 Duo, i3, i5, i7 or Xeon processor and 2GB of RAM, running the latest version of Snow Leopard):
or Mountain Lion: