To upgrade to OS X Mountain Lion Server, first you have to upgrade to OS X Mountain Lion, so first of all, check that your computer is compatible > http://www.apple.com/osx/specs
If it's compatible, make a backup of your data with Time Machine and/or Carbon Copy Cloner, and check that your applications are compatible > http://www.roaringapps.com
After checking that, open the App Store and purchase OS X Mountain Lion. When the download finishes, the OS X installer will appear, so follow the steps to upgrade to Mountain Lion.
After upgrading, you will be using the client OS X Mountain Lion version. Now, open the App Store again and purchase OS X Server. This will be downloaded as an app, so just open it and follow the steps to set up OS X Server