How do I upgrade Mac Pro Server 10.5.8 software?

Much better to do a clean install.

Clone your 10.5 install to another drive. (make sure you can boot from it)

Then do a clean install (erase drive first) of whatever OS you want to use.

You'll need to resetup services.. but its much easier to do this than to deal with all the possible issues that can come from an upgraded OS.