Congratulations on having a second backup plan, unlike some advice on this thread to delete the one and only backup you may have, before doing a reinstall. What a terrible idea! As they say, one backup is no backup...
As you discovered, the server build is a different beast and is not necessary for most people. Once it is installed, the only way to go back to a normal system is by an erase/install, so again, an external bootable clone of the original system would be a good idea.
I had the slow TM backup problem, as can be seen in Activity Monitor (AM) by looking at the process "backupd" which had stalled and I could have waited for ever for it to finish.
For the TM problem, three things I can think of:
1. In Spotlight preferences>Private, exclude all disks/partitions not active. Wait for Spotlight to complete it's initial indexing. This is "mdworker" in AM.
2. In the Backup Preferences, exclude all disks/partitions not active as well.
3. I have found with ML that I cannot copy my iPhoto library to an external backup while iPhoto is open. I would therefore suggest closing iPhoto (and perhaps iTunes) during the backup process, in case they are blocking it.
I hope this helps.