I've looked into it and found many ways to uninstall OS X server.
One is to download a dedicated uninstaller like AppZapper.
I personally use AppZapper to uninstall lots of apps and it always finds all of the programs related files.
Another option is to use MacKeeper.
I also use MacKeeper, and it is a great app.
I used its uninstaller feature in it to successfully uninstall server.
I know because no matter what I type into spotlight, nothing regarding server pops up.
There's also a shredder option in MacKeeper that will find any files related to whatever it is you want to delete.
Hope this helps.
Note: I know many people recommend against using MacKeeper, but I can assure you that it is completely safe.
I use it all the time, and not only is it NOT a virus, it actually sped up my Mac by clearing old cache and logs files.
I can assure you that I am in no way affiliated with MacKeeper or its creators; ZeoBit.