Q: This version of Server does not support upgrading the Server data on this volume

This is right!

But this was not your question in the beginning...

You have to decide to create a new server (EL Capitan) from the beginning or to stay at your old server.

If you made an archive of your OD-Master, you can import it.

Which services did you use?

Services running...DHCP, DNS, Open Directory, Web - these I'm not too concerned about exporting/importing, but there are fairly complex customised setups for Mail, Message, File Sharing which could take an age to try and reproduce, without the extra load of worrying about the actual data involved.

I think, you have to go from the scratch like you asked in your initial post.

Filesharing shouldn't be the problem, because the directories are there and you should be able to integrate them again.

The groups should be in the OD-Backup, if you made it with password.

That's what I had concluded - just hoping you had spotted what was stopping the upgrade at less brutal level.

Removing the Library/Server directory did not do anything for me.  

You shouldn't delete the folder! Just the content!

If you want help, we need more inputs (what do you want, what did you do...)

Hello, I also had the same problem when making mac Server installation on my system. I upgraded version 10:11 to 10:12 and now I can not use the App Server. Apple could help in this matter, no?

Works for me!

Did you try to remove all the contend of /Library/Server ?

Leave the folder on its place.