First the difference in names. .local is the bonjour address. Your .private is a DNS address. While .local can be used for both, it is highly discouraged.
So in Server Admin > DNS, you want to be on the Zones tab. In here you will want to do the following:
Prep: Let's assume you want to define your system as host name server, domain name myfamily.private. So the fully qualified name is server.myfamily.private. Change the examples below to match your needs.
1: Click on Add Zone and select Add Primary Zone (Master)
2: In the Primary Zone name, enter myfamily.private
3: In the Nameservers box click the + button.
4: An entry will appear. The zone colume should read myfamily.private and the Nameserver Hostname will likely grab your Bonjour name and enter <bonjour_name>.myfamily.private. (Please not many items in DNS end in . This is really important do not remove them. If you need to, override the <bonjour_name> and enter server in its place. This way, the Nameserver Hostname colume should say server.myfamily.private.
5: You can Save at this point if you want
6: Now, click on the Add Record button and choose Add Machine (A)
7: When the panel appears enter the following:
Machine name: server.myfamily.private.
IP Address: enter your fixed IP address of the server - this can be found in System Prefs > Network and should be the IP address assigned to the Ethernet port. For illustration, lets assume it is 10.0.0.10
8: Click save
9: If you are paying attention, you will note that a second zone file was created automatically for the reverse zone, It will have a name of 0.0.10.in-addr.arpa. This is the reverse zone file and contains number to name mapping (where the primary zone is name to number).
Now you have two valid zone files. Check your work by doing the following:
1: Open Terminal
2: Enter this command:
nslookup server.myfamily.private 10.0.0.10
3: The result should be 10.0.0.10.
4: Now test again with nslookup 10.0.0.10 10.0.0.10
5: This tests the DNS running on the machine to return the proper results to you.
Now, that it is working, go back into System Preferences > Network and make sure the first DNS server address is the address of your server. This way you will talk to your system first for name resolution. But since your systems is SOA for myfamily.private only, nearly all requests will be past to the public DNS servers provided by your ISP.
If you are good to this point, then go back into Server Admin > DNS and remove all those other zone files.
Ok, if you got this far, then reboot the server.
After the reboot, repeat those nslookup commands and make sure you are still resolving as expected.
Now, to check if there is any lingering damage. Since it sounds like you have not configured any/many services, this is a good thing. So enter this command in Terminal and tell me the results:
sudo changeip -checkhostname
Now, you will need to clean up after your server.
