An interesting little detail about DNS services... In terms of what happens between your DNS client and your DNS server, there's no particular difference between the processing of "whatever.private" and a real and registered domain. It's all about traversing the DNS directory tree. The salient difference between a bogus domain and a real domain is in the namespace coordination; a real and registered domain is globally unique, where a made-up domain can collide, and ICANN is bringing a number of new top-level domains online.
The ~$10 per year registration fee is a form of insurance against namespace collisions.
Do stay out of .local, as mixing multicast DNS (Bonjour) and unicast DNS (traditional DNS) can encounter weird problems.
As for your question around using a .private domain for Software Update per HT4069, sure, that should all work fine.
I have encountered some little software that expected — required — a two-dot domain name, FWIW. Something in the form of "foo.example.private.", though whether that's a bug or a feature, donno. I've been using registered domains for a while, so didn't end up researching that.
As for your lament around the documentation, welcome to running a server. The Snow Leopard Server documentation is still available at the Apple documentation web site, and that's more detailed than is the current Mountain Lion stuff, but it's still fairly common to need to use the command line and to research any associated open-source packages directly.
Thanks for the clarification! So far, I have been trying to get comfortable with the server in a local environment - at home with a couple of iMacs, Macbook Pros etc. But eventually I might venture out to truly making the Mac mini a real server, accessible from the outside, at which point I would consider a proper / real / registered domain name
Haven't yet explored how to go about having a globally accessible domain at home.... (familiar with domain registration etc. but those have been hosted elsewhere). I am assuming I have to have a static IP address (unless I go for DynDNS or such) and/or tell my ISP to map the domain name to the static IP address? Will have to look into it when I get a moment
The domain registration? That part is pretty simple. Find and register a domain. Then use that domain on your internal systems.
Don't bother setting up any external DNS translations for the new domain, and undo any that the registrar has established on your behalf.
When you install, specify the domain name as part of the host name for your OS X Server box.
You're running an OS X Server box and that means — like most other modern servers — you want and will usually need local DNS services operating on your LAN.
Getting DNS services going can be a bit more of a learning experience than getting the domain itself registered, but — once you grok the pieces — and particularly as your LAN grows in scope, you'll likely find having local DNS is very useful. You won't be using IP addresses or copying around the hosts files, for instance.