An interesting little detail about DNS services... In terms of what happens between your DNS client and your DNS server, there's no particular difference between the processing of "whatever.private" and a real and registered domain. It's all about traversing the DNS directory tree. The salient difference between a bogus domain and a real domain is in the namespace coordination; a real and registered domain is globally unique, where a made-up domain can collide, and ICANN is bringing a number of new top-level domains online.
The ~$10 per year registration fee is a form of insurance against namespace collisions.
Do stay out of .local, as mixing multicast DNS (Bonjour) and unicast DNS (traditional DNS) can encounter weird problems.
As for your question around using a .private domain for Software Update per HT4069, sure, that should all work fine.
I have encountered some little software that expected — required — a two-dot domain name, FWIW. Something in the form of "foo.example.private.", though whether that's a bug or a feature, donno. I've been using registered domains for a while, so didn't end up researching that.
As for your lament around the documentation, welcome to running a server. The Snow Leopard Server documentation is still available at the Apple documentation web site, and that's more detailed than is the current Mountain Lion stuff, but it's still fairly common to need to use the command line and to research any associated open-source packages directly.