After Updating to Server 4.1 Open directory and LPAD gone

Looks like there's a corruption, and that can mean restoring from backups.

That written, it's possible there are some DNS issues here as your host has a public CNAME (alias) and what might be a dynamic DNS provider. DNS configuration issues can cause LDAP to become confused.

I had a similar problem. A couple days after upgrading, I encountered OD's "Unable to load replica" problem and had my server's certificate deleted from my system keychain!

Server.app + OD + LDAP are all extremely fragile and I just don't trust them during transitions, so I always keep an independent bootable backup with Carbon Copy Cloner and this preflight script. I'll post my notes for recovering OD below, but in my case, nothing worked this time, and I couldn't start OD robustly across reboots. Fortunately for me, my 12 hour old bootable backup was working, so I just used CCC to copy my bootable backup back. Not sure what I would have done had that not worked short of rebuilding everything from scratch.

Pre-steps:

0. Bootable backups, Time Machine backups, and dirserv backups of everything.

1. Disk Utility: Fix disk permissions, Fix disk

2. PRAM reset, Command-Option-P-R at boot

3. DiskWarrior to rebuild the disk directory

Possible steps to fix OD:

# Fix Open Directory "Unable to load replica"

# Try this first:

# https://support.apple.com/en-us/HT200018

# Quit Server.app

sudo mkdir /var/db/openldap/migration/

sudo touch /var/db/openldap/migration/.rekerberize

sudo killall PasswordService

# Open Server.app

# Try this second:

# http://apple.stackexchange.com/questions/79141/how-to-fix-failing-open-directory -database-cn-authdata-cannot-be-opened-err

sudo serveradmin stop dirserv

sudo launchctl unload -w /System/Library/LaunchDaemons/org.openldap.slapd.plist

sudo db_recover -h /var/db/openldap/authdata/

sudo /usr/libexec/slapd -Tt

sudo launchctl load -w /System/Library/LaunchDaemons/org.openldap.slapd.plist

sudo serveradmin start dirserv

# Try this third:

# https://discussions.apple.com/thread/6018956

sudo serveradmin stop dirserv

sudo slapconfig -restoredb /private/var/backups/ServerBackup_OpenDirectoryMaster.sparseimage

sudo serveradmin start dirserv

# Try this fourth (assuming ccc_preflight od backup):

# https://discussions.apple.com/thread/6018956

sudo serveradmin stop dirserv

sudo slapconfig -restoredb /private/var/backups/odbackup/od_2015-04-11.sparseimage

sudo serveradmin start dirserv

# Try this last:

sudo rsync -va /your-backup-drive-possibly-TM/private/var/db/openldap/authdata/ /private/var/db/openldap/authdata/

If your server cert gets deleted from the System keychain, you'll need to boot into the bootable backup and export the certificate+key that looks like hostname.domainname.tld, signed by IntermediateCA_HOSTNAME.DOMAINNAME.TLD_1, copy this to the server drive, import back into the System keychain. The cert should then appear within Server.app again. See here for how to do this if all you have is the System keychain file.

If anyone has reliable advice how to fix a corrupt OD that would be a huge help.

Essandess, the steps you listed were a great help for me - thanks!

What worked for me was to restore /var/db/openldap/openldap-data/id2entry.bdb from timemachine until I could run your second surgestion without any errors. After that OD finally started up and all network users were back. Last thing was to set the correct cert for OD.

Glad that worked -- I'd test it across reboots, which was an issue for me until I threw in the towel and reverted to a 12 hour old clone.

I would have liked to just restore the files necessary to get Open Directory back, but I don't have a list of what these are.

When I restored using Carbon Copy Cloner, I asked it to make a Safety Net, which shows all the stuff it replaced. In case it's useful to others (or myself) in the future, here are some likely candidate directories to attempt restoring from backup:

/private/var/db/openldap

/private/etc/openldap

/private/etc/certificates

/private/var/db/krb5kdc/kdc.conf

/private/etc/krb5.keytab

/private/var/db/krb5kdc

/Library/Keychains/System.keychain

/Library/Preferences/com.apple.openldap.plist

/Library/Preferences/edu.mit.Kerberos.kadmind.launchd

/Library/Preferences/OpenDirectory

I don't know which specific restoration fixed my specific problem, but it had to do with the contents of one of these directories.

The changes in /private/etc/openldap were ldap.conf, rootDSE.ldi, slapd.conf, slapd.d, slapd.d.backup, slapd_macosxserver.conf.

The changes in /Library/Server/* were: Alerts, Calendar and Contacts, Configuration, Firewall, Logs, Mail, Messages, Network, PostgreSQL, ProfileManager, SetupWeb, Wiki, but I don't believe the problem is in the /Library/Server directory.

Another last resort option is to try a Safe reboot to wipe away all cache. Also, be sure to watch your System keychain if you run the command "slapconfig -restoredb" because I've seen this command delete the server certificate from the keychain at times, which will render LDAP authentication useless unless you have a backup of your certs.

The real solution would be an LDAP implementation that isn't fragile to occasional/inevitable db corruption or upgrades. One shouldn't have to think about reverting an entire server or deleting a directory master just to fix the relatively frequent LDAP corruption problems.

Not booting could be a disk problem. I've seen this even on new boxes with SSDs. I would try all the pre-steps above as a necessary precaution.

Try Command-R boot to use the Recovery partition>Disk Utility to fix the disk and permissions.

If that fails—or even if it works, use Disk Warrior, and use Disk Warrior anyway for regular directory rebuilds.

I've learned through hard experience that DIskWarrior along with Carbon Copy Cloner are essential. It's no fun to contemplate how much you'd spend to get your data back, certainly a lot more than DW and CCC.

Slow network performance is the hallmark of a DNS error. That's the DNS connections timing out.

Assuming that network address translation (NAT) is in use, what you're trying to do is possible, if you use a subdomain of the public-facing dynamic DNS provider, but I prefer to avoid that as that provider has the domain registered and not me — I much prefer to have and to use a domain that I have registered internally, and then use the dynamic provider for connections into the host when those are necessary.

FWIW, I have a detailed write-up on DNS configurations available. There is an article or two on the issues with dynamic DNS linked there, as well.

if DNS is hosed, much of the rest of the stack will be somewhere past flaky — server DNS is key to authentication and certificate-based network security, among other details. Open Directory typically needs DNS, and certificate-based security. Among other services.

TL;DR, assuming a NAT'd network... a) no. b) use your DNS server only, and forwarding from your DNS server is an optimization, and do not reference off-LAN DNS servers except as forwarders and only from your DNS server and c) reference only your local DNS server(s)

For internal DNS within a NAT'd network, I would not reference any domain associated with dynamic DNS provider. Even for a home network. I would not use any domain names related to the dynamic DNS provider anywhere on my internal network, outside of references to services such as mail that might be hosted at that provider. Dynamic DNS is great for home users, but it does not work very well with servers, and I definitely would not try to use the dynamic DNS name or subdomains of that domain on my internal network. Keep dynamic DNS separate from your operations and your configuration.

FWIW, I've posted a very detailed description of how DNS works, and the various configurations and trade-offs available; that's the earlier link.

Forwarding servers are a performance optimization — they're a way you can allow an upstream server to cache DNS translations for you (because the upstream server may have already translated the host for another user of that DNS server), but your own server will also be caching translations for you after the first translation. Forwarding servers are useful when you might want to use Google DNS (8.8.8.8 and 8.8.4.4), which are rather likely to have cached the translations for you. In terms of correct operations, forwarding servers are not relevant — they "just" avoid the overhead of the recursive DNS translations starting at the DNS root servers.

Unless your gateway firewall box contains a DNS server, you're probably just adding a hop. Some mid- and various upper-end gateway firewall boxes do contain a DNS server, but many mid and lower-end boxes do not. If your gateway firewall does have a DNS server, you can potentially use that in place of the OS X DNS server, or can use it in parallel for reliability. Most low-to-mid-range gateway firewall boxes contain what is known as a DNS forwarder. Not a DNS server. DNS forwarders pass the translation request along to the ISP DNS server. DNS forwarders don't have local IP address translations, either.

As discussed in the article and with the exception of any optional specifications of forwarding servers within the context of your DNS server, you do not want to reference any DNS servers off your local network from any of your clients — just your local DNS server. No ISP DNS servers, no off-LAN DNS servers, nothing. Just your local DNS servers. This requires you to configure your DHCP server and any static-configured hosts to reference and to vend only your local DNS server(s) IP address(es). When your DNS server refers to itself via its Network preferences setting, it should refer only to 127.0.0.1 — how IP refers to the local host — and to any other LAN-private DNS servers you may be running. If you are using NAT, do not reference off-network DNS servers. Off-network DNS servers do not have translations for your local addresses, and many devices will not re-try DNS translations — first server they get that says "nope", and the translation attempts then end. Off-private-LAN translations also cannot provide private IP address translations.

Again, please have a look at that linked article — that's been written in response to more than a few DNS configuration mistakes and confusions that have arisen over the years...

Please forget about your Dynamic DNS provider. That's not your domain. Yes, you do have permission of the registrant to use a subdomain, but the domain is not yours. Get yourself a real and registered domain. Saves hassles down the road. Again, I would avoid using a subdomain of a domain offered via a dynamic DNS provider. That doesn't end well, when you change dynamic DNS providers or hosting.

Your public (dynamic) IP address is not referenced within your network, except at your gateway-firewall box.

Your dynamic DNS name is not something that's at all used or referenced within your network perimeter.

I'd delete the current internal configuration and would follow what I posted in that article.

Using the same domain name inside your network perimeter and outside your network perimeter is possible, but I'd avoid it. Use a different (registered) domain, or a subdomain of a (registered) domain. Don't try to run with the same domain within your network, and configured in the public DNS servers from your domain registrar.

The book you're reading — I would encourage following what I wrote, as I can help with that, and can clarify that — probably does not cover using a dynamic IP address nor a dynamic DNS provider. Dynamic providers are far more common with client configurations. Servers get static addresses, as otherwise some services will not work as expected, or won't work reliably — you'll be able to start them and they can look OK, but other hosts deliberately will not interoperate.

I'm going to have to re-read that text you've posted and will try to post a more detailed answer in a few hours, as this message got posted prematurely; there's some "post it" sequence in this forum software, and I hit it.

Dynamic DNS provider: that's the service you are using to get a translation of some domain name to your dynamic IP address. Dynamic DNS differs from static DNS, in that dynamic DNS gets updated automatically each time your dynamic IP address changes.

Dynamic IP and dynamic DNS are not common with servers, and can preclude using some functions of the server such as mail. Mail will work, mail will start, mail will appear to run, but other mail servers can and variously will detect your dynamic address, and can and variously will decide not to receive mail from your server, and some may decide not to send mail to your server. You can configure a mailhop service, but that's beyond the scope of getting your internal DNS going,

Internal DNS. Inside your gateway firewall router box. Private DNS. Probably hosted by your OS X Server box.

External DNS. Outside your gateway firewall router box. Public DNS. Hosted by various providers.

1a: please follow the article I have linked. Get your internal DNS configured and working. Then look at getting the dynamic DNS provider configured. Do not use the same domain name or the same subdomain name for internal DNS and for external DNS. I'd avoid using any domain from the dynamic DNS provider — while you might have permission to use that domain, you are not the formal registrant, which means you are a tenant of the dynamic provider. That's fine for your public DNS — do not assume that you will be hosting mail or web services or other such access on a dynamic IP address — but you do not want to use somebody else's registration for your internal network. (Yes, it's possible to do that, but the configuration is more complex and — as you're discovering — more confusing.)

2a: please follow the article I have linked. Get your internal DNS configured and working. Ignore anything with your dynamic provider for now. That comes later. As mentioned in 1a, keep clear of the dynamic provider's domain name.

B1: When you're done getting it configured, you have your own DNS server now. You'll be using that.

B2: Okay.

B3: your server should be at a static IP address on your internal network, and not a DHCP-vended dynamic address. (Static IP? Dynamic?)

B3 (B4?): ignore port mapping for now. That's for remote access and entry into your server and to offer the miscreants on the Internet a way to try to hack into your server. When you get around to getting external access going, do not open up inbound DNS queries into your network, and do not open up any ports other than those that are strictly necessary.

B5: the default host stuff is again for remote access, and should be ignored for now.

B6: using your router as a forwarder adds overhead. You're running a DNS server. Have that reference the other DNS servers directly.

B7: Correct. Your gateway firewall router should vend the IP address of your DNS server. It should not vend the address(es) of your ISP DNS server(s), nor the address of the gateway firewall router box as a DNS server.

B8: No. You are running your own DNS server. Do not reference off-LAN DNS servers. Do not reference your gateway firewall router box as your DNS server. You are running a DNS server. Only your DNS server can translate local addresses. Attempting to reference off-network DNS serves will not work for any local DNS domain names — you're setting up your own domain name here, with your internal private IP address space — as the DNS servers off your network are from the public IP address space, and not the NAT'd (private) IP address space you are using.

C: No. see B8. You are running your own DNS server. Your own DNS server is the only DNS server you should reference here.

D: No. See C & B8. See the rest. You are running your own DNS server here. Referencing other DNS servers is the path to problems and timeouts and confusion.

D1: Yes. You are running your own DNS server. Only your local DNS server has local knowledge of the hosts on your network. The two most common hosts on your network are your gateway firewall router box, and your server. Other boxes with static IP addresses can include network printers, or other servers that you might add.

Again, please re-read the article. Set up your internal DNS first.

For your purposes here, your dynamic DNS and your dynamic IP is going to limit what sorts of services are accessible remotely.

So again — set up local DNS first, then get your public (external) DNS configured.

With dynamic DNS, there's really not all that much configuration that's even available. And again, you're not going to have a fully-capable server available, as ISPs tend to block some server-oriented traffic on dynamic IP accounts, and as some network services — mail services chief among these — really expect static IP. Mail servers with dynamic IP addresses are detected by other mail servers, and are assumed to be spam engines; random infected client boxes. There can also be other policy blocks here. In short, do not expect much in the way of externally-accessible services. Some will work. Some will not.

1) That is common and reasonable advice for a client computer system. Client computers get their services from servers. You're running a server now, and you want and need your local clients to use your server(s) for the various local network services. Accordingly, the details of the local requisite network configuration will necessarily change.

2) I don't know about Apple's configuration assistant here. I enter the domain or subdomain entries and the translations into Server.app. The default OS X Server install for folks that don't already have a local DNS server — if that is what you are referring to here — is enough to get your server going with correct forward and reverse DNS — and it is configured with a one-host DNS zone entry. The next step toward establishing a DNS server is to add the target zone configuration, and then delete the installation-default zone entry from the configuration.

2a) I'm not certain where you're headed here, and will repeat some advice: do not use your external domain name and your external host name on your internal network. Use a subdomain of that domain, or use a different domain. Do not have the same domain name configured in your internal network, and in your external DNS services. Do not refer to the dynamic DNS provider here.

Your host will have your internal domain name, with your internal host, with your internal DNS, from your internal server. Preferably in your own registered domain. Again, a "registered" domain name is not what you get from a dynamic DNS provider. You "only" get permission to use a subdomain within a domain that the provider has registered.

To create an externally-accessible WWW host, create a CNAME (alias) entry in the public DNS, and create a WWW alias entry in the local web server.

Unless you want the local host to be named WWW, you're not creating a DNS entry for your host using WWW.

Web services use a different mechanism for accessing the host — they use the specified name and the DNS server associated with the web browser, and then pass the text name of the web server in the HTTP or HTTPS traffic. This means that the web server gets the name of the host the client asked for, and can then present that host. This host name does not need to be known in local DNS services, it only needs to be configured as a "virtual host" — what Apple tends to call a "site" — to be available to requesting browsers.

2a) the "search domain" is not necessary. It's a convenience that's the local default domain that gets tried when some client asked for "stephan" as the host name, and did not specify the fully-qualified "stephan.example.com." host name or the not-fully-qualified "stephan.example.com" host name. This is unrelated to that WWW host, nor with the DNS resolution, nor with the web server virtual host configuration.

2b) Are you going to be doing eCommerce or such, or sensitive content? If so, you're going to have a whole project for security and data integrity here such as the incorporation of a DMZ and a proper firewall, probably also a move off of a dynamic DNS provider, and — in aggregate — that's rather more of a discussion than I'm prepared to add to this thread. If not, don't bother serving HTTPS port 443 here, which means you don't need a certificate, which means you also don't need to know about multi-domain certificates (also known as unified communications certificates) or other details of certificate-based security. I would not want to be doing this stuff from a dynamic IP address.

2c) that's the multi-domain or UCC cert mentioned in 2b.

The example.net domain is one of the three domain names that are reserved for use in discussions and code examples. These three domains — example.net, example.org and example.com — do not actually exist, should not be configured in your local DNS, and should never be seen in real networks.

There are probably several thousand new top-level domains coming online, so picking a short domain name is getting easier, but picking your own bogus domain name is getting harder. (Years ago, Microsoft had encouraged some of their users to set up domains in the then-unregistered .local top-level domain, and that later became a real and registered top-level domain. Hence... problems with Bonjour and its use of .local top-level domain. But I digress.)

My use of example.net for an internal network and example.com hosts for the external network is to avoid having to deal with two pools of DNS servers both authoritative for the same domain name, and also to avoid having to enter a usually much longer subdomain name as often as arises when that approach is used; a domain registration, versus having to type a longer name all the time. (In your case here, also so that you have ownership over the domain within your network, and cannot end up colliding with some real domain.) The different domain name also means I can very easily then tell if the service is externally accessible — www.example.com would be, in the example configuration — and www.example.net — or whatever you've chosen as example.net, your internal domain name — would not be. Hosts configured with the example.com domain would have public static IP addresses, and hosts using example.net would have internal (usually static) IP addresses from the private (NAT'd) address space. A subdomain of a real and registered domain does also work here, it's just that it appears you do not yet have a registered domain if you're using a subdomain of your dynamic DNS provider.

In general... This assumes your firewall is not an incompetent device, and that your firewall can correctly "reflect" the traffic sent to your public IP address back into your network. Mid-grade and higher-end firewalls can do this, but some of the cheap gear cannot.

In general... If you're going to be serving data publicly, you'll really want to look at a firewall with DMZ capabilities, as a breach to your web server can allow the attackers to access everything, and delete everything. If you have internal data on the same server that's offering public web sites and particularly via some content management system, all of that data is potentially (or actually) at risk in a server breach.

In general... I'd encourage getting some more formal help here, as I'm inferring — no offense is intended — that you're really not yet comfortable with DNS and web services and certificates and security and the rest. Mistakes here can unfortunately take a while to clean up, too. The web servers I deal with get probed nearly continuously, and they each get scanned for more than a few known-vulnerabilities multiple times a day. Any internet-connected server will suffer equivalent activities. This goes well beyond DNS, of course. This also means keeping backups, and preferably backups that cannot be accessed by the server by default, so that — if there is a successful attack — the attacker cannot access and delete the backups.

There are also various different ways to configure DNS, and the "best" approach depends on local requirements and local expectations and — where there are options — local choices.

There are a number of these sorts of "details" on the path to running your own server, too. DNS is just the tip of what will be a big — and quite possibly fun — learning experience. (There are, however, good reasons why folks host their web servers and mail servers.)