Open Directory - Local Network User/Group - GONE

Question

Level 2

306 points

Open Directory - Local Network User/Group - GONE

This morning everything seemed to be working normally, then an email couldn't be sent through my local server. Kept asking me to sign in. Eventually tracked it down to Open Directory playing up. In Server.app, the 'Local Network' option for both Users and Groups was empty. After a restart of the server (and half a dozen more since) there is no longer any pop-up in Users or Groups at all. There's just the list of locals and that's all.

Trying to look into this with either WorkGroup Manager or Directory utility just results in an error, so looks like OD has comprehensively shot itself in the foot, all by itself. Nothing has been done to the server for weeks, this is entirely of its own doing.

Admittedly there's not many Users and/or Groups to re-create, but at the moment I can't even do that as it doesn't even know there's an LDAP directory to add them to. So looks like I'll have to destroy the entire OD setup and start again from scratch. Just what I wanted to do this weekend. Thanks Apple.

Anyone got any info on how Server.app manages to do this and what can be done to fix it and hopefully stop it from occurring again?

Posted on Dec 14, 2013 6:04 AM

Reply

Answer 1

Best reply

Strontium90

Level 5

4,833 points

Dec 14, 2013 8:50 AM in response to UKenGB

While I don't have a magic bullet to solve your problem, I can suggest reviewing your logs. OS X does a rather decent job of recording events and something like the corruption and loss of an Open Directory master is likely recorded somewhere. While discovering this moment and possible cause may not result in the ability to fix the issue, it will at least provide some closure to why it happened and when.

Next, it is best practice to backup your Open Directory regardless of how small or large it is. LDAP can be a finicky technology and many things can cause it to flake out. If you were backing up your OD on a regular basis, you would likely be able to simply restore from a backup and everything would be back in place.

The importance of a backup can not be dismissed. Accounts in OS X are backed by GUID values and these can be very difficult to nearly impossible to rebuild in the event of a rebuild of the server. Many of the services in OS X will define access based on the accounts GUID value. If your OD blows up and you simply recreate new accounts, you end up with all new GUID values for the accounts. This can make linking users to data a challenging ordeal.

Now, LDAP does have some tools to attempt to repair an Open Directory database. This has historically not worked well in my experiences. However, you can research the db_recover tool. Generally sudo db_recover -v -h /var/db/openldap/openldap-data/ Research before attempting. Once you are back up and running, make sure you have a backup plan in place.

R-

Apple Consultants Network

Apple Professional Services

Author "Mavericks Server – Foundation Services" :: Exclusively available in Apple iBooks Store

Reply

Answer 2

UKenGB Author

Level 2

306 points

Dec 14, 2013 9:03 AM in response to Strontium90

I'm embarrassed to admit I haven't been backing it up properly. All the client Macs happily copy away to the Time Capsule and that's been a lifesaver in the past, but the server has always been for more transitory data and I was wanting to sort something else out first before getting it too dribbling everything up to the TC. Needless to say, I just never got around to it. Foolish, particularly as an experienced user who extols the virtues of backing up to others. Oh well.

I have just done an OD backup but no idea if the users are included. Where in the backup would be the User/Group data? Do you know?

Actually, I don't think it'll be a mammoth task to re-create everything even taking into account the change of GUID, but it will be VERY annoying. More at Apple for such a flaky product and a little less at me for not having it backed up sufficiently.

I did look at the logs, but non of the OD logs showed anything untoward. Only a small error log:-

Oct 26 2013 13:19:20 905947us Registration is finished error: (10, -72000).

Dec 14 2013 11:15:11 11203us Requested SASL mechanism not loaded: SMB-LAN-MANAGER

I'll have a look at those tools you mention.

Reply

Answer 3

UKenGB Author

Level 2

306 points

Dec 14, 2013 10:36 AM in response to Strontium90

Checked the logs again, but nothing obvious. However, the SMTP log indicates that all was well at 09:59:56 this morning, but that at 10:08:45, user auth failed. That was me trying to send an email - I'm a mobile user so not local. Auth failed because OD appears to have simply thrown away it's entire LDAP directory. Took it 8 minutes.

Reply

Answer 4

UKenGB Author

Level 2

306 points

Dec 14, 2013 11:57 AM in response to Strontium90

Well I turned OFF OD, ran db_recover, turned OD back on and to my surprise, the LDAP Directory is there again with the 'LocalNetwork' Users and Groups all showing again. You could have knocked me down with a feather. All network services seem to be fully restored.

As you can imagine, I now have >1 OD backups and will get Time Machine sorted out too.

Thanks for pointing me to db_recover.

Reply

Answer 5

Dec 14, 2013 12:18 PM in response to UKenGB

Glad to help. Count yourself a lucky one. Go get a lottery ticket. I think db_recover has saved me twice in the last 7 years. Glad it helped out!

And yes! Do run a backup. Since I am on a winning streak with advice, I will also point your toward slapconfig as a method of automating a backup outside of TM.

sudo slapconfig -backupdb ~/path/to/backup

This can be automated with an expect script.

Glad to save your weekend.

R-

Apple Consultants Network

Apple Professional Services

Author "Mavericks Server – Foundation Services" :: Exclusively available in Apple iBooks Store

Reply

Answer 6

UKenGB Author

Level 2

306 points

Dec 15, 2013 1:10 AM in response to Strontium90

Am I right in thinking that 'slapconfig -backupdb ...' is what Server.app actually runs when you archive the OD configuration? So you could use Server.app and the above command interchangeably, i.e. use one to backup and the other to restore from the same archive?

The backup DOES include the (OD/Local Network) Users and Groups doesn't it?

Using an expect script would require the inclusion of the admin password which wouldn't be a good thing from a security point of view would it. In any case, how often does one need to backup the OD config? Surely it doesn't change that much. For me, Users and Groups don't change, nor does anything else about OD that I can think of. Why would regular backups be required?

Not trying to be argumentative, just confirm that I'm not missing anything.

Reply

Answer 7

Dec 15, 2013 4:11 AM in response to UKenGB

No argument perceived. I tend to run a backup on a daily basis regardless of the environment simply because I never know when a user may change a password. Having a regular backup means that I will only miss a potential 23 hours and 59 minutes of changes.

And yes, slapconfig is the same as using the User Interface. I approach this from the field consultant perspective. I am not present at any of my deployments on a daily basis. That being said, I rely on automation to do the backups for me. This is why I will use an expect script to automate the creation of the backup dmg. And no, you do not need to embed the admin password in the script. You only need to embed the disk image's encryption password.

OD is one of those technologies that we feel we don't change often. But when I look at an environment, there are those "hey can you add me to a group" request or the "he we have a new user" or, as mentioned, the use who changes a password. The backup of OD contains all users, groups, and passwords. This allows for rapid restoration and reduces the amount of reconstruction required.

If you are the admin of the environment, you define an acceptable risk policy for backup interval. If you have a small set of users (under 20), with no password policy, a quarterly manual backup may be appropriate. If you are managing over 50 people with a password policy in place and high staff turnover, you will be surprised how much OD changes.

By the way, using TM on the server should accomplish the same task. I am just a paranoid sort and have seen TM fail me when needed most. Thus, I craft multiple backup routes for the important stuff. Trust but verify.

Reply

Answer 8

UKenGB Author

Level 2

306 points

Dec 15, 2013 5:04 AM in response to Strontium90

Your network environment is on a different scale to mine. This is just a home server and NO changes take place without my knowing. So I'm pretty sure that OD configuration is not changing year by year. But I'm not advocating any lack of backups:-)

I have found Time Machine to be THE best backup system I've ever used. In the past, even the expensive unix systems failed me when most needed. TM is the first I've ever felt comfortable relying on. I'll set it running on the server anyway and also run the occasional OD backup.

How can the expect script run 'sudo .....' without requiring the password? Not a problem anyway for me, but curious as to how you would avoid that. Unless you actually log in as root.

Reply

Answer 9

Dec 15, 2013 8:10 AM in response to UKenGB

Automate the script using cron or launchd. That way you define the user who runs the script. Cron example is:

30 1 * * * root /path/to/script

Launchd requires a properly formatted plist file but you can achieve the same.

Never log in as root 🙂

Reply

Answer 10

UKenGB Author

Level 2

306 points

Dec 15, 2013 8:17 AM in response to Strontium90

Ah yes. In which case no need to use sudo in the command.

But first stop - Time Machine.

Thanks for your assistance. Happy Christmas.

Reply

Answer 11

UKenGB Author

Level 2

306 points

Dec 15, 2013 11:45 AM in response to UKenGB

Coincidentally, a friend has just suffered the exact same problem with his Mac Mini Server, running the same OSX (Mavericks) and Server.app. No connection between them whatsoever and like mine, nothing had been touched on the Server. One minute it's fine and then, poof! All the 'Local Network' Users and Groups gone. OD having stuffed up its database all on its own.

Also like mine db_recover worked like a charm.

Is this something we're going to see a lot of with Mavericks Server. Whatever, this is NOT enterprise class software is it.

Reply

Answer 12

Dec 16, 2013 4:19 AM in response to UKenGB

I hope not. To be honest, your two reports are the first ones I've heard of under Mavericks. Mountain Lion was known to do this periodically as well. Right now, Mavericks Server (in my world) is still mostly in lab testing. Only customers buying new hardware are being placed on Mavericks at this point. All others are waiting on the first patch release or later.

So far, in my testing, I've been pleased with the 10.9.0 release. Usually, it takes until a .3 or so before a new OS is ready for release. So far, aside from some issues with Calendar server and SMB connections to Windows workstations, I have had great success and stability. But, there is always the specter of disaster.

Reply

Answer 13

Dec 18, 2013 2:03 PM in response to UKenGB

What is the command for db_recover from the terminal window?

Reply

Answer 14

Dec 18, 2013 2:49 PM in response to curriebwoi

It is in the solution post. Generally it is:

sudo db_recover -v -h /var/db/openldap/openldap-data/

Read the man page first and make sure you are understand what you are attempting.

Then craft a backup plan.

Reply

Answer 15

Dec 30, 2013 6:13 AM in response to Strontium90

Dear Strontium,

I have exactly the same problem. Suddenly everything went out. I tried db_recover but it didn't solve the problem. Quite desesperate now...I have Many time machine saves. I made a restore with a time machine partition saved the day before. But I Still have this **** "impossible to connect to ldav server", even after a full restore.

Any ideas ?

Thank you!

Reply