tim_r_66

Q: Addressbook service unreliable/sources fail/database corrupts

Hello Community,

 

I am at wits end on this one, and I'm losing data in the process.  I could definitely use some help solving this issue before I take more drastic measures.  The basic issue is that my local copy of my server-hosted addressbook repeatedly gets corrupted and then things go downhill from there.

 

I'm running now with Mavericks server/Mavericks client.  Server hosts calendars, contacts, wikis in addition to some core services for the network.  Both server and client have been recently rebuilt from scratch.  The user account in question has been wiped and rebuilt with the exception of the data in the collab and caldav databases.  I use Profile Manager but really had these issues before the last rebuilds and use of PM.

 

With yesterday mornings work to finish migrating the user to a new home directory and export/import in OD, I had finally hoped I had stabalized the addressbook service.  This morning when I started working I had some trouble accessing the addressbook service and general disarray with Internet Accounts Remote Service (com.apple.preferences.internetaccounts.remoteservice).  This was a pretty good indication I may not have solved my problems.  I ended up rebooting the server and then was able to connect again to the services.  So I set to work.  One of the things I did was to create an addressbook group in Contacts and I added about  13 contacts.  When I went to search for another name, everything suddendly froze and Console started reporting that the AddressBook-v22.abcddb file in the sources directory is corrupt.    I also got a slew of comp.apple.quicklook.satellite errors about obtaining permanment IDs "The model configuration used to open the store is incompatible with the one that was used to create the store" and CoreData: error (3) access permission denied.

 

I also note that if I intentially delete an addressbook connect from Contacts, the sources directory doesn't clean up.  There are now five separate folders under Addressbook/Sources with different UIDs.  I've also started having files with .uknown in them being created in the Addressbook or Sources folders.

 

I have searched and read multiple discussions about Addressbook/Contacts issues but haven't come across anything like this yet.  And if any of the solutions I've tried improve things at all, it is short-lived and I am right back to where I am now: A non-functioning addressbook for this user and lost data.

 

My next step is delete the source connections again in Internet Accounts, and try one again to restablish connections.  I'll leave the files in the Addressbook directory alone for now.

 

If this doesn't work (and I doubt it will), I'm considering asking the users (luckily there are not a lot of them) to export/archive all their contacts and calendars, and then completely nuke the postgres database, and then let the users reconnect and load their data. 

 

Is there a better way to proceed?  Other things I should be looking into?

 

Thanks.

 

Tim

Mac mini, OS X Server

Posted on Jan 2, 2014 10:33 AM

Close

Q: Addressbook service unreliable/sources fail/database corrupts

  • All replies
  • Helpful answers

Previous Page 2
  • by tim_r_66,

    tim_r_66 tim_r_66 Jan 6, 2014 3:53 PM in response to Linc Davis
    Level 1 (50 points)
    Jan 6, 2014 3:53 PM in response to Linc Davis

    Thanks, Linc.  I have had the data on an external drive for a couple of years now starting with SLS.  I skipped Lion but ran with ML for about a year.  Admittedly though, I'm pushing my personal use of the server much more now and working through these issues.  Other user accounts are generally not experiencing the issues I am though but their use is less taxing than mine.  Altough, they are using the contacts services and are not experiencing the issues I am haivng with this account.  I supposed I could move the user data for this account to /Users and repoint to that as the home directory and see if things improve?  Something to consider.  FWIW, this current OD was built with all the user data in the current location and user accounts' home folder settings pointing to that external drive location. 

     

    I read "com.apple.quicklook.satellite[804]: Move Address Book store at..." as a renaming of the current local addressbook file and therefore a modifcation of the file.  Agreed this occurred because it tried to read something it didn't--what, I don't understand, but in my mind I keep going back to the log entries about the store model being used to open not being compatible with that model used to create the store.  Whether this is ultimately being caused by a home directory/OD issue, bad data in database, or something else, is not clear.  I do know that I've now been running for near 24 hours without a single log entry from quicklook.satellite and I've used Contacts during that period fairly extensively.  Too early to say I've Isolated it for sure, but I do believe I can say it is a factor. 

     

    Thank you for the tip on /Library/Server/Calendar and Contacts/  I had thought about that approach.  It will probably be a few days before I can get around to trying that.

     

    Thanks again,

     

    Tim

  • by tim_r_66,

    tim_r_66 tim_r_66 Jan 6, 2014 4:21 PM in response to tim_r_66
    Level 1 (50 points)
    Jan 6, 2014 4:21 PM in response to tim_r_66

    Meant to say, "I had *NOT* thought about that approach".

  • by tim_r_66,

    tim_r_66 tim_r_66 Jan 7, 2014 11:14 AM in response to Linc Davis
    Level 1 (50 points)
    Jan 7, 2014 11:14 AM in response to Linc Davis

    Ended up spending this morning doing extensive testing of different account and home folder configurations after re-enabling contact quicklook and vCard importer.  The hypothesis in general was that the issues with quicklook are symptoms of the home directories being on an external drive.  I have four observations:

     

    1) With a newly created local account on a client machine wth no obvious interaction with the server, vCards can be viewed by quicklook without obvious errors.  They are nice and pretty and I suspect this is how it is supposed to be.

     

    2) With a newly created account where the AFP-provided home directory is pointing towards an external hard drive, quicklook of vCards fail.

     

    3) With a newly created account where the AFP-provided home directory is pointing towards the default location of /Users on the server's internal hard drive, quicklook of vCards fail.

     

    4) With a newly created account where the network account's home folder is set to Local Only, a local home folder is created on the client and quicklook/contact import work.

     

    Isn't #3, i.e., AFP (or SMB)-provided home directory pointed to the default location of /Users, the desired configuration? 

     

    If I have to nuke OD (and device management) again, I will.  But I don't want to overlook something obvious first.  Am I missing something?

     

    Cheers, Tim

  • by Linc Davis,Helpful

    Linc Davis Linc Davis Jan 7, 2014 4:04 PM in response to tim_r_66
    Level 10 (207,963 points)
    Applications
    Jan 7, 2014 4:04 PM in response to tim_r_66

    You're not overlooking anything that is obvious to me. What I would do in your place is to export the users with Workgroup Manager, delete them, and redo the whole OD setup. There are some tutorials on krypted.com that you might find helpful. Good luck.

  • by tim_r_66,

    tim_r_66 tim_r_66 Jan 7, 2014 4:20 PM in response to Linc Davis
    Level 1 (50 points)
    Jan 7, 2014 4:20 PM in response to Linc Davis

    Thanks.  I've started down that path.  Although one of my tests after my post above has me confused.  With a test account in config #3, I tried logging into the server itself and it handled the quicklook of the vCard without error.

     

    I initially thought this might be an indication that device management is a culprit in this.  So, I removed a client from device management, manually configured the binding to OD, and logged in.   The quicklook still fails. That seems to rule out device management as the culprit.

     

    But it leaves me confused as to why logging in via OD bound from a client is dramatically different than logging in using OD on the server?  I realize if this redo of OD works, my question will be academic.

     

    I'm moving user data over to the hard drive and will disconnect the external drive before reconfiguring.  I'll have to enforse stricter quotas and learn the proper way to scale user account storage.

     

    I'm also not going to destroy the profile management data yet.  I'll turn profile manager off for now and see if I can get through OD first.  If you think that is a mistake, please shout.

     

    Tim

  • by tim_r_66,

    tim_r_66 tim_r_66 Jan 8, 2014 1:15 PM in response to tim_r_66
    Level 1 (50 points)
    Jan 8, 2014 1:15 PM in response to tim_r_66

    Many many hours later and I know more, I don't feel any closer to a resolution.  I have steadily walked this system back towards a point where the only thing left to try is wiping and starting over:

     

    1) Wiped and recreated OD multiple times with no change of behavior

    2) Moved all users' home folders to the internal drive

    3) Removed wildcard cert and replaced with self-signed.  Also tried just the OD cert. (Binding doesn't work over SSL now)

    4) Wiped device management database

    5) Put home folder that would be the likeliest to have offending data in the Spotlight privacy bin

    6) Made sure the mail server was unbound

    7) Tried with and without using SMB as one of the sharing protocols

    8) Renewed push certs

    9) All services are off except DNS, DHCP, OD and File Sharing

     

    The behavior is the same in that if I have a home folder shared the errors above occur, quicklook screams loudlly and, I believe as a symptom, vcards can not be displayed by quicklook.  If the home folder is local, it works as expected.

     

    There are some entries in AFP Error Log:

     

    AppleFileServer <Info>: Kerberos fail: gss_acquire_cred major status value <458752> minor status_value <0>
    AppleFileServer <Info>:    major error <1>: No credentials were supplied, or the credentials were unavailable or inaccessible.
    AppleFileServer <Info>:    minor error <1>: unknown mech-code 0 for mech unknown
    

     

    There are som repeating errors in LDAP log about attribute "entryCSN" index delete failure.  I'm not sure what is relevent anymore.

     

    When I finish moving user data off, I'll try my best to clean things up and try one more time.  Part of what seems wrong in all this is that the system doesn't clean itself up well.  For example, AFS does not disconnect after a user logs out.  And uid's are immediately reused which strikes me as an opportunity for errors.

     

    After this next attempt, if no change, I'll probably capture the DNS settings and then wipe and reinstall.  I'll leave things as default as I can (including destroying the mirrored RAID set I guess).  And if that doesn't work?  Maybe I'll hand the users desk calendars and notepads :-~

     

    Tim

  • by tim_r_66,

    tim_r_66 tim_r_66 Jan 8, 2014 6:27 PM in response to tim_r_66
    Level 1 (50 points)
    Jan 8, 2014 6:27 PM in response to tim_r_66

    Complete failure. 

     

    Wiped clean server.

    Destroyed mirrored RAID set and reformatted both drives. 

    Installed OS X Mavericks on lower drive (using USB)

    Downloaded and installed updates

    Downloaded and installed Server.app

    Started Server.app and reconfigured host/computer name and set static ip (this time the script did not trigger DNS Config)

    rebooted

    Turned on DNS following krypted.com page as guide

    sudo changeip -checkhostname succeeds

    rebooted

    DNS still checks

    Created ODM following krypted.com as guide

    Set global password policy

    Imported users

    Added /Users sharepoint, home directories of AFP

    Turned on file sharing

    Joined client to server; no apparent issues

    Rebooted client

    Logged in and set pasword

    Downloaded vCard; quicklook started screaming, Contacts started and then froze, just like before.  Contacts has to be forced quit.

     

    No existing user data was brought in. 

     

    I will try from a ML client in a few minutes but unless you hear me shout EUREKA you can assume that fails too. 

     

    If anyone sees any glaring holes with the summary of the install, please let me know.

     

    In the short term I will give the users services back and tell them to avoid using vcards or Contacts.app on OS X clients.  We have not experienced issues when using contacts and calendars from iPads and iPhones.

     

    I may also slick the Mac Mini client I'm using, or perhaps delete the /var/private/folders directory as a trial first just to make sure all previous local cached user data is deleted.

     

    Thanks,

    Tim

  • by tim_r_66,

    tim_r_66 tim_r_66 Jan 16, 2014 10:34 AM in response to tim_r_66
    Level 1 (50 points)
    Jan 16, 2014 10:34 AM in response to tim_r_66

    Last Thursday I rebuilt the Mavericks client and used the Join button to bind it to OD, all to no avail.  I took a break over the weekend and on Monday morning I renewed my efforts by looking more closely at Open Directory.  In particular, I used dscl . -list /Computers to see which computers were present.  Only the ODM and ODR were listed, even though multiple machines were manually bound without obvious errors.  Because I had not tried to enroll the Mavericks client after rebuilding it the previous Thursday (which was after I had rebuilt the server on 8 Jan), I turned Profile Manager back on and gave that a whirl.  It worked!  Dscl now lists it and the errors regarding CoreData are gone.  I’ve been able to log in with my user account for three days now and have it work properly.   That was true until I tried to let quicklook process a vCard.  Numerous errors spit out in Console about incompatible store and Finder crashed.  Then error shows about the Addressbook/Sources/xxxxxx being empty.  I had to reboot and let the addressbook server rebuild the local store when Iogged back in.  FWIW, logging off and logging back in did not work; a restart seemed to be required.

     

    So the best reconstruction I think I can do includes a handful of opinions, observations and hypotheses:

     

    1. I have likely been dealing with at least two issues, one of which has been isolated and probably fixed.
    2. Profile Manger/device management is an all or nothing game.  In a way, this makes sense but it would also make sense that if Profile Manager is turned off, a system administrator should be able to manually bind clients and have them work (or give more meaningful errors.  Likewise, if the binding process is looking for something associated with profile management, errors/notices should be provided.
    3. I have no evidence supporting the hypothesis that the locations of /Users and /Groups on an external drive contributed to the problems. That said, I am now using /Users and /Groups and will start another discussion threat about the best way to manage this when more drive space is needed.
    4. I have no evidence supporting the hypothesis that third-party wildcard certs contributed to the problems.  I will be adding this cert back today or tomorrow, but will limit its use so as to not disrupt OD/Profile Management. 
    5. I have no evidence that DNS or OD by itself caused the problems; rather the errors seem to be a result of the interaction between device management and OD.   This strikes me as fragile to say the least.
    6. I believe that files related to device management remained on the client even after profiles associated with its enrollment were deleted and the server was wiped clean.  I suspect these left over configurations prevented successful enrollment after the server was wiped clean.
    7. Using dscl to list computers is a useful check to verify binding/enrollment.  One question I have about this:  if device management has not been configured and a client is manually bound to the server, should this client show up when dscl . -list /Computers is issued?

    In addition to vcf files, I still have an issue with Notes (I will start a new thread on that if necessary), but at least I can use Contacts, Calendar, Mail and Messages as designed.

     

    I continue to welcome ideas and discussion.

     

    Thanks.

     

    Tim

  • by Simon Slavin,

    Simon Slavin Simon Slavin Jan 17, 2014 2:56 AM in response to tim_r_66
    Level 4 (1,400 points)
    Jan 17, 2014 2:56 AM in response to tim_r_66

    When we had problems of the type you describe: unpredictable dropouts, missing records, confusing contradictions with diagnosis procedures, the cause turned out to be a faulty hard disk on the server.  Replaced the boot disk (which hosted the Open Directory database and most other things) and everything started working properly.

  • by tim_r_66,

    tim_r_66 tim_r_66 Jan 17, 2014 7:46 AM in response to Simon Slavin
    Level 1 (50 points)
    Jan 17, 2014 7:46 AM in response to Simon Slavin

    Thanks Simon.  I have considered hardware issues but have, at least for now, discounted the hard drive for a couple of reasons.  First, I just replace the one I'm now using a couple of months ago and it passes both Disk Utility and SMART Utility testing, and some of the issues I'm having predate the drive replacement as best I recall.  Secondly, the issues seem to be pretty predictable.  My experience with hardware issues is they tend to induce more random errors (which definitely leads to confusion) when they don't fail outright.  Have your experiences differed from that?

     

    As an example, if I let the system quicklook a vcf file, it wreaks havoc every time.  That last time I tested that it also triggered something that causes Safari to lose all the saved passwords.  Do you have this problem with vcf files on networked home folders?

     

    I believe Linc is right when he says this is related to OD.  A troubling aspect of this is that I've experienced these issues with different builds of the server (ML and Mavericks both).  And to the best of my knowledge the start of the issues *may* have coincided with the very first time I tried using Profile Manager.  At the time I was also using Workgroup Manager but pretty much read only.  Since then, I have wiped and recreated both the server and the client I'm using as my primary multiple times.  Because of this, I'm focusing most of my troubleshooting attention on the Profile Manger/OD/binding interactions, while keeping in mind my procedures may very well be the cause of these issues.  

     

    I'm also noticing now that while I've improved things with the addressbook (service hosted from the main server), if I reboot the server for any reason, I also have to reboot the Mavericks client.  If I do not, I see more errors in Console regarding finding Internet Passwords and errors with Messages databases.   Also, the messages do not load properly in Messages. The more I think about this, with the exception of stored passwords in Safari, these problems seem to now be focused on Mail and Messages.  My mail server is in a VM on a second physical server that I use as an ODR and backup DNS/DHCP.  Not surprisingly, Profile Manager doesn't handle the VM cleanly but the machine does seem to bind properly with OD.  When I run dscl . -list /Computers on the main server, it lists my Mavericks client but does not list the mail server.  Dscl against the LDAPv3 node lists both the mail server and the Mavericks client.  I also have a Lion client and a ML client, neither of which show in the dscl list; these computers are bound to the OD but not enrolled in Profile Manager yet. 

     

    Kind of a long way to get around to why I'm focusing on the Profile Manger/OD relationship.  Sorry to make this so long.  I've been trying to read up on Open Directory and Profile Manager, but I still haven't seen clear descriptions of what I should expect when I read through the directory.  Do you know what computers should show up in each node (LDAPv3 vs Local) when they are bound to the server or enrolled in Profile Manger?

     

    Thanks.

     

    Tim

  • by tim_r_66,Solvedanswer

    tim_r_66 tim_r_66 Jan 22, 2014 9:36 AM in response to tim_r_66
    Level 1 (50 points)
    Jan 22, 2014 9:36 AM in response to tim_r_66

    After a few more days of testing, I'm going to close this thread and start one or two new ones.  In general, the last complete rebuild has gotten me to the point where I can operate Contacts/addressbook server and other services from the client.  I still have issues when a second network user logs into the client and I try to return to my normal user.  Client apps won't connect to the services and passwords in Safari get wiped.  I have to reboot the client and then I can connect to the services but the passwords in Safari are gone. 

     

    More in another thread later.

     

    Cheers, Tim

  • by tim_r_66,

    tim_r_66 tim_r_66 Aug 8, 2014 10:09 PM in response to tim_r_66
    Level 1 (50 points)
    Aug 8, 2014 10:09 PM in response to tim_r_66

    I'm adding this post to my old thread here in the hopes that it will help someone else.

     

    I recently recreated a pure test network to try to tackle some of these issues. On a very clean and default set up, the workstation went crazy when I tried to view a vCard using Quick Look by a user with a home drive shared over the network.

     

    This time, taking just the one problem, I called Apple Care and the Server Team rep was able to reliably reproduce the problem. He is elevating the issue to the engineers.

     

    The workaround is one solution to prevent this problem. If Apple gets me a better one, I'll post it here.

  • by Picoscope,

    Picoscope Picoscope Jun 10, 2015 3:58 PM in response to tim_r_66
    Level 1 (14 points)
    Servers Enterprise
    Jun 10, 2015 3:58 PM in response to tim_r_66

    Tim- I'm experiencing similar issues here. Did you ever find a solution from Apple?

     

    I am about to upgrade my server OS, and due to this and other past issues related to our OD setup I'm likely going to blow away OD and start from scratch, which I expect will absolve my system from all manner of evils, but likely invite in new demons.  Anyway, hoping there might be another fix for this particular issue that I can employ *before* I upgrade.

     

    Thanks much.

     

    P

  • by tim_r_66,

    tim_r_66 tim_r_66 Jun 13, 2015 5:28 AM in response to Picoscope
    Level 1 (50 points)
    Jun 13, 2015 5:28 AM in response to Picoscope

    Hi Picoscope--

     

    Is your set up the same as I described using network accounts on Mavericks machines? 

     

    I never got a solution from Apple, and eventually gave up on the network account for that particular user, and the other main user accounts on this network are either PHDs or standalone accounts. I have since installed a larger network using network accounts, but this set up is not exactly the same and the users are heavily using contacts or vCards.

     

    I'll soon be building a new network and I now have a more extensive test lab.  I'll be testing this issue as part of the development.  If I come up with anything good, I'll post it here.

     

    Tim

  • by Fred de Gembloux,

    Fred de Gembloux Fred de Gembloux Jan 18, 2016 12:31 PM in response to Linc Davis
    Level 1 (10 points)
    Jan 18, 2016 12:31 PM in response to Linc Davis

    Hi,

     

    I have a carddav account that is corrupted.

    I ran the command which returned the number :       59

     

    How could I repair or delete the account?

     

    Thanks for your support.

Previous Page 2