jonttaylor

Q: Time machine missing files

Timeline

 

1. 11 days ago I buy a new 15" retina haswell macbook pro running mavericks, and a new 2TB time capsule.

2. Yesterday I returned the macbook pro because it was making creaking noises and had it replaced with a new one.

3. Now I try and restore my backup.

 

Story

 

I restored from my latest time machine backup by pressing CMD-R while booting and specifying to restore from the Time Capsule. It took a couple of hours over wifi and finally restored, when I came back to it, the machine had rebooted and I was at my normal login screen.

 

So I thought the restore had been successful at this point and I log in, however it asks me for my Apple ID login as if i've never entered it before (I have), and I have some new mavericks icons at the end of my dock that I had previously removed. I open up my terminal (i'm a web developer) and I notice its missing my custom prompt (oh-my-zsh) which I track down to the .zshrc file missing.

 

I check my databases and they are all corrupted. The MySQL error file is constantly spamming itself saying the data is corrupt and that it might be missing its binlogs. A number of my apps don't remember that they are licensed. The machine has forgotten most of my preferences such as tap to click, however it has remembered things like my desktop background and my safari preferences (they were not synced from icloud I made sure).

 

I dig into this a bit more by mounting the time capsule and having a look around in the terminal.

 

cd /Volumes/Time\ Machine\ Backups/Backups.backupdb

ls -lah 2013*/Macintosh\ HD/Users/jon

 

This prints out the contents of what the backup considers my home folder to be like at every backup (there were 23 backups stored over the 10 days). First backup being on the 2nd of November, most recent being on the 12th November.

 

I see that from the 7th my .zshrc went missing, it was not in any subsequent backups after that, so I tried restoring back to that date. After restoring my terminal preferences are back, but the databases are still corrupt and I notice other things missing instead.

 

I dig into it a bit more and list the files again.

 

Then I notice just how bad it is.

 

On certain backups it has missed massive numbers of files, including directories such as Downloads Library Desktop etc. Then they come back in the next backup, or go missing forever. I know time machine doesn't backup the entire computer each time, but I know that it should be creating HARD links in the filesystem, so that EVERY backup appears to be a complete system backup. This is not the case.

 

I can rebuild this new macbook pro to have everything I need on it again, but I CANNOT trust my £250 Time Capsule to take backups, in which case it becomes an extremely expensive router + hard drive.

 

I think this might be a bug in Mavericks, 10 days is not very long for backups to start going astray!

 

Is there any way I can repair this somehow? Has anyone else had this kind of problem? I phoned apple support and they just tried to guide me through doing a re-install, when I explained that the backups themselves seemed to be missing files the support agent just didn't seem to understand, not very helpful at all. As a fellow engineer I would prefer to speak to another engineer and I could provide all the diagnostics they could want, short of giving them ssh access.

MacBook Pro with Retina display, OS X Mavericks (10.9), Backing up to a Time Capsule

Posted on Nov 13, 2013 1:52 AM

Close

Q: Time machine missing files

  • All replies
  • Helpful answers

Previous Page 2 of 4 last Next
  • by IntoTheAir,

    IntoTheAir IntoTheAir Oct 30, 2014 11:46 AM in response to IntoTheAir
    Level 1 (0 points)
    Oct 30, 2014 11:46 AM in response to IntoTheAir

    I'll answer my own support Q here with the simplest reference from the manual.

     

    Notify after old backups are deleted

    Be notified if your backup disk is full and Time Machine deletes old backups to create space for new ones.

     

    I'm sure there's a heap of issues being discussed around this practice WITH/WITHOUT Notice! if the action occurs after the user is made aware of it... Seems in my case 'missing data' is likely blatantly deleted data. As my MBA image changes, older versions of the archive have been unceremoniously deleted to make room for new, and also far less valuable images. I've just read TM is not an archive tool. Making it's usefulness as a tool V. limited.

  • by jed104,

    jed104 jed104 Oct 30, 2014 12:52 PM in response to IntoTheAir
    Level 1 (0 points)
    Oct 30, 2014 12:52 PM in response to IntoTheAir

    Agreed but that's not the whole problem, I did a fresh backup to check and that instantly had files missing from it...

  • by Cave_Dweller_JC,

    Cave_Dweller_JC Cave_Dweller_JC Nov 3, 2014 3:32 PM in response to IntoTheAir
    Level 1 (0 points)
    Nov 3, 2014 3:32 PM in response to IntoTheAir

    November 3, 2014

     

    Same issue here.  My current T.M. backups go back to 8/2013.  I've always been confident that they are solid and if I ever had a problem, I also had a solution.

     

    But just week I tried to restore my last version of Mavericks before updating to Yosemite and after hours and hours of munching, I discovered that many apps and utilities were totally missing—I didn't bother to try to find out what else was missing. 

     

    I rooted around in the backup archives and found the first one that, on initial inspection, appeared complete.  It's dated Aug 3, 2014, exactly three months ago. The Mac was on 10.9.4 at that time.  No way to tell what happened after that, but the next archive is dated Aug 10, 2014 and it's obviously incomplete/

     

    I've retained the same name for my hard drive so the backups are contiguous.  One long backup with many upgrades to the Mac OS and software along the away. Yet even significant changes the same files/apps are missing in the archives—mainly about 20 apps from the Utilities folder and a few from the Applications folder.  No telling how many other files or parts are missing.

     

    I do hope that Apple gets a handle on this and soon. (I've left a bug report)

  • by Grayscaletx,

    Grayscaletx Grayscaletx Nov 6, 2014 6:11 PM in response to jonttaylor
    Level 1 (0 points)
    Nov 6, 2014 6:11 PM in response to jonttaylor

    I too have found a load of files missing after recovery from a Time Machine backup in Mavericks. This is from a standard SATA drive and not a Time Capsule. There seems to be no rhyme or reason. Just randomly missing. The folder structure is all there, which led me to believe that the recovery went well. But, as I'm using the machine for the last couple of days, I find files missing.

     

    I'm hoping this gets addressed by Apple at some point soon.

  • by Loner T,

    Loner T Loner T Nov 7, 2014 7:10 PM in response to Grayscaletx
    Level 7 (24,002 points)
    Safari
    Nov 7, 2014 7:10 PM in response to Grayscaletx

    The OP is from Nov 13, 2013 4:52 AM today is Nov 7, 2014.  (almost a year and posters are still reporting it).


    Caveat Emptor!


    See the highlighted part.

     

    From http://store.apple.com/Catalog/US/Images/MacOSXserver.html

     

    6. Disclaimer of Warranties. YOU EXPRESSLY ACKNOWLEDGE AND AGREE THAT USE OF THE APPLE SOFTWARE IS AT YOUR SOLE RISK AND THAT THE ENTIRE RISK AS TO SATISFACTORY QUALITY, PERFORMANCE, ACCURACY AND EFFORT IS WITH YOU. EXCEPT FOR THE LIMITED WARRANTY ON MEDIA SET FORTH ABOVE AND TO THE MAXIMUM EXTENT PERMITTED BY APPLICABLE LAW, THE APPLE SOFTWARE IS PROVIDED “AS IS”, WITH ALL FAULTS AND WITHOUT WARRANTY OF ANY KIND, AND APPLE AND APPLE’S LICENSORS (COLLECTIVELY REFERRED TO AS “APPLE” FOR THE PURPOSES OF SECTIONS 6 AND 7) HEREBY DISCLAIM ALL WARRANTIES AND CONDITIONS WITH RESPECT TO THE APPLE SOFTWARE, EITHER EXPRESS, IMPLIED OR STATUTORY, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES AND/OR CONDITIONS OF MERCHANTABILITY, OF SATISFACTORY QUALITY, OF FITNESS FOR A PARTICULAR PURPOSE, OF ACCURACY, OF QUIET ENJOYMENT, AND NON-INFRINGEMENT OF THIRD PARTY RIGHTS. APPLE DOES NOT WARRANT AGAINST INTERFERENCE WITH YOUR ENJOYMENT OF THE APPLE SOFTWARE, THAT THE FUNCTIONS CONTAINED IN THE APPLE SOFTWARE WILL MEET YOUR REQUIREMENTS, THAT THE OPERATION OF THE APPLE SOFTWARE WILL BE UNINTERRUPTED OR ERROR-FREE, OR THAT DEFECTS IN THE APPLE SOFTWARE WILL BE CORRECTED. NO ORAL OR WRITTEN INFORMATION OR ADVICE GIVEN BY APPLE OR AN APPLE AUTHORIZED REPRESENTATIVE SHALL CREATE A WARRANTY. SHOULD THE APPLE SOFTWARE PROVE DEFECTIVE, YOU ASSUME THE ENTIRE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION. SOME JURISDICTIONS DO NOT ALLOW THE EXCLUSION OF IMPLIED WARRANTIES OR LIMITATIONS ON APPLICABLE STATUTORY RIGHTS OF A CONSUMER, SO THE ABOVE EXCLUSION AND LIMITATIONS MAY NOT APPLY TO YOU.

  • by elfprince13,

    elfprince13 elfprince13 Dec 17, 2014 7:55 PM in response to jonttaylor
    Level 1 (0 points)
    Dec 17, 2014 7:55 PM in response to jonttaylor

    Just wanted to chip in that I've had the same problem, and I've filed a report using Apple Bug Reporter.

  • by Brian Kendall,

    Brian Kendall Brian Kendall Jan 13, 2015 10:42 PM in response to elfprince13
    Level 1 (5 points)
    Jan 13, 2015 10:42 PM in response to elfprince13

    I also experienced this bug, and I think it began right around the time I installed Yosemite. I almost lost some critically important data but thank god I had the good sense to back it up in an additional place.

     

    Unfortunately this means that Time Machine is now unusable garbage. I don't think I can ever, ever trust it again. So now I'm struggling to find a good replacement. Sure there's programs like SuperDuper or Carbon Copy Cloner but I want the iterative backups I got with Time Machine. Is there a good, reliable program that can do that?

  • by martin21045,

    martin21045 martin21045 Feb 12, 2015 6:03 AM in response to Brian Kendall
    Level 1 (0 points)
    Feb 12, 2015 6:03 AM in response to Brian Kendall

    Apple Time Machine is a really nifty concept, but is not quite ready for prime time since either I am missing something or Apple Time Machine has some issue. In either case Apple needs to perfect this. For now, at least for me, I have painfully learned the Apple Time Machine CANNOT BE RELIED UPON TO MAKE A COMPREHENSIVE BACKUP.

     

    Early in January 2015 I purchased a new MacBook Pro i7 2.8 (OSX 10.10.2), and to back it up I also bought the Apple AirPort Capsule 3TB. The AirPort Capsule mounted just fine, and I performed the first comprehensive backup of my Mac to it. Since then I have periodically backed up to the new Time Capsule over the WiFi at my home.

     

    A few days ago I accidentally deleted a long E-Mail draft from Microsoft Outlook. I attempted to simply restore my "Documents/Microsoft User Data/Office 2011 Identities/xx Main Identity/Database" database records file from Time Machine, currently about 82 MB on disk. I could find NO copy of the file Database, either in the original comprehensive Time Machine backup or on later, more recent backups. I was appalled to learn such a critical file to my work is apparently being skipped over by Time Machine.

     

    Last night I deleted com.apple.timemachine.plist, deleted the backups from the Apple AirPort Capsule, then ran Time Machine again. And yet again, most if not all other files on my MacBook Pro appear to have been backed up - EXCEPT MY MOST CRITICAL E-MAIL DATABASE FILE.

     

    I see no way to force a backup of "Database" into the Time Machine backup. I wish that somehow I could simply copy that manually from my MacBook to the AirPort Capsule Time Machine records, maybe after then Time Machine might know to look for it and compare modification dates on my MacBook's original file?

     

    I used Time Machine to backup my previous MacBook onto other drives, and had no issues using it - though I do not recall ever before having to access my Database file. Now I thank goodness that I have always used Carbon Copy Cloner too, backing up my Mac to another external drive every several days, and that does create comprehensive backups. Having more than one backup device, and ones that employ different software are among my core values.

     

    I feel really let down - if not misled by Apple.

  • by martin21045,

    martin21045 martin21045 Feb 12, 2015 6:28 AM in response to martin21045
    Level 1 (0 points)
    Feb 12, 2015 6:28 AM in response to martin21045

    Doing further research it appears the Microsoft Outlook 'Database' file is somehow coded so that Time Machine skips backing up that file.

     

    Apple Time Machine should at least initially open with some warning for the user explaining what files, especially those from major developers (Adobe, Microsoft, etc.) might not be backed up by Time Machine.

     

    In my past experience using "Carbon Copy Cloner" software to create backups of my Microsoft Outlook 'Database' files has allowed me to recover lost E-Mails with no problem. So I still fail to understand why Microsoft/Apple would not have that critical file copied by Time Machine.

  • by Loner T,

    Loner T Loner T Feb 12, 2015 7:27 AM in response to martin21045
    Level 7 (24,002 points)
    Safari
    Feb 12, 2015 7:27 AM in response to martin21045

    martin21045 wrote:

     

    Doing further research it appears the Microsoft Outlook 'Database' file is somehow coded so that Time Machine skips backing up that file.

     

    Apple Time Machine should at least initially open with some warning for the user explaining what files, especially those from major developers (Adobe, Microsoft, etc.) might not be backed up by Time Machine.

    Do you have a link from M$ to verify this? According to https://support.office.microsoft.com/en-US/Article/Automatically-archive-or-back -up-Outlook-items-441fcce5-2262-4b64-ac8c-fa949df989f5 , there is no such exclusion.

     

    Please also

     

    https://support.microsoft.com/kb/2448811?wa=wsignin1.0

    http://www.officeformachelp.com/2012/05/how-to-recover-your-data-when-rebuilding -restoring-and-upgrading-fails/

  • by martin21045,

    martin21045 martin21045 Feb 12, 2015 8:27 AM in response to Loner T
    Level 1 (0 points)
    Feb 12, 2015 8:27 AM in response to Loner T

    Before posting, I had come across and read the first link you referred me to, and read the others too. None of these links appear to apply to the question of why the E-Mail Database records file is being excluded from being backed up, at least by my Time Machine.

     

    The purposeful exclusion of Database seems to make the most sense at the following link; it seems to offer the most plausible, if not the only answer that could make any sense. I read that at:

     

    http://answers.microsoft.com/en-us/mac/forum/macoffice2011-macoutlook/time-machi ne-is-excluding-the-database-file-in/bb98a824-bcdd-4421-87c2-fff426259218

     

    PS: I have found no other files (yet) that also fail to back up to Time Machine, though Filemaker Pro does post some warnings about using Time Machine to back up/restore their files too. All in all, this experience is simply disconcerting because it has compromised the trust I had put in an Apple product - and the act of posting my experience on line is the best that I can do to warn others, and possibly to convey the experience to Apple Inc.

  • by Loner T,

    Loner T Loner T Feb 12, 2015 10:26 AM in response to martin21045
    Level 7 (24,002 points)
    Safari
    Feb 12, 2015 10:26 AM in response to martin21045

    Let me verify the validity of the assertion in the link your provided tonight.

  • by Loner T,

    Loner T Loner T Feb 12, 2015 6:02 PM in response to martin21045
    Level 7 (24,002 points)
    Safari
    Feb 12, 2015 6:02 PM in response to martin21045

    The M$ Outlook Identity Management layer sets the extended attribute, not TM. TM just honors what Outlook sets it to be.

     

    The extended attributes are (the highlighted value is what tell TM to exclude it).

    mdls ~/Documents/Microsoft\ User\ Data/Office\ 2011\ Identities/Main\ Identity/Database

    com_apple_backup_excludeItem   = "com.apple.backupd"

    kMDItemContentCreationDate     = 2014-02-08 18:18:24 +0000

    kMDItemContentModificationDate = 2015-02-13 01:07:30 +0000

    kMDItemContentType             = "dyn.ah62d4rv4gk8y8vnekk"

    kMDItemContentTypeTree         = (

        "public.data",

        "public.item"

    )

    kMDItemDateAdded               = 2014-02-08 18:18:24 +0000

    kMDItemDisplayName             = "Database"

    kMDItemFSContentChangeDate     = 2015-02-13 01:07:30 +0000

    kMDItemFSCreationDate          = 2014-02-08 18:18:24 +0000

    kMDItemFSCreatorCode           = "OPIM"

    kMDItemFSFinderFlags           = 0

    kMDItemFSHasCustomIcon         = (null)

    kMDItemFSInvisible             = 0

    kMDItemFSIsExtensionHidden     = 0

    kMDItemFSIsStationery          = (null)

    kMDItemFSLabel                 = 0

    kMDItemFSName                  = "Database"

    kMDItemFSNodeCount             = (null)

    kMDItemFSOwnerGroupID          = 20

    kMDItemFSOwnerUserID           = 501

    kMDItemFSSize                  = 100347904

    kMDItemFSTypeCode              = "OEDB"

    kMDItemKind                    = "Microsoft Outlook database"

    kMDItemLogicalSize             = 100347904

    kMDItemPhysicalSize            = 100347904

     

    Regarding

    Apple Time Machine should at least initially open with some warning for the user explaining what files, especially those from major developers (Adobe, Microsoft, etc.) might not be backed up by Time Machine.

    There is no mechanism to do this. If you install TM today and start backing up, and then install Outlook tomorrow, how will TM know if this was intentional or not. There could be hundreds of such files installed by applications which cannot be listed in the System Preferences -> Time Machine -> Options list.  If such a warning has to come, it should come from M$.

     

    This does not help you at all with your data loss, but may help those who follow. To prevent such data loss in the future, there is a nice little article at which may help others.

     

    http://hints.macworld.com/article.php?story=20080328025026826

     

    Please also see http://forums.macrumors.com/showthread.php?t=659839

     

    It can be removed using the following Terminal command

     

    xattr -dv com.apple.metadata:com_apple_backup_excludeItem  ~/Documents/Microsoft\ User\ Data/Office\ 2011\ Identities/Main\ Identity/Database

    which results in

     

    $ xattr -dv com.apple.metadata:com_apple_backup_excludeItem  ~/Documents/Microsoft\ User\ Data/Office\ 2011\ Identities/Main\ Identity/Database

     

    $ mdls ~/Documents/Microsoft\ User\ Data/Office\ 2011\ Identities/Main\ Identity/Database

    kMDItemContentCreationDate     = 2014-02-08 18:18:24 +0000

    kMDItemContentModificationDate = 2015-02-13 01:07:30 +0000

    kMDItemContentType             = "dyn.ah62d4rv4gk8y8vnekk"

    kMDItemContentTypeTree         = (

        "public.data",

        "public.item"

    )

    kMDItemDateAdded               = 2014-02-08 18:18:24 +0000

    kMDItemDisplayName             = "Database"

    kMDItemFSContentChangeDate     = 2015-02-13 01:07:30 +0000

    kMDItemFSCreationDate          = 2014-02-08 18:18:24 +0000

    kMDItemFSCreatorCode           = "OPIM"

    kMDItemFSFinderFlags           = 0

    kMDItemFSHasCustomIcon         = (null)

    kMDItemFSInvisible             = 0

    kMDItemFSIsExtensionHidden     = 0

    kMDItemFSIsStationery          = (null)

    kMDItemFSLabel                 = 0

    kMDItemFSName                  = "Database"

    kMDItemFSNodeCount             = (null)

    kMDItemFSOwnerGroupID          = 20

    kMDItemFSOwnerUserID           = 501

    kMDItemFSSize                  = 100347904

    kMDItemFSTypeCode              = "OEDB"

    kMDItemKind                    = "Microsoft Outlook database"

    kMDItemLogicalSize             = 100347904

    kMDItemPhysicalSize            = 100347904

  • by Loner T,

    Loner T Loner T Feb 12, 2015 6:23 PM in response to martin21045
    Level 7 (24,002 points)
    Safari
    Feb 12, 2015 6:23 PM in response to martin21045

    It can be verified that the xattr command works, by checking after a "Backup Now" and looking at the latest backup or using

    tmutil isexcluded ~/Documents/Microsoft\ User\ Data/Office\ 2011\ Identities/Main\ Identity/Database

    [Included] /Users/MyName/Documents/Microsoft User Data/Office 2011 Identities/Main Identity/Database

  • by martin21045,

    martin21045 martin21045 Feb 12, 2015 7:18 PM in response to Loner T
    Level 1 (0 points)
    Feb 12, 2015 7:18 PM in response to Loner T

    I am glad to read you confirmed my understanding of what I had mentioned reading elsewhere about the tag. But even more I appreciate the explanation and demo about how to go into Terminal and change that. I learned about as much about using Terminal as I had ever known before. And I read the referenced articles you mentioned.

     

    Next I double-checked and confirmed the TM is told to exclude backing up Outlook's Database file. I changed that, then entered the Terminal command again to verify that Database file is not among those to be excluded. Next I ran Time Machine "Back Up Now" command and found the file "Database" is now backed up onto my AirPort Capsule.

     

    Just to see what all the fuss was about with Microsoft tagging, I copied the backup Database over and opened it, it does not open as perfectly as it might since a new message window came up with some 57 mb (as I recall) attachment. But once I deleted the attachment, then deleted the empty E-Mail form, I see that it will be possible to recover other E-Mails by exporting them from Outlook. For my money it is better to have some Database backup on TM than none.

     

    Regardless of the limited success with TM I will also continue to use Carbon Copy Cloner on another hard drive to back up all files. I seem to have no problem reloading "Database" from the cloned drive, installing that and opening it up with no problems.

     

    Thanks again.

     

    PS: I do wish Apple would have some small application or some function in TM that would allow users to scan their Mac to look for and reveal the 'exclude TM' tagging. The app could be run just prior to the first Time Machine Backup, or just provide some hints of what software might not have this tagging, or be there to be run on demand. I do not claim to know the best way to implement a way to warn users, but after I lost my E-Mail and could not find it on TM I did lose some sleep over this until you and others on the Web helped me to comprehend what was going on.

Previous Page 2 of 4 last Next