jonttaylor

Q: Time machine missing files

Timeline

 

1. 11 days ago I buy a new 15" retina haswell macbook pro running mavericks, and a new 2TB time capsule.

2. Yesterday I returned the macbook pro because it was making creaking noises and had it replaced with a new one.

3. Now I try and restore my backup.

 

Story

 

I restored from my latest time machine backup by pressing CMD-R while booting and specifying to restore from the Time Capsule. It took a couple of hours over wifi and finally restored, when I came back to it, the machine had rebooted and I was at my normal login screen.

 

So I thought the restore had been successful at this point and I log in, however it asks me for my Apple ID login as if i've never entered it before (I have), and I have some new mavericks icons at the end of my dock that I had previously removed. I open up my terminal (i'm a web developer) and I notice its missing my custom prompt (oh-my-zsh) which I track down to the .zshrc file missing.

 

I check my databases and they are all corrupted. The MySQL error file is constantly spamming itself saying the data is corrupt and that it might be missing its binlogs. A number of my apps don't remember that they are licensed. The machine has forgotten most of my preferences such as tap to click, however it has remembered things like my desktop background and my safari preferences (they were not synced from icloud I made sure).

 

I dig into this a bit more by mounting the time capsule and having a look around in the terminal.

 

cd /Volumes/Time\ Machine\ Backups/Backups.backupdb

ls -lah 2013*/Macintosh\ HD/Users/jon

 

This prints out the contents of what the backup considers my home folder to be like at every backup (there were 23 backups stored over the 10 days). First backup being on the 2nd of November, most recent being on the 12th November.

 

I see that from the 7th my .zshrc went missing, it was not in any subsequent backups after that, so I tried restoring back to that date. After restoring my terminal preferences are back, but the databases are still corrupt and I notice other things missing instead.

 

I dig into it a bit more and list the files again.

 

Then I notice just how bad it is.

 

On certain backups it has missed massive numbers of files, including directories such as Downloads Library Desktop etc. Then they come back in the next backup, or go missing forever. I know time machine doesn't backup the entire computer each time, but I know that it should be creating HARD links in the filesystem, so that EVERY backup appears to be a complete system backup. This is not the case.

 

I can rebuild this new macbook pro to have everything I need on it again, but I CANNOT trust my £250 Time Capsule to take backups, in which case it becomes an extremely expensive router + hard drive.

 

I think this might be a bug in Mavericks, 10 days is not very long for backups to start going astray!

 

Is there any way I can repair this somehow? Has anyone else had this kind of problem? I phoned apple support and they just tried to guide me through doing a re-install, when I explained that the backups themselves seemed to be missing files the support agent just didn't seem to understand, not very helpful at all. As a fellow engineer I would prefer to speak to another engineer and I could provide all the diagnostics they could want, short of giving them ssh access.

MacBook Pro with Retina display, OS X Mavericks (10.9), Backing up to a Time Capsule

Posted on Nov 13, 2013 1:52 AM

Close

Q: Time machine missing files

  • All replies
  • Helpful answers

first Previous Page 3 of 4 last Next
  • by brenowden,

    brenowden brenowden Feb 12, 2015 7:38 PM in response to jonttaylor
    Level 1 (13 points)
    Desktops
    Feb 12, 2015 7:38 PM in response to jonttaylor

    Has anyone figured out what has happened.   My new iMac less than 5 months old, went "gray screen".  Genius bar wiped out my computer and reinstalled everything to fix gray screen.  Assumed since I had a recent time machine back up all was good.  Fast forward a few days and find ALL my documents are MIA.  First the documents folder was missing, so the tech had me reinstall using the date before gray screen happened.  The document folder reappeared and I thought all was well.  Nope.  No contents in the document folder.  All documents are gone.  Not on external and not on iMac anywhere.  The tech is back to helping but even his is perplexed and has lead me to believe all is missing.  I'm not sure why time machine is touted as a great feature on the apple products when it appears to have major glitches which are catastrophic.  Anyone have any ideas what has happened? 

  • by Loner T,

    Loner T Loner T Feb 12, 2015 8:14 PM in response to martin21045
    Level 7 (24,394 points)
    Safari
    Feb 12, 2015 8:14 PM in response to martin21045

    Thanks for running the confirmation after the changes. I also ran the Compare, but did not test the restore, since it is my work email.

     

    martin21045 wrote:

     

    PS: I do wish Apple would have some small application or some function in TM that would allow users to scan their Mac to look for and reveal the 'exclude TM' tagging. The app could be run just prior to the first Time Machine Backup, or just provide some hints of what software might not have this tagging, or be there to be run on demand. I do not claim to know the best way to implement a way to warn users, but after I lost my E-Mail and could not find it on TM I did lose some sleep over this until you and others on the Web helped me to comprehend what was going on.

     

    Please see

    1. http://hints.macworld.com/article.php?story=20120716005852530

    2. http://support.code42.com/CrashPlan/Latest/Troubleshooting/CrashPlan_and_OS_X_Me tadata

  • by Loner T,

    Loner T Loner T Feb 12, 2015 8:26 PM in response to brenowden
    Level 7 (24,394 points)
    Safari
    Feb 12, 2015 8:26 PM in response to brenowden

    1. After the re-install on the iMac, when TM was set up again, did it ask you for inheriting the previous backups?

    2. You should periodically run tmutil compare, perhaps to ensure that you are expectations are being met vis-a-vis backup files.

    3. If the TM Destination is on a non-Time Capsule device, there is something you should be aware of -  AirPort base stations: Time Machine hard drive compatibility - Apple Support. This states explicitly that external non-Apple NAS storage devices are not supported. The 80211.ac model will support a USB-connected device directly connected to it.

  • by suprimeau8,

    suprimeau8 suprimeau8 Mar 20, 2015 7:14 PM in response to jonttaylor
    Level 1 (0 points)
    Mar 20, 2015 7:14 PM in response to jonttaylor

    I just wanted to add that I had the same problem. I decided to do a clean install of Yosemite and noticed a bunch of iTunes music missing. I restored from backup to see if that would bring them back but I had the same issue. I then noticed a bunch of Mail messages missing from "On My Mac". I was able to go back to September 2014 in Time Machine and retrieve the files so I was lucky. I've got a lot of photos so who knows if that's all there. Seems to be some issues from Mavericks to Yosemite upgrade possibly. This is very unnerving to see this. There must be tons of people unknowingly backing up stuff thinking it's working. Hope this gets resolved. I've always had no issues in the past.

  • by Loner T,

    Loner T Loner T Mar 20, 2015 7:22 PM in response to suprimeau8
    Level 7 (24,394 points)
    Safari
    Mar 20, 2015 7:22 PM in response to suprimeau8

    My recommendation is to use Terminal and execute tmutil compare to see what is the delta between two backups.

  • by suprimeau8,

    suprimeau8 suprimeau8 Mar 20, 2015 7:41 PM in response to Loner T
    Level 1 (0 points)
    Mar 20, 2015 7:41 PM in response to Loner T

    It's a good idea before doing a restore or clean install. Unfortunately it's not much of use after the fact and a few years of backups. No idea when the problems started.

  • by Loner T,

    Loner T Loner T Mar 20, 2015 8:50 PM in response to suprimeau8
    Level 7 (24,394 points)
    Safari
    Mar 20, 2015 8:50 PM in response to suprimeau8

    Yes, I quite agree with your observation.

     

    Unfortunately, the TM issues have been there since post-SL days. They have lingered through many iterations of OS X versions in one form or other. Some users use two different tools and run two backups at least once a day to separate external storage for redundancy and recoverability. There is also a suggested weekly restore and boot exercise of sensitive data using tertiary external storage. This is a very expensive and sometimes very labor intensive process.

     

    Loss of information causes lost trust in a tool and can raise questions regarding brand loyalty. There is enough legalese to cover and indemnify the software vendor.

  • by rationull,

    rationull rationull Jul 31, 2015 8:11 PM in response to jonttaylor
    Level 1 (0 points)
    Jul 31, 2015 8:11 PM in response to jonttaylor

    I first noticed this issue last November after having run Time Machine since it was first introduced, and it shook my confidence quite a bit. I have always been able to "fix" the problem temporarily by using "touch" on the affected files and rerunning the backup, at which point they would get copied. The problem persisted through an internal drive replacement and now on a new machine, which went a few months without any detectable missing copies until I found one today.

     

    I've been thinking of ways to streamline the "touch" workaround but then I realized that if, in order to get a reliable backup, you need to run tmutil compare, modify some files and potentially run another (quick) backup EVERY TIME then what's the point? I think I'm going to rely 100% on cloning solutions with an archive of changed files from here on out, which I've been running alongside TM anyway since I first hit this issue.

     

    I filed a bug on bugreport.apple.com when I first hit this, and it's still open. I hope others have filed bugs as well. It'd be great if this could be fixed at some point but it would obviously take some time for TM to build trust again.

  • by MRCHRISD73,

    MRCHRISD73 MRCHRISD73 Aug 6, 2015 1:58 AM in response to rationull
    Level 1 (0 points)
    Aug 6, 2015 1:58 AM in response to rationull

    After restoring a Time machine backup this morning I have hundreds of missing documents, going through the time machine backups manually even though I have tens of backups dated yesterday, the last time my scanned documents were backed up is over a week ago. I've lost 10 hours worth of duplex scanning.

    Time machine is worthless and I wouldn't trust it from this point forward.

  • by Loner T,

    Loner T Loner T Aug 6, 2015 10:35 AM in response to MRCHRISD73
    Level 7 (24,394 points)
    Safari
    Aug 6, 2015 10:35 AM in response to MRCHRISD73

    I usually run tmutil compare to get a listing of files backed up. It may not help you now, but may be worth exploring further.

  • by MRCHRISD73,

    MRCHRISD73 MRCHRISD73 Aug 8, 2015 8:02 AM in response to Loner T
    Level 1 (0 points)
    Aug 8, 2015 8:02 AM in response to Loner T

    Thanks, there are so many ways I could have recovered my machine and not lost the data, the thing is I believed Time machine had the data backed up. Now I know Time Machine cannot be trusted to backup my data I'll change the way I handle it altogether, boot from another drive first and lift off any data if needed but I'll keep critical stuff off the boot drive.

    I just thought people should know it's unreliable, a folder I added files to daily and dated them hadn't had new files added in over 20 backups going back 8 days.

    Hopefully your tip may help somebody, for anyone in my situation data recovery software didn't recover anything wiped by the time machine restore.

  • by Loner T,

    Loner T Loner T Aug 8, 2015 8:49 AM in response to MRCHRISD73
    Level 7 (24,394 points)
    Safari
    Aug 8, 2015 8:49 AM in response to MRCHRISD73

    One option to consider is to backup to two different TM disks, and alternate between the two. It is supported by TM.

  • by MRCHRISD73,

    MRCHRISD73 MRCHRISD73 Aug 8, 2015 11:58 AM in response to Loner T
    Level 1 (0 points)
    Aug 8, 2015 11:58 AM in response to Loner T

    Hi,

    I don't see how that would have helped in this situation, I had TM backups going back a month on one drive, I also had backups on a portable drive. I just would have two drives with TM backups without the latest files backed up.

    For some reason TM was completing backups and not saving the extra files added in the directories where I saved them for over a week, I can't see how a different destination drive would have changed that fact.

     

    I can only now use TM as an OS recovery tool, data will be handled in a more reliable way.

  • by lieut_data,

    lieut_data lieut_data Aug 16, 2015 6:45 PM in response to jonttaylor
    Level 1 (0 points)
    Aug 16, 2015 6:45 PM in response to jonttaylor

    I encountered this problem for the first time after upgrading to Yosemite, when I realized that my backups from Mavericks had been silently missing certain iTunes files for many months. Fortunately, I was able to isolate the missing files by comparing the extensive TM backups and restoring them.

     

    I should have switched away from TM then, but didn't learn my lesson. I replaced the failing iMac a few weeks ago, restoring from the latest TM backup to the new Mac Mini, and lo and behold, I'm missing large swathes of my documents now. Fortunately, I have several redundant backups (two TM drives, a clone of the hard-drive back from ~January, a daily tarsnap backup of my Documents directory, as well as the presumably still good iMac hard-drive), but now I have the painful process of reconciling all these together to recover my data.

     

    Note to self: stop using time machine. Maybe switch to something like https://blog.interlinked.org/tutorials/rsync_time_machine.html instead.

  • by rationull,

    rationull rationull Aug 16, 2015 8:23 PM in response to lieut_data
    Level 1 (0 points)
    Aug 16, 2015 8:23 PM in response to lieut_data

    lieut_data wrote:

     

    I encountered this problem for the first time after upgrading to Yosemite, when I realized that my backups from Mavericks had been silently missing certain iTunes files for many months. Fortunately, I was able to isolate the missing files by comparing the extensive TM backups and restoring them.

     

    I should have switched away from TM then, but didn't learn my lesson. I replaced the failing iMac a few weeks ago, restoring from the latest TM backup to the new Mac Mini, and lo and behold, I'm missing large swathes of my documents now. Fortunately, I have several redundant backups (two TM drives, a clone of the hard-drive back from ~January, a daily tarsnap backup of my Documents directory, as well as the presumably still good iMac hard-drive), but now I have the painful process of reconciling all these together to recover my data.

     

    Note to self: stop using time machine. Maybe switch to something like https://blog.interlinked.org/tutorials/rsync_time_machine.html instead.

     

    Can you confirm (if you did the comparisons to know this) whether your multiple TM backups had the same or different files missing?

first Previous Page 3 of 4 last Next