Time machine missing files

One option to consider is to backup to two different TM disks, and alternate between the two. It is supported by TM.

Hi,

I don't see how that would have helped in this situation, I had TM backups going back a month on one drive, I also had backups on a portable drive. I just would have two drives with TM backups without the latest files backed up.

For some reason TM was completing backups and not saving the extra files added in the directories where I saved them for over a week, I can't see how a different destination drive would have changed that fact.

I can only now use TM as an OS recovery tool, data will be handled in a more reliable way.

I encountered this problem for the first time after upgrading to Yosemite, when I realized that my backups from Mavericks had been silently missing certain iTunes files for many months. Fortunately, I was able to isolate the missing files by comparing the extensive TM backups and restoring them.

I should have switched away from TM then, but didn't learn my lesson. I replaced the failing iMac a few weeks ago, restoring from the latest TM backup to the new Mac Mini, and lo and behold, I'm missing large swathes of my documents now. Fortunately, I have several redundant backups (two TM drives, a clone of the hard-drive back from ~January, a daily tarsnap backup of my Documents directory, as well as the presumably still good iMac hard-drive), but now I have the painful process of reconciling all these together to recover my data.

Note to self: stop using time machine. Maybe switch to something like https://blog.interlinked.org/tutorials/rsync_time_machine.html instead.

lieut_data wrote:

I encountered this problem for the first time after upgrading to Yosemite, when I realized that my backups from Mavericks had been silently missing certain iTunes files for many months. Fortunately, I was able to isolate the missing files by comparing the extensive TM backups and restoring them.

I should have switched away from TM then, but didn't learn my lesson. I replaced the failing iMac a few weeks ago, restoring from the latest TM backup to the new Mac Mini, and lo and behold, I'm missing large swathes of my documents now. Fortunately, I have several redundant backups (two TM drives, a clone of the hard-drive back from ~January, a daily tarsnap backup of my Documents directory, as well as the presumably still good iMac hard-drive), but now I have the painful process of reconciling all these together to recover my data.

Note to self: stop using time machine. Maybe switch to something like https://blog.interlinked.org/tutorials/rsync_time_machine.html instead.

Can you confirm (if you did the comparisons to know this) whether your multiple TM backups had the same or different files missing?

I can confirm that my second TM backup was /not/ missing the files. (I don't know if it's missing any other files!)

Having same problems. With mine, I'm restoring to a new drive from a TM backup (Mavericks) and finding several non apple applications including all Microsoft applications were not restored. Nor was iphoto, but more importantly who knows what else. Currently trying previous backup instances, but I'm definitely frightened. This is my girlfriends personal machine, but to think this backup system is what I'm using on my own business machine, on which EVERYTHING is relying.... Totally terrifying. Did I miss something, or have Apple reps not chimed in at all on this serious issue? A lot of us don't have the skills or the patience to be running terminal commands on are gulag basis to confirm that TM is doing what it says it will. I'm ordering a new backup drive tomorrow and going back to a nightly disk clone. The time machine concept never quite added up to me. I far prefer a consistent single reliable complete backup than N number of incomplete backups over the last N number of weeks.

TM has gotten better, but it still has software bugs. 😉. It is better to have two different backup tools and offsite storage as the criticality and value of data increases.

UPDATE! I wish I could edit my first post. In my particular situation it turned out to be user error. I was able to copy over all the basic mac applications and utilities from another system - enough to be able to launch TM and see that my partner actually did have Applications excluded from her TM backups, though she swears she didn't set that. Oops. :/

As long as you get all your files back, it is all good. 🙂

I have the same problem. Soooo frustrating!

I have been doing regular backups onto an external hard drive, which I only connect when I need to, so it's not connected permanently. I did a clean install of El Capitan the other day, then manually restored folders from the TM backup (I actually copied and pasted them via the Finder window). I only wanted to restore Documents, Pictures, Music as I decided to do a clean install of all my Apps as well (it wasn't that much of a big deal to do this, I don't have that many apps outside of the usual - Office, Firefox, etc.).

All looked great until my boyfriend noticed missing documents. I didn't really believe him at first, I thought he must have deleted them before the last backup. Of course he hadn't - Time Machine messed up! 😠

I did a bit more investigating and seem to have identified a bit of a pattern:

• New files that have been added to existing folders are being backed up
• New folders and their contents are missing from the backup
• Folders that have been renamed are missing their content (the folder itself is there but it's empty)

I am not entirely certain about this though. It feels like Time Machine is getting confused between what has and hasn't changed?!

This line in Pondini's post http://pondini.org/TM/Works.html makes me think Time Machine is sort of only half doing its job. Is it treating renamed folders as "nothing has changed"?

If nothing in a folder is changed, Time Machine makes a single hard link for the folder, not one for each file inside it.

Ugh, I am really upset about this. My boyfriend is quite angry with me because "my careless action" (i.e. not doing another kind of backup outside of Time Machine) has lost him a bunch of files that he can't retrieve from anywhere. *sigh*

Definitely lost my faith in Apple, don't think I'll stick with them for my next laptop purchase. Shame.

I'd like to make an update about my situation...

I decided to give Time Machine a try again, and in doing so wrote a python script to specifically check to see if it was missing any important files of mine. The tool scans the entire filesystem and then compares against Time Machine's backup, and then reports back any files that are missed. If the files are marked as being excluded (as reported by using 'tmutil isexcluded path-to-file') it will list them separately. It then gives the option to remove the exclusion marker from the excluded files.

It turns out all of the important files of mine that Time Machine was missing were marked as being excluded. After I ran this tool, unmarked those files, and did a Time Machine backup, everything was once again backed up properly. I'm hoping this will fix the issue for me and allow me to use Time Machine once again, because when it's working it simply is the best tool available for making local backups.

(I suppose it's also possible that starting again from scratch or all of the OS updates that have happened between then and now could have fixed the underlying issue, but there's no way to be sure at this point.)

Would any of you in this thread consider trying out my script? I'm very curious if the files that people are concerned about being missing are in fact just being marked as excluded, and Time Machine is actually functioning as it should (for some definitions of functioning). Here's a link to my github page: https://github.com/briankendall/check-time-machine

You simply run the script as root from Terminal, i.e.:

sudo python checktimemachine.py

- or -

sudo python checktimemachine.py /path/to/specific/directory

and then wait for a bit while it scans everything. It's programmed to automatically ignore intentionally excluded operating system files (from Apple's private list), as well as several directories and files that are marked as excluded that I was pretty sure were safe to ignore. These exclusions are listed at the top of the script and you can modify them as you want.

Note that it requires the Time Machine volume to be present and accessible from the system... not sure how this will work with Time Capsule or networked Time Machine volumes, but in theory it should work.

I recently realized that my time machine backups are missing files. REAL FILES. WORKING & CURRENT PROJECT FILES. UNACCEPTABLE!!!!!!

Yosemite with all updates.

I'm going to try your routine if I can figure out how to.

Thanks,

Billy Earnest

Were any subfolders relocated elsewhere? I typically use a tmutil compare on a regular basis.