Time Machine Not Backing Up New Files

G'day

Further to yesterday's post I ran my new utility on my other Time Machine drive. (I run two TM backup drives. Each evening I bring the current one up to date, swap it for the other, and then bring that one up to date while I have dinner.)

The results were:

Checked 1,443,492 files

Found 13,466 files not in latest backup but should be

Found 22 files modified since latest backup

Found 2 files with zero last used date that were backed up

Found 65 files with zero last used date that were not backed up

I didn't feel the need to check that the 65 and 2 files with zero last modified date were the same as the 65 and 2 files that were and weren't backed up on the other drive. It seems TM has a problem with files with zero last modified date even though some of them are of Apple's creation.

Similarly the 5,300-odd files that were not on the first backup drive are also not on the 2nd backup drive. In addition 8,000 files that I created during development work today weren't backed up either. Were I to rerun the utility over the first backup drive I'm confident it would now show 13,000 plus files not backed up.

So TM is consistent in its errant ways. Which is good news because it makes it easier to find the issue and gives more confidence in the eventual fix.

I think its time to approach Apple.

G'day

And finally for the night: I used the suggestion of Linc Davis and got the list of files and path prefixes that TM excludes from the backup, stripped it, and read it into my utility so that its exclusion list was the same as TMs (except I also exclude anything with cache, Caches, or .Trash in the path). This made very little difference - still over 13,000 files missing from the backup, including the same 65 with zero last modified date.

I'm wondering how you got (or find) files with a zero modification date.

I can't find any on my system, either via a Finder search (including system and hidden files), or via the FindAnyFile app, including hidden files and package contents. I don't have most of the files you listed, but the few I do all have dates. I can't even find one before 1/1/2003.

Can't find a way to make one, either -- touch won't accept zeros, and midnight on 1/1/1970 (or any variation trying to adjust to GMT and DST) just shows as the date, not a zero, and TM backs it up.

Is it possible there's something else on those files, such as the com.apple.backupd extended attribute, or inclusion in the ExcludeByPath section of the plist file?

And/or, is it possible those files have been created or dates have been set to zero by some process running on a non-OSX volume somewhere (that doesn't update fseventsd)? OSX should catch that, of course, but if it doesn't, those changes won't get backed-up.

For an initial backup, TM is supposed to back up everything that's not on one of the exclusion lists. I guess it's possible that coding somehow skips items with a zero modification date, but it does seem unlikely.

I've never seen it documented (surprise!) but I understand that on an incremental backup, when TM gets a notice (via fseventsd) that something has changed somewhere in a folder; and compares the contents to the most recent backup, it compares both the modification date and size of each item. So it ought to catch nearly everything that's been changed, unless of course it also skips zero modification dates as above and the size didn't change by even a single byte.

But by all means, do involve Apple, either via AppleCare or by filing a detailed bug report.

G'day

A few that might be on your system are:

Not in backup: file at /System/Library/Frameworks/JavaFrameEmbedding.framework/Headers with last used date of Thu Jan 01 09:30:00 CST 1970

Not in backup: file at /System/Library/Frameworks/Python.framework/Versions/2.3/Extras with last used date of Thu Jan 01 09:30:00 CST 1970

Not in backup: file at /usr/bin/wx-config with last used date of Thu Jan 01 09:30:00 CST 1970

Not in backup: file at /usr/share/java/junit-4.1-src.jar with last used date of Thu Jan 01 09:30:00 CST 1970

Not in backup: file at /usr/share/java/junit-4.1.jar with last used date of Thu Jan 01 09:30:00 CST 1970

Not in backup: file at /usr/share/java/Tools/Java VisualVM.app/Contents/Resources/VisualVM.icns with last used date of Thu Jan 01 09:30:00 CST 1970

Not in backup: file at /usr/share/man/man1/jvisualvm.1 with last used date of Thu Jan 01 09:30:00 CST 1970

Finder won't find them, as you say. I discovered them when I asked my utility to print the last modified date along with the name of any file that it found wasn't in the backup.

Ah. A little investigation shows that all of the first few files with zero last modified dates are aliases that are broken. My guess is that the zero last modified date is that of the file being linked to (no file, so no date) - the aliases themselves have valid last modified dates in Finder. My utility is coded to ignore symbolic links as best it can but, lacking Java 7, this is done with a heuristic algorithm from Apache which is known not to be 100% accurate. Looks like the code doesn't work correctly when the link is broken.

I can't say I care much whether a broken link is backed up or not - trying to follow it in a program might give a different error message if the link is present, but broken, from no link being present. But either way the file won't be found. I think I'll get the program to ignore any file with a zero last modified date presuming it to be a broken link.

I've rerun the progam ignoring files with zero last modified date. All missing files, with 3 exceptions, are now rooted in /Users/Ian and are later than the last backup that I did in which I created a directory in /Users/Ian to force a deep scan therein (lots of backups since without doing that). The three odd files are:

Not in backup: file at /.dbfseventsd with last used date of Sat Jul 27 08:35:35 CST 2013

Not in backup: file at /Library/Application Support/ArcSoft/Connect Service/._csconfig.arg with last used date of Wed Apr 03 17:01:52 CST 2013

Not in backup: file at /Library/Application Support/ArcSoft/Connect Service/._regdata.arg with last used date of Wed Apr 03 17:01:46 CST 2013

The first is obviously related to fseventsd, which is excluded, so I think I can ignore that. The other 2 have file names starting with ._ . I have one other such file on my system and that is not backed up either. So I guess a file name starting with ._ isn't backed up by TM. All 3 belong to a product I don't care about so I can ignore these.

So the problem reduces to the 13,000 (and increasing) files under /Users/Ian. I have a work around. But since the problem is reproducible I think I will still involve Apple to see if we can resolve this.

Ian Blavins wrote:

. . .

I can't say I care much whether a broken link is backed up or not

They aren't. Aliases and symbolic links are backed-up, but the links are only backed-up normally.

The other 2 have file names starting with ._ . I have one other such file on my system and that is not backed up either. So I guess a file name starting with ._ isn't backed up by TM. All 3 belong to a product I don't care about so I can ignore these.

There's nothing like that not on the standard exclusion list. But they they may have the extended attribute or be listed in the ExcludeByPath section of the TM plist (see the gray box in How Time Machine works its Magic for details).

So the problem reduces to the 13,000 (and increasing) files under /Users/Ian.

Those don't have the zero dates, right? Is there some pattern to them? All in the same folder(s)? Or something in the above link?

As noted earlier in this thread, we have see that rarely -- something (no clue what) wrong with a folder, so nothing inside it ever gets on an incremental backup. When it does happen, it affects everything in the folder, not just some files.

I have a work around. But since the problem is reproducible I think I will still involve Apple to see if we can resolve this.

If you're still on Snow Leopard, don't expect much, since support for it stopped when Lion was released.

That's why I was trying to see if I could reproduce it on Mountain Lion.

G'day

They aren't. Aliases and symbolic links are backed-up, but the links are only backed-up normally.

In the case of the broken link the alias or symbolic link is not being backed up. It is that file that is missing and that's what I don't care about since its pretty much worthless anyway.

There's nothing like that not on the standard exclusion list. But they they may have the extended attribute or be listed in the ExcludeByPath section of the TM plist (see the gray box in How Time Machine works its Magic for details).

Don't appear to be deliberatly excuded in the plist:

<?xml version="1.0" encoding="UTF-8"?>

<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">

<plist version="1.0">

<dict>

<key>sourcePaths</key>

<array>

<string>/</string>

<string>/Volumes/Pacific</string>

</array>

<key>standardExclusionPaths</key>

<array>

<string>/.DocumentRevisions-V100</string>

<string>/.MobileBackups</string>

<string>/MobileBackups.trash</string>

<string>/.MobileBackups.trash</string>

<string>/.Spotlight-V100</string>

<string>/.Trashes</string>

<string>/.fseventsd</string>

<string>/.hotfiles.btree</string>

<string>/Backups.backupdb</string>

<string>/Desktop DB</string>

<string>/Desktop DF</string>

<string>/Network/Servers</string>

<string>/Previous Systems</string>

<string>/Users/Shared/SC Info</string>

<string>/Users/Guest</string>

<string>/dev</string>

<string>/home</string>

<string>/net</string>

<string>/private/var/db/efw_cache</string>

<string>/private/var/db/Spotlight</string>

<string>/private/var/db/Spotlight-V100</string>

<string>/Volumes</string>

<string>/Network</string>

<string>/automount</string>

<string>/.vol</string>

<string>/tmp</string>

<string>/cores</string>

<string>/private/tmp</string>

<string>/private/Network</string>

<string>/private/tftpboot</string>

<string>/private/var/automount</string>

<string>/private/var/folders</string>

<string>/private/var/run</string>

<string>/private/var/tmp</string>

<string>/private/var/vm</string>

<string>/private/var/db/dhcpclient</string>

<string>/private/var/db/fseventsd</string>

<string>/Library/Caches</string>

<string>/Library/Logs</string>

<string>/System/Library/Caches</string>

<string>/System/Library/Extensions/Caches</string>

<string>/private/var/log</string>

<string>/private/var/spool/cups</string>

<string>/private/var/spool/fax</string>

<string>/private/var/spool/uucp</string>

</array>

<key>systemFilesExcluded</key>

<false/>

<key>userExclusionPaths</key>

<array>

<string>/Volumes/Pacific/Not TimeMachined</string>

</array>

</dict>

</plist>

and don't appear to have the extended attribute:

-rw-r--r-- 1 root admin 8 3 Apr 17:01 ._csconfig.arg

-rw-r--r-- 1 root admin 962 3 Apr 17:01 ._regdata.arg

Those don't have the zero dates, right? Is there some pattern to them? All in the same folder(s)? Or something in the above link?

As noted earlier in this thread, we have see that rarely -- something (no clue what) wrong with a folder, so nothing inside it ever gets on an incremental backup. When it does happen, it affects everything in the folder, not just some files.

These don't have the zero dates. The commonality is that they are all under /Users/Ian, directly or indirectly but within that folder they are all over the place. They do however come in clumps - lots within a single directory here, some more in a single directory there. Doesn't appear to be all the files in such a directory (but it might be all files in such a directory modified since some date for instance, though given the last modified dates on some of the missing files is >10 years ago that probably isn't relevant).

If you're still on Snow Leopard, don't expect much, since support for it stopped when Lion was released.

That's why I was trying to see if I could reproduce it on Mountain Lion.

I thought there was sufficient evidence in the earlier portion of this thread to suggest the problem persists in Lion. I'm thinking that Apple and I could use my machine as a test bed to get to the bottom of the issue with the expectation that it would solve the problem for all versions of Mac OS X ie. we could take the opportunity presented to find out what the "rarely -- something (no clue what) wrong with a folder" is and eliminate it. But if we can reproduce the problem on Lion that would obviously be better and give Apple more motivation to solve the problem.

Since my last post I've put a new folder into /Users/Ian and done a fresh TM backup. That dropped the number of missing files from 13,000 odd to about 1,500. But the number will steadily grow again as I do work. So the 13,000 odd divide into two categories - those that will backup on a deep traversal and those that won't. I would say that all or most of the latter files weren't backed up in the full backup either, which is a bit of a worry.

Ian Blavins wrote:

There's nothing like that not on the standard exclusion list. But they they may have the extended attribute or be listed in the ExcludeByPath section of the TM plist (see the gray box in How Time Machine works its Magic for details).

Don't appear to be deliberatly excuded in the plist:

Right. Have you checked the TM plist per the above link?

These don't have the zero dates. The commonality is that they are all under /Users/Ian, directly or indirectly but within that folder they are all over the place.

Anything common in what created them, file types, etc.?

Perhaps I missed it earlier, but are other user home folders affected, too?

I thought there was sufficient evidence in the earlier portion of this thread to suggest the problem persists in Lion.

There seem to be two problems that a few folks have seen, mostly (or entirely) on Snow Leopard, but different from yours: some sort of problem with fseventsd; and something with individual folders. It seems you've ruled out both of those.

To get much attention, it would have to continue to Mountain Lion (and soon to Mavericks). The last update to Lion (other than security updates) was last September; since Mavericks is supposed to be released in "the fall," there probably won't be many more updates to it.

G'day

Right. Have you checked the TM plist per the above link?

Yes. And I posted it so others could verify that.

Anything common in what created them, file types, etc.?

Nothing obvious. There are examples that are created in different ways with different tools. The older files that haven't ever been backed up are mainly Application Support files.

Perhaps I missed it earlier, but are other user home folders affected, too?

Don't know. There is only one user of consequence defined on the machine. I could define others but not sure we would learn enough to justify the effort. The problem we have is big enough.

I've worked through some of the old files that have never been backed up by TM and sufficient have extended attributes excluding them that I'll assume they are all correctly excluded from the backup. I can't detect extended attributes in my program till Java 7 (which implies some version of Lion which I don't want to do just now).

Ian Blavins wrote:

G'day

Right. Have you checked the TM plist per the above link?

Yes. And I posted it so others could verify that.

Sorry, no, I meant the preferences file (/Library/Preferences/com.apple.TimeMachine.plist). Check for FixedPathExclusions (or anything else odd) per the Sample Plist section at the bottom of the gray box inHow Time Machine works its Magic. Given the number of files excluded, that seems very unlikely, but any possibility's worth a look.

Perhaps I missed it earlier, but are other user home folders affected, too?

Don't know. There is only one user of consequence defined on the machine. I could define others but not sure we would learn enough to justify the effort. The problem we have is big enough.

It might be worthwhile creating a second user account and trying a small experiment with it. Anything that might find or rule out a common condition.

Hello all,

I've finally had some time to get back to investigating this issue. You can use rsync in "dry run" mode to compare the latest TM backup with the current state of the drive/folder in question. This will show which files are not present or have been updated, but can't do any of the filtering like Ian's program.

Here's how I use it to check for differences on my lab's drive. Note that the trailing slash on the source drive (i.e. /Volumes/LabDrive/) is required. Also, don't forget to include the -n option for rsync, since this specifies a "dry run" (i.e. only print what would be changed, and don't actually sync the folders!!!)

sudo rsync -avn --exclude='._*' --exclude='.Spotlight*' --exclude='.TemporaryItems*' --exclude='.Trashes*' --exclude='.fseventsd*' /Volumes/LabDrive/ /Volumes/LabDrive_Backup/Backups.backupdb/tau/Latest/LabDrive/ > ~/rsync_TM.txt &

Using this, I've found >5,000 files that have not been backed up correctly. As noted before, the "tmutil compare" seems to also identify the same missing files. However, rsync's output is more exhaustive, and will show not only new folders that have been added, but the files inside them as well. (tmutil compare will just show that a new folder has been added, and not list its contents).

I'm writing a python program to pull all available information about the un-backed-up files to look for a common thread.

I pulled all the available filesystem info (inode, perms, a/c/mtime, etc.) for both the missed and properly backed-up files. Long story short, after comparing the two sets of data I cannot find any difference between them.

N'thing the "take it to Apple" sentiment - Ian, have you opened a bug report yet? If not, I'll go ahead and open one.

G'day

Go ahead and open it - I'm on Snow Leopard which is out of support and Pondini doesn't expect I'll get much help. Better coming from a Lion user. But you might mention that I have a reproducible failure too that could also be used for diagnosis and/or for testing any candidate fixes.

Opened bug ID #14591014, and have already been contacted by apple engineering for more info. They asked me to run sudo tmdiagnose, which generated a zip archive containing lots of system info. I would offer to upload your versions of this file as well, but it contains a lot of private info.

O157-H7 wrote:

. . .

I would offer to upload your versions of this file as well

Ian is on Snow Leopard; the tmdiagnose command doesn't exist there; it was introduced with Lion.