Time Machine seems to miss a LOT of files

I am aware of those exclusions and my program takes them into account (see list at the end of this post).

Right now the missing files are updates of NeoOffice, web pages I've saved in Firefox vie "Save Page as", 2 NeoOffice spreadsheets of mine that I've updated, VMWare virtual machines. I've observed other things beforehand, like PDF files that I filled in and Python programs I wrote.


current known exclusions:
pumacbook:~ root# tmcheck showex
info: comparing to /Volumes/TimeDisk 2/Backups.backupdb/pumacbook/Latest/Macintosh HD ( Sun Apr 11 16:45:32 2010 )
info: general exclusions
info: /.fseventsd (StdExclusions)
info: /.hotfiles.btree (StdExclusions)
info: /.spotlight-v100 (StdExclusions)
info: /.trashes (StdExclusions)
info: /.vol (StdExclusions)
info: /automount (StdExclusions)
info: /backups.backupdb (StdExclusions)
info: /cores (StdExclusions)
info: /desktop db (StdExclusions)
info: /desktop df (StdExclusions)
info: /dev (StdExclusions,mount)
info: /home (StdExclusions,mount)
info: /library/caches (StdExclusions)
info: /library/logs (StdExclusions)
info: /net (StdExclusions,mount)
info: /network (StdExclusions)
info: /network/servers (StdExclusions)
info: /previous systems (StdExclusions)
info: /private/network (StdExclusions)
info: /private/tftpboot (StdExclusions)
info: /private/tmp (StdExclusions)
info: /private/var/automount (StdExclusions)
info: /private/var/db/dhcpclient (StdExclusions)
info: /private/var/db/fseventsd (StdExclusions)
info: /private/var/db/spotlight (StdExclusions)
info: /private/var/db/spotlight-v100 (StdExclusions)
info: /private/var/folders (StdExclusions)
info: /private/var/log (StdExclusions)
info: /private/var/run (StdExclusions)
info: /private/var/spool/cups (StdExclusions)
info: /private/var/spool/fax (StdExclusions)
info: /private/var/spool/uucp (StdExclusions)
info: /private/var/tmp (StdExclusions)
info: /private/var/vm (StdExclusions)
info: /system/library/caches (StdExclusions)
info: /system/library/extensions/caches (StdExclusions)
info: /tmp (StdExclusions)
info: /users/guest (StdExclusions)
info: /users/ly/music/itunes/album artwork/cache (mdfind)
info: /users/ly/music/itunes/itunes music library.xml (mdfind)
info: /users/pu/library/calendars/calendar cache (TimeMachine)
info: /users/pu/library/itunes/ipod software updates (mdfind)
info: /users/pu/library/mail/envelope index (TimeMachine)
info: /users/pu/library/mail/metadata/backingstoreupdatejournal (TimeMachine)
info: /users/pu/library/preferences/byhost/com.apple.dotmacsync.0019e349c05e.plist (TimeMachine)
info: /users/pu/music/itunes/album artwork/cache (mdfind)
info: /users/pu/music/itunes/album artwork/local (mdfind)
info: /users/pu/music/itunes/itunes music library.xml (mdfind)
info: /users/pu/pictures/iphoto library/ipod photo cache (mdfind)
info: /users/shared/sc info (StdExclusions)
info: /users/sy/music/itunes/album artwork/cache (mdfind)
info: /users/sy/music/itunes/itunes music library.xml (mdfind)
info: /volumes (StdExclusions)
info: /volumes/oldtimedisk (mount)
info: /volumes/timedisk 2 (mount)
info: /volumes/uselpa (mount)
info: user exclusions
info: .bash_history (tmcheck)
info: .trash (tmcheck)
info: library/application support/syncservices/data.version (StdExclusions)
info: library/caches (StdExclusions)
info: library/logs (StdExclusions)
info: library/mail/availablefeeds (StdExclusions)
info: library/mail/envelope index (StdExclusions)
info: library/mirrors (StdExclusions)
info: library/preferences/com.apple.recentitems.plist (tmcheck)
info: library/pubsub/database (StdExclusions)
info: library/pubsub/downloads (StdExclusions)
info: library/pubsub/feeds (StdExclusions)
info: library/safari/historyindex.sk (StdExclusions)
info: library/safari/icons.db (StdExclusions)
info: library/safari/webpageicons.db (StdExclusions)

Message was edited by: uselpa

Pondini wrote:

uselpa wrote:
. . .

I don't use NeoOffice, so can't comment on it.

All I can say that TM picks those files up, generally speaking. But not always.

I'm not sure why you'd want TM to back up vm files (most users exclude them, and TM often chokes on them). I'm sure you know, if it's on a MBR volume, TM won't back it up.

What do you mean by "on a MBR volume"? It's VMWare Fusion, and the files are in my /Users/pu, on the boot volume. I don't suppose that TM looks inside the VMWare files, so I don't see where MBR comes in here, or am I missing something?

One possibility: for some users on Snow Leopard, if you do a +*Verify DIsk+* on your boot volume, or alter any partitions on your boot drive, using the version of Disk Utility on your boot volume, TM will stop finding changes. Backups run, but nothing gets saved. A Restart will fix it. Apple knows of this, but it wasn't fixed in 10.6.3.

Yes, I've run across that one, but I've rebooted since the last check. I actually also did fsck -fy on both the boot volume (from the install disk) and the TM volume. Nothing got fixed.

Another possibility is, something corrupted in a folder (the folder object itself, apparently). We've seen a few cases of this, where nothing inside the affected folder gets backed-up.

I have the impression that fseventd misses events in certain circumstances, which scares me slightly since other software depends on that mechanism as well. Would you happen to be aware of such problems?

Regards,
-Patrick

For the things that got missed -- boot up from your Install disc (you don't have to do anything there), then reboot normally. That should force a "deep traversal" on the next backup. See if the missing items get backed-up.

I did that, but that didn't change anything.

Now my program has a 'force' option which changes the access and modified times of the missed files. I ran that, and then ran a TM backup again. Et voilà, the number of missed files is down to 0 (from 450).

Now this tells me (please correct me if I'm mistaken) that those files were meant to be backed up by TM, because otherwise they would have been ignored during the last run as well. So to me it's either a bug in fseventd, or in TM itself.

I'd really love some experienced people to run the same tests on their machines, and see whether this is a general problem or not.

Regards,
-Patrick

I don't think so:

pu@pumacbook:/var/log$ grep com.apple.backupd system.log | wc -l
740
pu@pumacbook:/var/log$ grep -i deep system.log | wc -l
0
pu@pumacbook:/var/log$ grep -i traversal system.log | wc -l
0

Yes, I have it installed, but the result is the same. No mentioning of a deep traversal.

Is there any other known way to trigger a deep traversal?

infty wrote:
Can you release a fixed version that looks for both?

I've uploaded a small change that should work for you. Can you try it please?