Time Machine Not Backing Up New Files

O157-H7 wrote:

I've already tried swapping in an empty TM target drive. That worked, but then it went back to not backing up some new files and changes.

Then those rsync scripts are likely the cause. Rsync is not a "few lines of perl". It is 3.2 MB of C source. It is wrtten by GNU and they are not concerned with either Mac or Time Machine compatibility. What it sounds like is that the OS detects changes in those directories but when it goes to check, discovers that their timestamps are older than those in Time Machine, so it ignores them. Once that happens, they aren't ever going to get backed up until you blow away your entire Time Machine drive. If my theory is correct, then deleting those .fseventsd file would not be sufficient to correct the problem.

I would not recommend running rsync with Time Machine. I suggest you find a new way to perform your network backups. A few lines of Perl is probably a good idea, as long as rsync isn't involved. You can keep searching for an answer that allows you to blame Mountain Lion if you want. I'm satisfied I have solved your problem, but only SeattleMacUser can definitively say for sure.

Then those rsync scripts are likely the cause. Rsync is not a "few lines of perl". It is 3.2 MB of C source. It is wrtten by GNU and they are not concerned with either Mac or Time Machine compatibility. What it sounds like is that the OS detects changes in those directories but when it goes to check, discovers that their timestamps are older than those in Time Machine, so it ignores them. Once that happens, they aren't ever going to get backed up until you blow away your entire Time Machine drive. If my theory is correct, then deleting those .fseventsd file would not be sufficient to correct the problem.

I would not recommend running rsync with Time Machine. I suggest you find a new way to perform your network backups. A few lines of Perl is probably a good idea, as long as rsync isn't involved. You can keep searching for an answer that allows you to blame Mountain Lion if you want. I'm satisfied I have solved your problem, but only SeattleMacUser can definitively say for sure.

You'd be wrong. Look, I realize that many of the people that come here looking for help are not really familiar with CLIs or the complex underpinnings of OS X. As should be clear from my posts in this thread, I am not that kind of user. The timestamp issue was literally the first thing that ocurred to me, so I modified both the file creation and modification dates, and they are not the responsible for the problem.

You also misunderstand my statement. My backup scripts are a few lines of perl that call rsync, which send any changes on the shared drive to an offsite server. But this is besides the point. The offsite rsyncs are not the issue. As I said in the previous post, TM worked fine before. There is some change in the time machine implementation in 10.8, or some other change in the filesystem that is causing the problem.

Please, read carefully what I have already posted in the thread.

When I get back, I'll in all likelyhood try and resurrect the old 10.5 machine and do a side-by-side comparison. Hopefully that will provide some clues as to what's going on.

G'day

1. I had in mind to develop the utility that I spoke of for checking that files are backed in TM and put it in the App store.But from Lion on Java isn't installed on Macs by default so lots of people won't now have it. I'm not about to learn Objective C just for one small utility.

So there is less value for me to make it. If you are serious about making it then maybe these thoughts would help. I was thinking about 3 threads (possibly literally) in the utility. The first would: run continuously; sleep till it finds a new TM backup; get the list of recently changed files; divide them into 60 lumps and process one lump a minute (to limit system impact), ensuring that each file was in the latest TM backup; any file that isn't would have to be checked to ensure it hadn't been deleted, isn't newer than the backup, is otherwise within the scope of TM etc before being brought to the users attention; in bringing to the users attention you'd maybe want to filter the list - no point in listing 1000 files from the same directory. The second thread would: iterate from 0 upwards; get the list of files modified in that many days prior to today; remove from the list any files already processed; then do as thread 1; at some point it would have to set the last modified date to infinity to get all files so it can eventually terminate; this thread would run once and probably never run again execpt on request. The third thread would do as the second but would only look at files if the (hash of) the list of files modified (on or since) day n differs from that found by thread 2 or as updated by thread 3 earlier; on exhausting the list of hashes thread 2 would start over.

The purpose of thread 1 is to give early warning that recently changed files haven't been backed up by TM. The purpose of thread 2 is to do a full system sweep looking for files not in the backup. The purpose of thread 3 is to check files that are added to the system with a back-dated last modified date, as happens with software installs, untars etc. One obvious gotcha - thread 3 has to make sure it compares against the right hash from thread 2 since n days ago is a relative term. Thread 2 will also need to replace hashes it found to differ.

2. I thought you were a bit hard on O157. I found his comments relevant to the discussion and helpful.

3. For O157. Its still possible that the rsync is the problem. That you still have the TM problem after stopping rsyncs isn't diagnostic. If rysncs run in the past caused some kind of folder damage it may be that that damage, which persists though you have stopped running rsyncs, is causing the TM problem. The fact that newly added files are not being backed up may simply mean they were added to folders already damaged by rsync. Folder damage certainly looks like the or a trigger for the TM problem both from what I am seeing and from Pondinis contributions on how to fix the TM problem. (One thing you might do is: find a file that was recently added that TM didn't backup; put another new file in the same folder; if TM doesn't back that up then folder damage looks increasingly likely as a cause; if TM does back it up its something else.)

G'day

Well there is good news and bad news.

I recreated my desktop folder. I was making the operation more complex than it needed to be in my thinking above. In the end I: did a full (non TM) backup; created a folder /Users/Ian/D2; in terminal moved each folder and file individually from /Users/Ian/Desktop to D2 (there were only about 15, being a tidy person); logged out of the Mac; logged into the Mac as Fred; in Fred logged into terminal as Ian; deleted /Users/Ian/Desktop with sudo; logged into the Mac as Ian, thereby creating a new Destop; logged into terminal as Ian; individually moved each file and folder from D2 back to the new Desktop; logged out as Ian; logged back into the Mac as Ian; and all seems to be well in terms of using the Mac.

I then put a new folder, lets call it B, into A (the folder on the Desktop whose new contents were not being backed up by TM) and a new folder C on the desktop. I then did a TM backup. Neither B nor C was backed up. So no change to behaviour. So replacing the Desktop folder hasn't fixed anything.

But in my work in terminal I spotted a file called .com.apple.timemachine.supported. So I put a copy of that in /Users/Ian/Desktop and did another TM backup. No change.

Then I put a copy of .com.apple.timemachine.supported into /Users/Ian and did another backup. This time B and C were backed up. So maybe a missing .com.apple.timemachine.supported file in /Users/Ian was the cause of the problem of files in that directory and below not being considered by TM as needing backup. I guess this is 'folder damage' in a sense.

Just to be sure I then put a new folder called D on the desktop and removed the above com file from /Users/Ian and did another TM backup. I was expecting D not to be backed up. But it was. I did a bit more experimenting with com files in /Users, /Users/Ian, and /Users/Ian/Desktop. It begins to look at though, once a directory has had a com file in it TM remembers that fact and the com file doesn't actually need to be there any more. Or maybe something else I did while testing caused TM to backup folders it was previously ignoring.

So I think I am warm, but not quite there yet. I will think about this some more, let thing settle, and dream up some more experiments.

This discovery intrigued me, so I thought I'd give it a try via ssh. I've seen those hidden files from time to time, but haven't paid them much mind. I'm curious who is the owning user of your .supported files, and what are the permissions on the files?

Just to test, I touched a .com.apple.timemachine.supported file in a directory containing stuck items, and for good measure in every enclosing directory above it. (e.g. a .com.apple.timemachine.supported in /Volumes/LabDrive/users/username/Projects/escrt/testing, /Volumes/LabDrive/users/username/Projects/escrt, /Volumes/LabDrive/users/username/Projects, etc. etc.)

I triggered a manual TM backup after the creation of each .supported file, but to no avail. Oh well. I too will continue to ruminate on potential experiments.

G'day

Bummer. The .supported files look like this in terminal:

-rw-r--r--@ 1 Ian staff 0 4 Jul 10:47 .com.apple.timemachine.supported

where Ian is the owner of the directory.

Please post the output:

{ diskutil list; echo; diskutil cs list; } | open -f -a TextEdit

The partition table on the backup drive is totally screwed up. It needs to be GUID and without all that weird stuff. Back up to another drive, then repartition the 1 TB drive with the default options.

You should be OK now. For the record, this thread reveals a horrendous bug in Time Machine.

Excellent work SeattleMacUser! So just to recap, it would appear that the problem may be related to the use of the older Apple Partition Map format on the TM drive. I am currently backing up to a drive I've reformatted to a GUID partition scheme, and I'll see if this fixes my problems as well.