Time Machine Not Backing Up New Files

G'day

I'm happy for SeattleMacUser. But screwed up disk partitions doesn't explain my issue. My disk partition output is below:

/dev/disk0

#: TYPE NAME SIZE IDENTIFIER

0: GUID_partition_scheme *250.1 GB disk0

1: EFI 209.7 MB disk0s1

2: Apple_HFS Mediterranean 249.7 GB disk0s2

/dev/disk1

#: TYPE NAME SIZE IDENTIFIER

0: FDisk_partition_scheme *1.0 TB disk1

1: Apple_HFS Pacific 1.0 TB disk1s1

/dev/disk2

#: TYPE NAME SIZE IDENTIFIER

0: GUID_partition_scheme *1.0 TB disk2

1: EFI 209.7 MB disk2s1

2: Apple_HFS Arctic 999.9 GB disk2s2

/dev/disk3

#: TYPE NAME SIZE IDENTIFIER

0: FDisk_partition_scheme *1.5 TB disk3

1: Apple_HFS Arabian 1.5 TB disk3s1

The disk I am backing up to with TM is Arctic, which looks to be in good shape.

I still have one problem for certain and probably two.

The first problem is TM not finding stuff that needs to be backed up. I currently have what I think is a work around for this. Each night before I do the final backup for the day I copy (with authentication) an empty folder into /Users. This causes the last modified data of the /Users folder to be updated causing a deep traversal on the following final backup. This backs up, or appears to back up, my work for the day. I often forget to do the folder copy and what was intended to be the final backup doesn't find much to do. This acts as a reminder so I then copy the folder as described and rerun the backup and it finds lots to do. So reliably TM is failing to find stuff that should be backed up. My work around only works for anything under /Users. Just checking the last modified dates for all the folders at the root of the startup disk the last modified date is the same in the latest TM backup as it is 'Now'. So prima facie it looks like /Users is my only issue. Tonight I might do a pre-final backup and then copy an empty folder into all the other folders at the root of the startup disk and see if TM finds anything extra to backup.

Now I say 'appears to backup my work for the day' above because I have also seen SeattleMacUers issue of the TM progress bar progressing smoothly through a large backup and then suddenly zipping to the end and claiming to be done. Without being on Lion and having TMUtil available I wasn't able to say, with reasonable effort, whether it was the progress bar reporting or files not being backed up that was at fault. From the elapsed time though I'm pretty sure it was the latter. I'm not creating and modifying as much stuff during a day as SeattleMacUser so the problem, if it still occurs, at my lower volume, is masked. Without TMUtil I can't say, with reasonable effort, whether its still occurring. I think it might be worth copying some large folders before the final nightly backup to see if I can reproduce the problem.

G'day

Looking back I think my post on .com.apple.timemachine.supported files was a red herring. I was also copying folders around at the time (so I could see easily what got backed up). I suspect I copied a folder into /Users each time I copied a .supported file and it was this, rather than the presence of the .supported folder, that caused files to be found. This would explain why files were still being found and backed up by TM after I removed the .support file.

Reformatting the TM drive to a GUID partition map has not worked for me either. I still have some files being backed up successfully, and some that are not. Here is my diskutil list, for reference.

/dev/disk0

#: TYPE NAME SIZE IDENTIFIER

0: GUID_partition_scheme *500.1 GB disk0

1: EFI 209.7 MB disk0s1

2: Apple_HFS Macintosh HD 499.2 GB disk0s2

3: Apple_Boot Recovery HD 650.0 MB disk0s3

/dev/disk1

#: TYPE NAME SIZE IDENTIFIER

0: Apple_partition_scheme *2.0 TB disk1

1: Apple_partition_map 32.3 KB disk1s1

2: Apple_HFS LabDrive 2.0 TB disk1s3

/dev/disk2

#: TYPE NAME SIZE IDENTIFIER

0: GUID_partition_scheme *2.0 TB disk2

1: EFI 209.7 MB disk2s1

2: Apple_HFS LabDrive_Backup 2.0 TB disk2s2

/dev/disk3

#: TYPE NAME SIZE IDENTIFIER

0: CD_partition_scheme *804.4 MB disk3

1: Apple_partition_scheme 700.4 MB disk3s0

2: Apple_partition_map 32.3 KB disk3s0s1

3: Apple_HFS WD SmartWare 320.7 MB disk3s0s2

G'day

I did one of my two proposed experiments this morning. I left the machine up overnight and TM did its usual hourly backups. First thing, before I did anything else, I copied an emtpy folder into each folder at the root of the boot disk - /Users, /System, /Library etc and called for a TM backup. If TM was doing its job properly overnight then, on the face of it, at most it should have found 9 zero-length directories to backup. It actually found 430-odd MB to backup. However two of the things at the root of the boot disk, and onto which I copied the empty folder, are encrypted disk images, which TM has to back up in their entirety. These are about 410 MB in total. I confirmed on the TM backup disk that it had in fact backed these images up. So that leaves about 20MB unaccounted for. What the test tells me is there wasn't a major backlog of unbacked up data lurking in these top level folders as there was, dating back over months, in /Users/Ian/Desktop. So the problem of not finding stuff that needs backing up appears to be confined to /Users. Reassuring.

The other problem, of not actually backing up things identified for backup, didn't occur today. TM found 430MB of stuff to backup and at leat 410MB of that was backed up. I'll look at running the big data test later in the day.

Hi,

I am experiencing the same problems (I think). Last week I installed a new hard drive. So I manually triggered a fresh TM backup to be sure it's up to date. Then I powered down the Mac, installed the hard drive, booted to recovery mode and reinstalled everything from the TM backup. After the boot-up everything looked fine until some days later. I missed recent files from my iPhoto library. In fact, every single photo from the last month was missing! The events were still there, but the actual files were missing. I activated TimeMachine and browsed back through the backups: The files obviously had never been part of the backup. Same situation when I opened the backupdb in the Finder and manually navigated to "Originals" inside the iPhoto library bundle.

I got an uneasy feeling, because if these files are missing, but others from the same time period are present, then the whole backup is unreliable! I still don't know what else is missing.

Luckily the photos where still present on the SD card inside the digital camera, so I reimported them. Today, I wanted to reinstall the HD because CoreData obviously prefers the rotating hard drive instead of the ssd for the FusionDrive, and it seems I need to change the order of disks in the createVolume command. Now I checked the backup BEFORE erasing my drive and... the newly imported files are missing AGAIN!

While browsing through the support forums I came to this discussion. Yes, maybe the exact date when TimeMachine stopped working was the update to 10.8.3! The trick of creating new folders all over the place now made TimeMachine finally back up the files I checked, but what else is missing? How can I know?

I don't dare to recreate my drive from the backup as long as this evil bug is not identified and fixed. I am using Macs for about 20 years now. I am an experienced software developer myself. This is the first time I actually used a full backup. And right before that, Apple managed to break the backup system. Bravo! Just in time! This is almost unbelievable.

I think, this was the successful backup attempt, after I created all those empty folders:

Jul 12 22:17:23 imac-10.fritz.box com.apple.backupd[17928]: Backing up to: /Volumes/Archivplatte/Backups.backupdb Jul 12 22:17:24 imac-10.fritz.box com.apple.backupd[17928]: Using file event preflight for Festplatte Jul 12 22:17:24 imac-10.fritz.box com.apple.backupd[17928]: Will copy (4.9 MB) from Festplatte Jul 12 22:17:24 imac-10.fritz.box com.apple.backupd[17928]: Found 46 files (4.9 MB) needing backup Jul 12 22:17:24 imac-10.fritz.box com.apple.backupd[17928]: 2.14 GB required (including padding), 382.7 GB available Jul 12 22:17:39 imac-10.fritz.box com.apple.backupd[17928]: Copied 1369 files (215.5 MB) from volume Festplatte. Jul 12 22:17:39 imac-10.fritz.box com.apple.backupd[17928]: Using file event preflight for Festplatte Jul 12 22:17:39 imac-10.fritz.box com.apple.backupd[17928]: Will copy (Zero KB) from Festplatte Jul 12 22:17:39 imac-10.fritz.box com.apple.backupd[17928]: Found 5 files (Zero KB) needing backup Jul 12 22:17:39 imac-10.fritz.box com.apple.backupd[17928]: 2.14 GB required (including padding), 382.48 GB available Jul 12 22:17:42 imac-10.fritz.box com.apple.backupd[17928]: Copied 154 files (93 bytes) from volume Festplatte. Jul 12 22:17:42 imac-10.fritz.box com.apple.backupd[17928]: Created new backup: 2013-07-12-221742

This looks like backupd found a couple of files it wasn't looking for. The tries before did not show any difference in "will copy" and "copied" and I did not observe any "hopping progress". As soon as I import the next fotos (probably tomorrow) I will post the result. Good night.

Message was edited by: orkuss

orkuss wrote:

. . .

I missed recent files from my iPhoto library. In fact, every single photo from the last month was missing!

If it's only the iPhoto library that's missing things, that's likely because the iPhoto app was open during the backups. Time Machine usually won't back up open files -- normally that's not a big problem, as most files aren't actually open for long -- an app will copy them to memory for editing, and the copy still on disk will get backed-up.

iPhoto, however, is different, apparently because of the complex structure of the files and folders inside it. See iPhoto '11: Back up your iPhoto library.

For some reason, Time Machine does not send any sort of message in that case; on the next backup it does send a message to the log that it missed it the last time, and will try again.

So, make sure to quit iPhoto before doing your backup. Then see if the files are missing.

Thanks for the hint. So you suggest iPhoto was continously open for the whole month? I made especially sure it wasn't open yesterday, when I read through this thread, manually triggering backup after backup. I am still on iPhoto 09 (V. 8.1.2), in case that makes any difference.

I am using a 2007 iMac with that unlucky Intel chipset, which recognizes only 3 GB of RAM, even when installing more. So I have to actually close applications. So I usually close iPhoto immediatly after importing my new photos. Sometimes I send an email or export them before quitting iPhoto. So I believe, there had been plenty of occasions for backupd to find my photos without iPhoto running.

I regret, but it does not seem to be that cheap.

How can I test the backup for completeness? rsync --dry-run?

orkuss wrote:

Thanks for the hint. So you suggest iPhoto was continously open for the whole month? I made especially sure it wasn't open yesterday, when I read through this thread, manually triggering backup after backup.

Ok, that eliminates that (some folks do seem to leave iPhoto open more or less indefinitely, or only back up while it's open).

I am still on iPhoto 09 (V. 8.1.2), in case that makes any difference.

Shouldn't be much, if any, difference.

How can I test the backup for completeness? rsync --dry-run?

You can use the tmutil command via Terminal to compare your internal HD to the most recent backup, but that's more than a bit tedious.

Usually, forcing TM to do a "deep traversal" will find and back up whavever's missing -- it compares everything on the most recent backup to your Mac, and backs-up anythinng that was missed. Takes somewhat longer than a normal backup, of course.

Kill two birds with one stone: Start up from your Recovery HD and use Disk Utilty to repair both your internal HD and the TM drive, just to be sure there aren't any directory problems. Then reboot normally and run a TM backup. That should trigger a "deep scan" or "deep traversal" message in the log. See if that gets the missing items.

I haven't tried to repair the disks and do a full backup yet, but I have imported a couple of new photos, quit iPhoto, triggered a backup and they are NOT included in the backup. Neither are they in the subsequent automatic backups.

So I believe even if a manage to trigger a full backup ONCE, the error still seems to be present.

And another strange issue I noticed: The Finder won't update the backups directory, I have to use the terminal to be able to see the most recent backups. This may be totally unrelated, but it's strange nevertheless.

G'day

As to the finder not updating the backup directory. If, when you say Finder, you mean Finder within TM, then there is a small delay between TM saying a backup is done and the backup becoming visible as a pane in the TM window. TM doesn't comment on this and unless you look at the time on the pane you won't be aware that you aren't looking at the latest backup. This would give the impression that the backups directory hasn't been updated.

If instead you mean that Finder directly then, under some circumstances, Finder can be a bit slow to recognise that the contents of a folder have changed if the window displaying the contents of the folder is open while the change is being made. The display of folder contents is sometimes not dynamic and you typically have to do something in the window to refresh it, such as unexpand and rexpand the folder in list view or close and reopen it in icon view. I think this occurs when the change to the folder contents is done other than through Finder e.g. through a program (such as TM).

I assume that after a while the update to the backupd directory does become visible in Finder otherwise you'd have a massive problem.

Indeed, I was talking about the finder directly. Of course I tried to navigate in and out of the backups-folder. I am using and programming Macs since the early Nineties. But it didn't update, even after a couple of hours. I had to do a restart. Then of course it updated. I left the window open since then, showing my TimeMachine drive, and it didn't update since then. So I used the terminal...

G'day

No offence intended - you get users of all levels of experience in the forums. Out of interest, while Finder is not showing the update that has taken place, what does the TM display show. I'm not sure which would be worse - TM and Finder agreeing but both disagreeing with terminal, or TM and terminal agreeing but both disagreeing with Finder. Either way this looks like a separate issue. That might be worth a thread on its own with some keywords that would help others with the same problem find it - the current thread has become quite large and someone coming to this thread with a hit on the lack of update symptoms would likely find the reference to the issue hard to find.

G'day

I think the next step in this is to involve Apple officially in the not-found-for-backup aspect of this. 1) Its a significant problem with the potential for loss of data (and if the problem is with a high level folder, as mine is/was, the potential for large scale loss of data (3.5GB in my case)). 2) I suspect the problem is more widespread than is appreciated for the reasons advanced earlier - you would only discover the problem under limited circumstances. 3) its cross release - occurs from Snow Leopard (at least) and forward so its not going away with OS changes.

I have a reproducible case that Apple could work with running TM in trace mode to find out what is going on. I'm on Snow Leopard but I suspect that won't matter.

I'm going away for a few days but I think I will raise this with Apple when I get back.

G'day

I wrote the first version of the utility to sweep the catalog and check whether each file is in the latest backup. The first, crude, version skips everything under /Volumes, /dev, /private/var, /Library/Logs, and anything with cache or Cache in the name. It also doesn't follow symbolic links (with thanks to Apache for the code for that).

Its quite quick - can check over 1M files in a few minutes despite being Java.

One result is that it appears that if a file has a last modified date of "9.30am 1st January 1970" then TM won't back it up. Since I live at GMT + 9.30 that time equates to midnight 1st January 1970 GMT, or the start of time according to Unix. So if the file has a last modified date of zero TM won't back it up. Bizarre. (I say 'appears' here because, while there lots of files with that date that aren't backed up, when I asked Finder to find all files with that last modified date it didn't find any, despite the fact that they clearly exist. So I can't say all such files won't be backed up because I have no way of finding

... (as I was saying before discovering Cmd-S sends the post rather than saving it) out how many there are. But of course I do - my new program. I found 67 files with a zero Last Modified date in my system of which 65 were not backed up. Most of these files would have existed when the first, full, TM backup was done which suggests that even a full TM backup is incomplete. These files occur in all sorts of places, but many of them are within applications. At least in principle the integrity of the backup of these applications is compromised and they include: (see below) ... (I say in principle because lots of application files never get used and you might be lucky.)

This is the summary of the run:

Checked 1,433,369 files

Found 5,385 files not in latest backup but should be

Found 34 files modified since latest backup

Found 2 files with zero last used date that were backed up

Found 65 files with zero last used date that were not backed up

The files that are not backed up are all over the place but the main directories are:

- Dropbox finder plugins

- Desktop

- Application Support for iDVD installed themes

- Application Support for NetBeans file history

- Application Support for Skype

- Mail files including Mailboxes

- Flash preferences

- PubSub feeds

- Safari local storage

- iTunes media automatically add files

- iPhoto faces detected

- iPhoto database versions

- iPhoto masters

- iPhoto previews

- iPhoto thumbnails

plus, with zero last used dates:

- Applications

- Adobe Bridge CS4

- Google Sketchup

- iWork

- Bento

- Eye TV

- Font Book

- Toast

- Frameworks

- Java

- iLife

- Python

So 5,000+ missing files out of 1.4 million is not a good story ! I suspect that some of the missing files are deliberately skipped by TM. But the ones off the desktop, which is my work and the most important, are definitely not to be excluded.

If anyone else wants to run the utility to compare results we can do that. There is some hard coding to be changed before it will run for someone else. If there is sufficient interest I can add a configuration file to overcome that need.