Skip navigation

Time Machine NOT backing up all new files

1006 Views 15 Replies Latest reply: Nov 18, 2012 1:35 PM by Pondini RSS
1 2 Previous Next
raka Level 1 Level 1 (5 points)
Currently Being Moderated
Jan 1, 2012 6:18 PM

I have 3 internal disks in my Mac Pro: SYSTEM (System and Apps), DATA (all my user data), and TM (Time Machine)

 

i use TM in manual mode to backup my DATA disk, typically every time i copy new images from my camera to my DATA disk.

 

i notice that sometime the latest files i copied to my machine are NOT backed bup by TM.

 

Every time i run TM, it finishes really quick, i.e. seeing nothing new to back up.

 

I then created a simple text file, and invoked TM again - and this very latest file was backed up - but my images from the past few days are STILL not backed up (and likely never to be, it seems).

 

Other things i;ve tried (w/ no success):

 

- ran Disk Utility verify on all disks

- rebuilt Directories w/ Disk Warrior

- tried (in vain) to force a TM Deep Traversal by:

   a- restarting,

   b- starting up using my install CD

   c- restarting in Safe Mode

 

 

ps: here is an archived thread i found about this, but no solution mentioned

 

https://discussions.apple.com/message/11359191#11359191

 

help!

 

thanks!

Mac Pro, Mac OS X (10.6.8)
  • Pondini Level 8 Level 8 (38,710 points)
    Currently Being Moderated
    Jan 1, 2012 6:21 PM (in response to raka)
  • Pondini Level 8 Level 8 (38,710 points)
    Currently Being Moderated
    Jan 2, 2012 10:16 AM (in response to raka)

    raka wrote:

    . . .

    in all cases this caused the previosuly un-backed up items to be found & backed up (along w/ the new dummy folder)

     

    now here is where it gets weird: i then DELETED these dummy test files & folders & did another TM backup. TM did NOT remove these test folders from the new backup it created.

     

    at this point i guess i need to start all over w/ a new/ fresh TM disk?

    I doubt that will help.  It sounds more like something in your installation of OSX is damaged.  You might want to try installing the "combo" update, per  Installing the ''combo'' update and/or Reinstalling OSX

     

    If that doesn't help, yoiu might want to install a fresh copy of Snow Leopard from your Install disk, per the same article.

     

    ps: not sure if this is relevant - my TM disk icon changed from generic to a tm specific icon when i first told TM about it, but after a restart & subsequent use, it changes back to the generic icon again. (i only use TM in manual mode)

    Yes, that does seem to happen on occasion when you change disks or have TM off.  I've never tried to pin down exactly what the circumstances are, though.

     

    May I ask why you don't let it do it's hourly updates?  Especially since you're backing-up to one of your internal HDs, it should be very fast and unobtrusive.

  • Pondini Level 8 Level 8 (38,710 points)
    Currently Being Moderated
    Jan 2, 2012 1:04 PM (in response to raka)

    raka wrote:

    . . .

    i have Lightroom running most of the time - and i've hear that there is a possibility of corruption when TM does a  backup while thwe LR catalog is open. Also, since the catalog is a single file (and 1GB in size) and even the smallest change will cause TM to keep making a new backup of it - so i tend to invoke TM only after i make a lot of changes, or just after i imported a lot of images from my camera.

     

    i'm also excluding teh LR preview file (50GB) for the same reason - plus it can easily be reconstructed/ rebuilt if needed.

    You might try excluding it from Time Machine, and backing-it up separately, with a different app, such as CarbonCopyCloner, perhaps once a day, to a separate partition on the TM drive.

     

    i think i'll try starting a new TM disk (a bit less severe than re-installing OS X).

    Reinstalling isn't severe.  It just gets you a fresh copy of OSX, without disturbing anything else.

  • Ian Blavins Calculating status...
    Currently Being Moderated
    Nov 16, 2012 1:24 AM (in response to raka)

    G'day

     

    I'm having the same problem. I also have three disks - internal, external large capacity, and time machine. The latter two are excluded from TM backups. I am doing software development. When I reach a recognisable state I copy the (Netbeans) development folder into my Product Versions folder, rename the copy with the next version number, and ask time machine to do a backup. I do this maybe from 2 to 10 times a day depending on the rate of development progress. When I have more than about 20 versions I move the oldest to the external large capacity drive. Recently I have noticed that the Product Versions folder in TM does not contain all the version folders that are on the internal disk. I have asked TM to make a new backup a couple of times and it continues to ignore the version folders it doesn't have.

     

    One thing that may be confusing TM is that the copies of the development folder in the product versions folders all have the same last-modified date. Presumably this is because during software development I am changing the contents of files and folders in the development folder, but not the structure of the folder itself, so its last-modified date isn't being changed.

     

    I have done what was done above - create a text file, put it in a new folder, and copy the folder into the version folders that are not being backed up. I copied into even numbered version folders expecting those to be backed up leaving odd-numbered ones, where I didn't copy the folder, not backed up. No such luck. Even though finder clearly showed a changed last-modified date on the folders on the internal drive into which I did the copy they still weren't backed up. Nor were the odd-numbered ones.

     

    Just to add some spice to this, I run a two TM disk backup scheme. I use one TM disk for about a day and then swap it out for the othe disk. I do this about dinner time since the first backup to the newly swapped in drive is likely to be a large one. I just did a swap. Not only did the some of the product version folders not get copied, there were more that weren't copied than with the other disk.

     

    And yes, before doing any of this, I ran the steps of #D5 referenced above.

     

    This is all on a stable 10.6.8 environment.

  • Pondini Level 8 Level 8 (38,710 points)
    Currently Being Moderated
    Nov 16, 2012 8:25 AM (in response to Ian Blavins)

    Ian Blavins wrote:

    . . .

    When I reach a recognisable state I copy the (Netbeans) development folder into my Product Versions folder, rename the copy with the next version number, and ask time machine to do a backup.

    . . .

    Recently I have noticed that the Product Versions folder in TM does not contain all the version folders that are on the internal disk. I have asked TM to make a new backup a couple of times and it continues to ignore the version folders it doesn't have.

    I'm not familiar with Netbeans.  It's possible that (like Apple's Xcode) it puts an extended attribute on some such items to exclude them from backups.   See the yellow box in Time Machine - Frequently Asked Question #11 for a way to tell if that's the situation.

     

     

    One thing that may be confusing TM is that the copies of the development folder in the product versions folders all have the same last-modified date. Presumably this is because during software development I am changing the contents of files and folders in the development folder, but not the structure of the folder itself, so its last-modified date isn't being changed.

    That shouldn't matter.  Ordinarily, Time Machine uses the File System Event Store, a log of changes to the file system, to determine which folders have been changed and have contents that need to be backed-up. 

     

    I have done what was done above - create a text file, put it in a new folder, and copy the folder into the version folders that are not being backed up.

    Did you make the files with a standard app (that won't be marking things to be excluded)?

     

     

    Just to add some spice to this, I run a two TM disk backup scheme. I use one TM disk for about a day and then swap it out for the othe disk. I do this about dinner time since the first backup to the newly swapped in drive is likely to be a large one. I just did a swap. Not only did the some of the product version folders not get copied, there were more that weren't copied than with the other disk.

    Each set of backups should be independent;  when a backup is run, Time Machine should back up the changes since the previous backup to that particular volume.

     

     

    I'm not sure about your folder structure or what is, and is not, getting backed-up:  is it possible that no changes to particular folders have been backed-up recently, or are some backed-up and others missed? 

     

    If no changes to a particular folder are getting backed-up, and you don't find the extended attributes above, that folder may be damaged.  That's rare, and we don't know what the actual damage is, but it does happen.  If so, the only known fix is to create a new folder, copy the contents of the damaged one into it, then delete the damaged one.

     

    Another thing you might try is starting from your Snow Leopard Install disc and repairing your internal HD.  Whether it needs it or not, that will usually trigger a "deep traversal" on the next backup, where Time Machine doesn't use the File System Event Store, but compares everything on your system to the most recent completed backup.  (You'll see "scanning nnnn items" messages on the Time Machine Preferences window.) 

    Unless the extended attribute is set to ignore the folder, that should "catch up" with all the changes.

  • Ian Blavins Level 1 Level 1 (5 points)
    Currently Being Moderated
    Nov 16, 2012 12:31 PM (in response to Pondini)

    G'day

     

    I'm not familiar with Netbeans.  It's possible that (like Apple's Xcode) it puts an extended attribute on some such items to exclude them from backups.   See the yellow box in Time Machine - Frequently Asked Question #11 for a way to tell if that's the situation.

     

    Its not the situation. And this makes sense because NetBeans is an IDE and that last thing you would want to do is exclude program development files from backup. Also many of the version folders are getting backed up.

     

    Did you make the files with a standard app (that won't be marking things to be excluded)?

     

    Yes, TextEdit to make the file and Finder to make the enclosing folder.

     

    If no changes to a particular folder are getting backed-up, and you don't find the extended attributes above, that folder may be damaged.  That's rare, and we don't know what the actual damage is, but it does happen.  If so, the only known fix is to create a new folder, copy the contents of the damaged one into it, then delete the damaged one.

     

    This looked like a possibility for a while. On closer examination of the backups on the Arctic backup drive it looks like versions 311 and onwards are not being backed up. Each version folder is created by copying the same development folder. If that development folder was damaged between the v310 and v311 copy, and the damage is passed on in folder copy, then we would see the behaviour we see. It would also explain how I've been able to make 310 version folders without, apparently, having version folders being ignored by TM. (I say apparently because, while I have good written records of what version folders where made when and therefore know what version folders should be on each TM backup, I can't actually check because TM throws away yesterday's intra-day backups each day.)

     

    On the Southern backup drive (remember the dual TM drive strategy) we see the same kind of behaviour so this theory is looking good. But there is a flaw. When backing up to Southern TM has copied v311 through v315. However it ignores v316 through v318. If these folders have inherited damage from the development folder I suppose results could be a bit unpredictable, with folders being backed up to Arctic but not to Southern. But the theory looks a lot less promising.

     

    To further weakn the damage theory: I asked TM to make a backup just before I shut down last night. It ignored version folders v311 and onwards, as it had been doing. This morning, with the machine being shut down all night, TM has found nearly a GB of files to backup. They certainly didn't change overnight. And yes, the GB that it found to backup include v311 to v318. (There are comlicating factors, including that I moved everything prior to v311 to the big capacity external disk just before shutdown last night. That change may be what has triggered TM to rethink whether it should backup v311 to v318 this morning.)

     

    Another thing you might try is starting from your Snow Leopard Install disc and repairingyour internal HD.  Whether it needs it or not, that will usually trigger a "deep traversal" on the next backup, where Time Machine doesn't use the File System Event Store, but compares everything on your system to the most recent completed backup.  (You'll see "scanning nnnn items" messages on the Time Machine Preferences window.)

    Unless the extended attribute is set to ignore the folder, that should "catch up" with all the changes.

     

    Understood. I did one of these recently but I'll schedule one for tonight. I can't risk losing time to a large backup this morning.

  • Pondini Level 8 Level 8 (38,710 points)
    Currently Being Moderated
    Nov 16, 2012 12:58 PM (in response to Ian Blavins)

    Ian Blavins wrote:

    . . .

    Its not the situation. And this makes sense because NetBeans is an IDE and that last thing you would want to do is exclude program development files from backup.

    Xcode is an IDE, too.  It automatically excludes its "build" folders.

     

    Also many of the version folders are getting backed up.

    It's still worthwhile to take a few moments to check, via the yellow box in Time Machine - Frequently Asked Question #11, just to rule that out.

     

    On closer examination of the backups on the Arctic backup drive it looks like versions 311 and onwards are not being backed up. Each version folder is created by copying the same development folder. If that development folder was damaged between the v310 and v311 copy, and the damage is passed on in folder copy, then we would see the behaviour we see.

    Yes, that's a possibility.  Try making/copying a different folder.

     

    (I say apparently because, while I have good written records of what version folders where made when and therefore know what version folders should be on each TM backup, I can't actually check because TM throws away yesterday's intra-day backups each day.)

    True, but the folders should appear on subsequent backups (unless the originals were moved or deleted in the meantime).  In other words, anything backed-up on Monday will still appear on the first Tuesday backup, even after all the Monday backups are deleted.  You just can't tell which backup first backed them up.

     

    How are you determining what's backed-up?  Are you using TimeTracker or BackupLoupe, per #A2 of Time Machine - Troubleshooting?

     

    On the Southern backup drive (remember the dual TM drive strategy) we see the same kind of behaviour so this theory is looking good. But there is a flaw. When backing up to Southern TM has copied v311 through v315. However it ignores v316 through v318. If these folders have inherited damage from the development folder I suppose results could be a bit unpredictable, with folders being backed up to Arctic but not to Southern. But the theory looks a lot less promising.

    Without knowing the dates & timing, I agree it doesn't seem consistent.  If you're getting different results at nearly the same time on the two drives, that may not be the problem.  But it may be worth making some new ones to determine what's actually happening.

     

    You might want to repair both TM drives, just to be sure, per #A5 in Time Machine - Troubleshooting.  While you're at it, you might as well verify your internal HD, just to be sure there's not some sort of odd directory problem there.

     

     

    (There are comlicating factors, including that I moved everything prior to v311 to the big capacity external disk just before shutdown last night. That change may be what has triggered TM to rethink whether it should backup v311 to v318 this morning.)

    If you move or rename a folder, it will be backed-up in it's entirety, including all contents.   But that wouldn't affect other folders that had no changes.

  • Ian Blavins Level 1 Level 1 (5 points)
    Currently Being Moderated
    Nov 16, 2012 1:22 PM (in response to Pondini)

    G'day

     

    Xcode is an IDE, too.  It automatically excludes its "build" folders.

     

    It's still worthwhile to take a few moments to check, via the yellow box in Time Machine - Frequently Asked Question #11, just to rule that out.

     

    Sorry, lack of clarity on my part. I did make the check first time around. The files are not excluded.

     

    Yes, that's a possibility.  Try making/copying a different folder.

     

    Because this morming TM decided to copy the folders it was ignoring yesterday it is now up to date. I need to re-establish the errant behaviour i.e. wait until TM ignores a new version or two. Then I'll make a new folder and copy the contents of the IDE development folder into it. If folder damage is the issue then from then on new versions made will be copied by TM. I predict in fact this won't make a difference because I don't reckon folder damage is the issue. (But of course now that it is being watched TM will behave faultlessly and won't re-estabish the errant behaviour.)

     

    True, but the folders should appear on subsequent backups (unless the originals were moved or deleted in the meantime).  In other words, anything backed-up on Monday will still appear on the first Tuesday backup, even after all the Monday backups are deleted.  You just can't tell which backup first backed them up.

     

    Indeed. I can do a bit more analysis there. I will schedule that for tonight too.

     

    How are you determining what's backed-up?  Are you using TimeTracker or BackupLoupe,per #A2 of Time Machine - Troubleshooting?

     

    No. I simply open the Product Versions folder (that contains the copies of the developmet folder) on the desktop and then go into TM. There I step backwards through the backups. TM shows me what folders it has in in the Product Versions folder of each backup.

     

    Without knowing the dates & timing, I agree it doesn't seem consistent.  If you're getting different results at nearly the same time on the two drives, that may not be the problem.  But it may be worth making some new ones to determine what's actually happening.

     

    Not that I know I have to keep an eye on this I will be watching much more closely as I make versions during the day. To push the process along I will make versions, and associated TM backups, more frequently than normal and check each TM backup.

     

    You might want to repair both TM drives, just to be sure, per #A5 in Time Machine - Troubleshooting.  While you're at it, you might as well verify your internal HD, just to be sure there's not some sort of odd directory problem there.

     

    Understood. I'll start by re-establishing the errant behaviour which I'll do via the accelerated process of development and versioning described above. You must be coming up to quitting time for the week. By the time you come in Monday morning I should have more to report. (I'm in Australia and I presume you are in the US.)

  • Pondini Level 8 Level 8 (38,710 points)
    Currently Being Moderated
    Nov 16, 2012 1:57 PM (in response to Ian Blavins)

    Ian Blavins wrote:

    . . .

    I predict in fact this won't make a difference because I don't reckon folder damage is the issue.

    Probably true.

     

     

    (But of course now that it is being watched TM will behave faultlessly and won't re-estabish the errant behaviour.)

    No doubt. 

    You must be coming up to quitting time for the week. By the time you come in Monday morning I should have more to report. (I'm in Australia and I presume you are in the US.)

    Yeah, I'm in Florida.  But not at work (I gave that up years ago ).

     

    Keep us posted . . .

  • Ian Blavins Level 1 Level 1 (5 points)
    Currently Being Moderated
    Nov 16, 2012 2:03 PM (in response to Pondini)

    G'day

     

    Yeah, I'm in Florida.  But not at work (I gave that up years ago ).

     

    Me too (the 'gave that up' bit, not the 'Florida' bit). Makes you wonder why I'm developing a cloud based load testing system and you're accumulating a vast number of support points.

     

    Keep us posted . . .

     

    Will do.

  • Pondini Level 8 Level 8 (38,710 points)
    Currently Being Moderated
    Nov 16, 2012 2:05 PM (in response to Ian Blavins)

    Ian Blavins wrote:

    . . .

    Makes you wonder why . . . you're accumulating a vast number of support points.

    Keeps me off the streets!

  • Ian Blavins Level 1 Level 1 (5 points)
    Currently Being Moderated
    Nov 18, 2012 12:12 PM (in response to Pondini)

    G'day

     

    As expected, when under observation, TM has behaved like a lady. Can't reproduce the errant behaviour so no point in applying the various potential remedies.

     

    I think at this point that the Mac briefly got confused vis a vis the File System Event Store, decided nothing had changed, and didn't back anything up. Sounds unlikely but that is the simplest explanation for the observed behaviour (which in many years of technical debugging has a strong track record of being the right one).

     

    It would explain why:

     

    - TM stopped backing version folders up to the Arctic drive

    - TM also stopped backing up version folders to the Southern drive

    - TM copied some folders to one drive but wouldn't copy them later to the other - the problem occurred after v315 was made and backed up to Southern but before Arctic, which was up to v310, was reintroduced

    - when I created a little text file put it in a folder, and dropped the folder into some of the not backed up version folders TM still wouldn't backup the folders

     

    A reboot cleared the problem or at least the problem went away after a reboot. TM then brought both disks up to date at the first opportunity and has behaved properly ever since. The only think that puzzles me is that the reboot that cleared the problem was not the first since the problem occurred. My guess is it will have something to do with the structure of the File System Event Store and how it is used.

     

    I'll keep an eye on it but I think this was just a temporary glitch that is now fixed.

     

    Thanks for your help.

1 2 Previous Next

Actions

More Like This

  • Retrieving data ...

Bookmarked By (0)

Legend

  • This solved my question - 10 points
  • This helped me - 5 points
This site contains user submitted content, comments and opinions and is for informational purposes only. Apple disclaims any and all liability for the acts, omissions and conduct of any third parties in connection with or related to your use of the site. All postings and use of the content on this site are subject to the Apple Support Communities Terms of Use.