Skip navigation

Time Machine always requires deep traversal?

13578 Views 174 Replies Latest reply: Feb 18, 2014 8:41 AM by bernuli RSS
  • bernuli Level 1 Level 1 (50 points)
    Currently Being Moderated
    Oct 12, 2012 3:24 PM (in response to Pondini)

    I don't think it is the directory repair coming from booting in safe mode.

     

    It does not happen every time, but if .fseventsd has a huge number of files in it, I have seen safe mode resulting in a clean out.  This is after a running disk utility and verify reporting no problems.

     

    Or perhaps safe mode allows the fseventsd process to properly take care of it's directory.

     

     

    B

  • Pondini Level 8 Level 8 (38,710 points)

    Markus Altendorff wrote:

    . . .

    Just to add some idle speculation, maybe these old log files should be cleared out periodically by some of those maintenance scripts that the system is supposed to run every day/week/month, and something broke in there yet nobody noticed...? Then again, I've not seen these in a while, does Apple still use them?

    According to this ArsTechnical article, the Event Store is not purged periodically, but that doesn't seem reasonable.  It's 4 years old, though, and there are some inaccuracies in some of their other articles.  

     

    In some cases, such as power failures or system crashes, when the store is determined to be inaccurate, it will be replaced entirely, with a message in the system.log.  The "Event Store UUIDs don't match" message sent by Time Machine means it's detected a new Event Store.  It's followed by a deep traversal/deep scan, because it can't trust the Event Store.

  • Pondini Level 8 Level 8 (38,710 points)
    Currently Being Moderated
    Oct 12, 2012 3:55 PM (in response to bernuli)

    bernuli wrote:

     

    I don't think it is the directory repair coming from booting in safe mode.

    BRumble posted "safe user mode."

     

    My question was, did he/she mean Single User Mode or Safe Mode?  

     

    If we can determine that simply booting into Safe Mode is a reliable fix, that's something much safer and more reliable to recommend to the casual user than booting into Single User mode and running UNIX commands.

  • bernuli Level 1 Level 1 (50 points)
    Currently Being Moderated
    Oct 12, 2012 4:08 PM (in response to Pondini)

    my guess is BRumble meant "safe mode". 

     

    The safe mode trick works when you have months worth of files in the .fseventsd directory.

     

    I agree better to boot in safe mode occasionally, is the best solution.  But if you want the best TM performance possible, occasionally blowing away /.fseventsd works well.  Just make sure you are backed up first, cause a small typo in single user mode could be the start of a very bad day.

     

     

    B

  • Pondini Level 8 Level 8 (38,710 points)
    Currently Being Moderated
    Oct 12, 2012 5:09 PM (in response to bernuli)

    bernuli wrote:

    . . .

    my guess is BRumble meant "safe mode".

    That's my guess, too, but I'd like BRumble to clarify.

     

    But if you want the best TM performance possible, occasionally blowing away /.fseventsd works well.

    Unless the Event Store is damaged, that really shouldn't make any difference.  Time Machine keeps track of the last "event" it processed, so only looks in the Event Store for the directories that have been changed since the previous backup. 

     

    If deleting it really improves performance repeatedly, then something else is wrong, and this just deals with the symptom.

     

    You might want to look at the hidden log Time Machine puts in each backup folder -- among other things, it shows the starting Event ID (and the last one is stored in the backup folder as an extended attribute, along with the File System UUID).  A clue may be lurking there.

  • bernuli Level 1 Level 1 (50 points)
    Currently Being Moderated
    Oct 12, 2012 5:21 PM (in response to Pondini)

    I agree.  I think something is wrong with fseventsd.  Deleting the its directory deals with the symptom, but I continue to look for a fix.

     

    I'll look at .Backup.log again but I hate doing that, cause they always say "Some filesystem changes made during the course of the backup may not be accounted for. Still busy after 2 retries." and I hate seeing that.

     

    B

     

     

     

     

    Pondini wrote:

     

    bernuli wrote:

    . . .

    my guess is BRumble meant "safe mode".

    That's my guess, too, but I'd like BRumble to clarify.

     

     

    But if you want the best TM performance possible, occasionally blowing away /.fseventsd works well.

     

    Unless the Event Store is damaged, that really shouldn't make any difference.  Time Machine keeps track of the last "event" it processed, so only looks in the Event Store for the directories that have been changed since the previous backup. 

     

    If deleting it really improves performance repeatedly, then something else is wrong, and this just deals with the symptom.

     

    You might want to look at the hidden log Time Machine puts in each backup folder -- among other things, it shows the starting Event ID (and the last one is stored in the backup folder as an extended attribute, along with the File System UUID).  A clue may be lurking there.

  • Pondini Level 8 Level 8 (38,710 points)
    Currently Being Moderated
    Oct 12, 2012 5:29 PM (in response to bernuli)

    bernuli wrote:

    . . .

    I'll look at .Backup.log again but I hate doing that, cause they always say "Some filesystem changes made during the course of the backup may not be accounted for.

    Aha!  Clue!  

     

    From the  ArsTechnical article above:

     

     

    As with all kernel-based file system notification mechanisms, including /dev/fsevents, there's still the possibility of file system changes occurring without going through the kernel. For example, a removable disk may be mounted on another non-Leopard computer and modified there. When it returns, the local kernel has no idea what's changed.

    The FSEvents API includes callbacks for these situations, effectively telling the client, "Unknown changes have occurred. You'll have to do a full rescan yourself, then pick up on the new event stream going forward." That's certainly not what a program wants to hear, but it's the unavoidable truth. and FSEvents is upfront about it. In effect, it's a form of reliability. FSEvents will not lie to you.

     

    Does that ring a bell?

     

    Or, since it says "during the course of the backup," something similar?

     

    Does Time Machine then do a deep traversal?

     

     

    Message was edited by: Pondini

  • bernuli Level 1 Level 1 (50 points)
    Currently Being Moderated
    Oct 12, 2012 5:36 PM (in response to Pondini)

    Huh, that does sort of ring a beep.

     

    One question though, after a standard reboot, should there be files in /.fseventsd that date a month or more back?  I guess if it was trying to keep track of removable media than the answer could be yes.

     

    Thanks for the input

     

     

    B

     

     

     

    Pondini wrote:

     

    bernuli wrote:

    . . .

    I'll look at .Backup.log again but I hate doing that, cause they always say "Some filesystem changes made during the course of the backup may not be accounted for.

    Aha!  Clue!  

     

    From the  ArsTechnical article above:

     

     

    As with all kernel-based file system notification mechanisms, including /dev/fsevents, there's still the possibility of file system changes occurring without going through the kernel. For example, a removable disk may be mounted on another non-Leopard computer and modified there. When it returns, the local kernel has no idea what's changed.

    The FSEvents API includes callbacks for these situations, effectively telling the client, "Unknown changes have occurred. You'll have to do a full rescan yourself, then pick up on the new event stream going forward." That's certainly not what a program wants to hear, but it's the unavoidable truth. and FSEvents is upfront about it. In effect, it's a form of reliability. FSEvents will not lie to you.

     

    Does that ring a bell?

  • Pondini Level 8 Level 8 (38,710 points)
    Currently Being Moderated
    Oct 12, 2012 5:44 PM (in response to bernuli)

    bernuli wrote:

     

    Huh, that does sort of ring a beep.

     

    One question though, after a standard reboot, should there be files in /.fseventsd that date a month or more back?  I guess if it was trying to keep track of removable media than the answer could be yes.

    I don't know -- I've never been able to find much documentation on how it works, just the API for how an app should use it (the File System Events Programming Guide, if you're interested).

     

    But each volume (drive or partition) has a separate Event Store, so it should contain only the records for that volume.

     

    Perhaps the Safe Boot examines the store, determines when something bad happened, and deletes the earlier entries.

     

    You answered before my second thought:

     

    Since the message says "during the course of the backup," something similar?

     

    Does Time Machine then do a deep traversal?

  • bernuli Level 1 Level 1 (50 points)
    Currently Being Moderated
    Oct 12, 2012 5:55 PM (in response to Pondini)

    I don't have the deep traversal problem other have been having.  I do of course end up with

     

            Scanning nodes needing deep traversal

            Node requires deep traversal: / reason:must scan subdirs|new event db|

     

    after  removing .fseventsd

     

    My problem with Time Machine has been that it backs up loads of directores, for no reason.  For instansce, it travels through my iPhoto Library and backs up 50 or so directores, but no files.  Even though I never touched iPhoto or had an iphone plugged in etc. 

     

     

    B

     

     

    You answered before my second thought:

     

    Since the message says "during the course of the backup," something similar?

     

    Does Time Machine then do a deep traversal?

  • Pondini Level 8 Level 8 (38,710 points)
    Currently Being Moderated
    Oct 12, 2012 6:03 PM (in response to bernuli)

    bernuli wrote:

     

    I don't have the deep traversal problem other have been having.

    Sorry, what I meant was, when TM sends the "Some filesystem changes made during the course of the backup may not be accounted for" message, does it then do a deep traversal?  Seems like it ought to, either during that same backup, or the next one..

     

     

    My problem with Time Machine has been that it backs up loads of directores, for no reason.  For instansce, it travels through my iPhoto Library and backs up 50 or so directores, but no files.  Even though I never touched iPhoto or had an iphone plugged in etc.

    Hmm.  Sounds like those directories are being reported as having something changed, but nothing actually is.  So TM backs-up the directory itself, then looks at each folder/file inside it, trying to find which ones need to be backed-up.

     

    Do you have any apps running that might touch a lot of directories without actually changing anything?

  • bernuli Level 1 Level 1 (50 points)
    Currently Being Moderated
    Oct 12, 2012 7:04 PM (in response to Pondini)

    Yeah that is what is happening.  This will happen on a fresh boot, with nothing running.  All I have in login items is Terminal and /Library/Application Support/BlackBerry/BlackBerry Device Manager.app

     

    Since I just cleaned out .fseventsd, I will have to wait a month for TM to degrade then I can go through what is running to see if there is a conflict.  But for your reference, I have the usual things running:

     

    me@mbp:~$ ps ax

    COMMAND

    /sbin/launchd

    /usr/libexec/kextd

    /usr/sbin/ntpd -c /private/etc/ntp-restrict.conf -n -g -p /var/run/ntpd.pid -f /var/db/ntp.drift

    /usr/sbin/cron

    /System/Library/PrivateFrameworks/MobileDevice.framework/Versions/A/Resources/us bmuxd -launchd

    /usr/sbin/syslogd

    /usr/sbin/securityd -i

    /System/Library/Frameworks/CoreServices.framework/Frameworks/Metadata.framework/ Support/mds

    /usr/sbin/mDNSResponder -launchd

    /System/Library/CoreServices/loginwindow.app/Contents/MacOS/loginwindow console

    /usr/sbin/KernelEventAgent

    /usr/libexec/hidd

    /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/CarbonCo re.framework/Versions/A/Support/fseventsd

    /sbin/dynamic_pager -F /private/var/vm/swapfile

    /usr/libexec/dpd

    /usr/sbin/diskarbitrationd

    /usr/sbin/DirectoryService

    /usr/libexec/configd

    /usr/sbin/blued

    autofsd

    /usr/sbin/notifyd

    /usr/sbin/distnoted

    /System/Library/CoreServices/coreservicesd

    /System/Library/Frameworks/ApplicationServices.framework/Frameworks/CoreGraphics .framework/Resources/WindowServer -daemon

    /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/cvmsServ

    /sbin/launchd

    /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/cvmsComp_x86_64 1

    /usr/sbin/coreaudiod

    /sbin/launchd

    /System/Library/CoreServices/Dock.app/Contents/MacOS/Dock

    /System/Library/CoreServices/SystemUIServer.app/Contents/MacOS/SystemUIServer

    /System/Library/CoreServices/Finder.app/Contents/MacOS/Finder

    /usr/sbin/pboard

    /System/Library/Frameworks/ApplicationServices.framework/Frameworks/ATS.framewor k/Support/fontd

    /usr/libexec/UserEventAgent -l Aqua

    /System/Library/CoreServices/ServerScanner

    /Library/Application Support/Tablet/PenTabletDriver.app/Contents/MacOS/PenTabletDriver

    BBLaunchAgent

    RimAlbumArtDaemon

    /System/Library/CoreServices/CCacheServer.app/Contents/MacOS/CCacheServer

    /Applications/Utilities/Terminal.app/Contents/MacOS/Terminal -psn_0_53261

    /Library/Application Support/Tablet/PenTabletDriver.app/Contents/Resources/ConsumerTouchDriver.app/C ontents/MacOS/ConsumerTouchDriver -psn_0_77843

    /Library/Application Support/Tablet/PenTabletDriver.app/Contents/Resources/TabletDriver.app/Contents /MacOS/TabletDriver -psn_0_81940

    /System/Library/Frameworks/QuickLook.framework/Resources/quicklookd.app/Contents /MacOS/quicklookd

    /System/Library/CoreServices/backupd.bundle/Contents/Resources/backupd-helper -attach

    login -pf me

    -bash

    login -pf me

    -bash

    login -pf me

    -bash

    ps ax

    me@mbp:~$

     

     

     

    Hmm.  Sounds like those directories are being reported as having something changed, but nothing actually is.  So TM backs-up the directory itself, then looks at each folder/file inside it, trying to find which ones need to be backed-up

     

    Do you have any apps running that might touch a lot of directories without actually changing anything?

  • Pondini Level 8 Level 8 (38,710 points)
    Currently Being Moderated
    Oct 12, 2012 8:13 PM (in response to bernuli)

    bernuli wrote:

     

    Yeah that is what is happening.  This will happen on a fresh boot, with nothing running.

    Then it's probably caused by something that ran before the restart.

     

    Since I just cleaned out .fseventsd, I will have to wait a month for TM to degrade then I can go through what is running to see if there is a conflict.

    Doubtful it's a standard Apple process -- or lots of folks would have seen it.  Since it's intermittent, most likely it's something that only runs occasionally.

     

    One other long-shot possibility, of course, is a problem with your installation of OSX.  Have you tried installing the 10.6.8 "combo" update?   And/or, a fresh version of OSX?

     

    A few folks with very slow backups on 10.6.8 solved it with a fresh install of OSX, but only going back to 10.6.7, not 10.6.8.  No idea why, but that did work for a few.

  • bernuli Level 1 Level 1 (50 points)
    Currently Being Moderated
    Oct 12, 2012 9:25 PM (in response to Pondini)

    Pondini wrote:

     

    bernuli wrote:

     

    Yeah that is what is happening.  This will happen on a fresh boot, with nothing running.

    Then it's probably caused by something that ran before the restart.

    No.  I restart, then without running anything, I run a backup.  Then I run another backup.  Each incremental backup can take up to 20 minutes.  Looking at the actual items that are backed up (not the links) there are lots and lots of folders with very few files.

     

    This did seem to start after I upgraded to 10.6.8.  I reapplied the combo update but it did not fix it.

     

    Possibly a fresh install of OSX would fix it but I would like to know exactly what is going on.  But for now I have been clearing .fsevents which always brings the backup down to under a minute.

     

     

    B

  • Pondini Level 8 Level 8 (38,710 points)
    Currently Being Moderated
    Oct 13, 2012 8:04 AM (in response to bernuli)

    bernuli wrote:

    . . .

    No.  I restart, then without running anything, I run a backup.  Then I run another backup.

    What I'm suggesting is, the problem may have happened before the restart;  you wouldn't see the effect until you run a backup. 

     

    Or, of course, something that happens during a restart, which would implicate either OSX or something in /Library/StartupItems, /Library/LaunchAgents, /Library/LaunchDaemons, or ~/Library/LaunchAgents.

     

     

    Possibly a fresh install of OSX would fix it

    That's certainly worth a try.

1 ... 3 4 5 6 7 ... 12 Previous Next

Actions

More Like This

  • Retrieving data ...

Bookmarked By (5)

Legend

  • This solved my question - 10 points
  • This helped me - 5 points
This site contains user submitted content, comments and opinions and is for informational purposes only. Apple disclaims any and all liability for the acts, omissions and conduct of any third parties in connection with or related to your use of the site. All postings and use of the content on this site are subject to the Apple Support Communities Terms of Use.