Time Machine always requires deep traversal?

Podini,

I discovered something today. My problem seems to be with the File System Event Database as described in your FAQ.

Today I did the following.

Action 1) Created an empty folder.

Result 1) TM will backup that folder once, then use hard links on subsequent backups.

Action 2) Opened text edit and saved a small text file in the empty folder.

Result 2) TM will now create a new instance of the folder on each backup. The text file is backed up correctly via hard links on subsequent backups.

If I clear /.fseventsd, TM will behave correctly, but if I modify the text file with the empty folder, TM will backup the folder (and scan what is in it) every time.

This could definitely slow down TM as will have to scan numerous folders for no reason.

Can you try the above 2 actions on your system and let me know if you have the same behavior?

B

bernuli wrote:

. .

If I clear /.fseventsd, TM will behave correctly, but if I modify the text file with the empty folder, TM will backup the folder (and scan what is in it) every time.

If you mean it looks at the folder every time the file is changed, then yes, that's exactly how it's supposed to work.

The Event Store doesn't track each file that's added/changed/deleted; only each folder that has anything inside it changed. So when a change is made to a file, the enclosing folder is also marked as changed, and an entry made in the Event Store. On the next backup, TM compares the folder contents to the contents of the most recent backup of that folder.

Unless there's a problem with the Event Store, or some process marking lots of things changed that aren't changed, it works quickly.

Pondini wrote:

The Event Store doesn't track each file that's added/changed/deleted; only each folder that has anything inside it changed. So when a change is made to a file, the enclosing folder is also marked as changed, and an entry made in the Event Store. On the next backup, TM compares the folder contents to the contents of the most recent backup of that folder.

Right. But what is happening to me, is TM is scanning the folder and files within it, even though neither files within the folder, nor the parent, nor the folder istelf has had any change.

This is of course slowing things way down.

B

Then either something is marking the folders as changed, or the Event Store is getting damaged.

Have you installed a fresh version of OSX? If not, I'd start there.

Then I'd check all 3rd-party apps and kernel extensions for anything that might be suspicious.

If so, you might be able to figure out what's happening via the fseventr app. It's not been updated in quite some time, but should work on SL and might provide a clue.

Thanks, I'll have a go with fseventr.

I've been holding off on a clean install of 10.6.8 or upgrading to Lion (which would fix the problem for sure) until I figure this out.

B

bernuli wrote:

. . .

I've been holding off on a clean install of 10.6.8

No, I'm not recommending a clean install. Just a fresh version of OSX. That won't distrub anything else, and if the problem is something damaged in your current installation, it will fix it relatively easily.

See Reinstalling OSX in the pink box of Installing the ''combo'' update and/or Reinstalling OSX.

Hello all,

I used to be active in this conversation but then gave up on trying to solve the problem. However, I have been following the conversation to see if any new ideas were worth trying.

I see that CrashPlan came up as a question a few comments back. I use CrashPlan to back up my Documents folder to my mom's iMac. I also use Dropbox and SugarSync to sync files across my work and personal computers. I also do a TimeMachine backup to an external drive with LOTS of exclusions (mainly worried about documents, photos, music, media, etc. not applications). I then have a SuperDuper clone to a different external hard drive. So I have lots of redundancy and protection of loss. Perhaps one of these services or a combination of them could be part of the problem for these long backps of thousands of files when barely anything has actually been used or changed. Any thoughts?

RJ

RJ,

I don't use CrashPlan, SugarSync or dropbox and am affected by slow backups.

My system is pretty straight forward, only have installed, Office 2008, then 2011 and Adobe CS5 Design Premium, VMware, and Firefox.

I am pretty sure the TM slowdown occured sometime after I updated to 10.6.8.

RJ

gogogut wrote:

Hello all,

I used to be active in this conversation but then gave up on trying to solve the problem. However, I have been following the conversation to see if any new ideas were worth trying.

I see that CrashPlan came up as a question a few comments back. I use CrashPlan to back up my Documents folder to my mom's iMac. I also use Dropbox and SugarSync to sync files across my work and personal computers. I also do a TimeMachine backup to an external drive with LOTS of exclusions (mainly worried about documents, photos, music, media, etc. not applications). I then have a SuperDuper clone to a different external hard drive. So I have lots of redundancy and protection of loss. Perhaps one of these services or a combination of them could be part of the problem for these long backps of thousands of files when barely anything has actually been used or changed. Any thoughts?

RJ

Similary to Bernuli above, I don't use any backup software other than Time Machine. I do have Dropbox installed, but I only sync this manually, and the deep traversals occur even when I've not been using it.

Well things seem to have stabilised for me - backups take about 7-8 minutes for the backup itself + another 7-8 minutes to remove expired backups from the external. (I have 50GB free on a 1TB volume.)

Overall it always copies 50,000 or so files (at least it thinks it is) amassing just a few MB if nothing changed, or ~500MB if my Dropbox or Sente database files were modified.

I can live with it, but it is too slow to run every hour in my opinion - the external is too noisy. It also eats too much space too quickly on the exernal due to the dropbox folder changes. If I let it backup every hour, I'd use up about 6-10GB per working day, which is just too much in my opinion. I've set it to backup once at night and every 4 hours when the system is awake. (Totalling about 2GB of writes per day.)

However - perhaps my system is now behaving completely normally? Are these times reasonable for a (full) FW800 drive? I have no deep traversals since deleting .fseventsd - so that seems to be a permanet fix for me (at least for now.)

Thanks Bernuli/Pondini et. al. And good luck to those chasing down the bug.

No, that's definitely not normal. 😟

As comparison, when I back up to a F/W 800 drive without having done anything significant, TM backs up perhaps 25-50 MB, says about 2500 files, deletes one expired backup, and takes 40-50 seconds from start to finish. That's on Mountain Lion, which is usually somewhat quicker than Snow Leopard, but not dramatically.

Something is wrong. You've apparently tried most of the known fixes. But if you haven't yet, try deleting the Spotlight index on your TM drive, per the pink box in #D2 of Time Machine - Troubleshooting. On the next backup, Time Machine will re-create it and re-index the backups, so that one will take quite a while, but thereafter backups may be very quick, but shouldn't be worse than now.

If still no help, try any of the other things in the green box that you haven't tried already.

My external HD (for TM) is currently excluded from Spotlight (as are three other partitions).

However, I've just had a poke in there (after enabling hidden files in TinkerTool).

Shoudl I delete the .Spotlight-V100 folder anyway? (It's only 29 KB)

Also, I have a folder named .fseventsd at the root of the external HD.

It is 8.6MB in size and contains 1013 items dating back to 15th Jan 2013. Is that normal?

ZX48 wrote:

My external HD (for TM) is currently excluded from Spotlight

Not really. 😉 Time Machine backups are indexed automatically. That setting applies only to other data on the disk. Effective with Lion, when you exclude a TM drive/partition, it puts up a message to that effect. See the paragraph that starts "Note. . ." in Mac OS X 10.6: Setting Spotlight preferences

Shoudl I delete the .Spotlight-V100 folder anyway? (It's only 29 KB)

Worth a try.

Also, I have a folder named .fseventsd at the root of the external HD.

It is 8.6MB in size and contains 1013 items dating back to 15th Jan 2013. Is that normal?

Yes, that's the File System Event Log for that volume. OSX keeps one on each HFS+ drive/partition.

OK. I tried to delete the .Spotlight-V100 folder. It went straight in the trash, but now I can't empty it form the trash...

Just tried "Trash it!' . The files are still sitting there...

And why do the .fseventsd files only go back to Jan 15th?

OK....I deleted the spotlight index and...now TM is doing:

Oh dear.

Looks like I'll be deleting .fseventsd in single user mode again tomorrow morning!

:-)