Time Machine always requires deep traversal?

bernuli wrote:

I agree. I think something is wrong with fseventsd. Deleting the its directory deals with the symptom, but I continue to look for a fix.

I'll look at .Backup.log again but I hate doing that, cause they always say "Some filesystem changes made during the course of the backup may not be accounted for. Still busy after 2 retries." and I hate seeing that.

B

You and I seem to have identical problems, although did you start with Lion and roll back to 10.6? I did not - I've been on 10.6 since the start, but am clearly having the same symptoms.

I get the same .Backup.log messages with "...2 retries..." and it's been driving me nuts. I can't find anything anywhere that explains what is going on here.

BRumble wrote:

. . .

reinstalled OS (light version and full version)

I'm not sure what that means, but have you tried installing Snow Leopard, then only going back to 10.6.7, not 10.6.8, via the "Combo" update (at http://support.apple.com/kb/DL1361)?

That's worked for a few folks with similar problems, when nothing else did.

Pondini wrote:

BRumble wrote:

. . .

reinstalled OS (light version and full version)

I'm not sure what that means, but have you tried installing Snow Leopard, then only going back to 10.6.7, not 10.6.8, via the "Combo" update (at http://support.apple.com/kb/DL1361)?

That's worked for a few folks with similar problems, when nothing else did.

I invented my own terminology, didn't I? Light = using installer to reinstall OS without messing around with Home directory. Full = reinstall manually from the ground up. The first takes about 45 minutes or so, the second takes around 2.5 - 3.0 hours.

I saw a couple of posts about rolling back to 10.6.7. This specific problem does seem to be isolated to .8, and probably in only a few isolated cases. I'll download and give it a spin; the only risk I could see is missing the security patches that come with the newer .8. While the risk is probably insignificant from a security standpoint, it's also not non-trivial. It'll probably be at least tomorrow before I know anything, so I'll try to post back as soon as I can.

Thanks for the suggestion!

-H

For me, the still busy after 2 retries does not seem to be the source of slow backups.

However, tt looks as though that message did become a regular occurance after the update to 10.6.8. I see it occasionally from before the .8 update and once in a while on a Lion machine.

B

BRumble wrote:

I get the same .Backup.log messages with "...2 retries..." and it's been driving me nuts. I can't find anything anywhere that explains what is going on here.

Problem solved - I think. Logs show my overnight backups ran without a hitch on the hour, down to the second (backups were also starting to not run as scheduled). Each backup was approximately 22 seconds long, down from what crept up to about 13 minutes one week after cleaning out

.fsevents

The solution? Check out the screen recording I made. The link is below. The recording shows the fseventsr application recording file system events as Time Machine is performing a backup. Occasionally I click the menubar to show the progress, but it's not quite accurate. Apparently doing a screen recording and running an application like fseventsr slows things down a bit. 🙂 This is about 15 minutes of video edited down to about 5 1/2 minutes. It is not exciting, but it shows definitively that files are being created that should not be.

http://youtu.be/DRDev-4bFCk

Ultimately, what I saw here and by piecing together evidence via Time Machine Buddy, BackupLoupe, System Log files and Time Machine .Backup.log files was that approximately 25K files were being backed up on pass #1, followed by 17K on pass #2. The actual number would vary slightly each time, likely as a function of the files that needed to be legitimately backed up getting added to those constants of 25 and 17K. The offending items were literally (and I am using that word properly, not hyperbolically) thousands of

.plist

~/Library/Preferences

~/Library/Preferences/ByHost

iTunes.plist.xxxx

.plist

.plist

I hope this helps anybody out there having these problems.

Thanks for posting. Those are interesting findings. After seeing fseventsr in action, I definitely need!

B

Yes, that makes a lot of sense, and does explain why there's a huge number of files, but the size isn't large.

I do recall seeing some of those odd plists (not hundreds, much less thousands) on Snow Leopard. Since there weren't many, they didn't seem to be causing a problem (and I don't use iTunes much).

I've added a note to this effect to the yellow box in #D2 of Time Machine - Troubleshooting.

(It's odd that this seems to be caused by something in OSX, but doesn't seem to affect very many users.)

Thanks very much for running this down. 🙂

Message was edited by: Pondini

Another thought . . .

Can you tell what process is creating or changing those odd .plist files on your OSX volume? It's fairly hard to read, but all I can see on the screen recording are the backup copies being created by backupd.

It might be helpful if we could determine whether the originals are being created or changed by OSX or some other app.

Pondini wrote:

I've added a note to this effect to the yellow box in #D2 of Time Machine - Troubleshooting.

(It's odd that this seems to be caused by something in OSX, but doesn't seem to affect very many users.)

Thanks very much for running this down. 🙂

Very cool; I really do hope this proves to be the long-term fix for this. I was tempted to hold off on posting this til Monday, as I wanted to keep from jumping the gun. Ultimately it seemed that posting this might encourage others to try it, thereby speeding up the validation process. I agree that this is really odd behavior. I'll comb through my system logs (again) and see if anything turns up.

Cheers

Ultimately this did not work either, although getting rid of all those mystery .plist files didn't hurt anything either. I upgraded to 10.8.2 a couple of weeks ago. Backups are smooth as butter, and the madness is over. Really loved 10.6.8 otherwise and didn't really want to upgrade, but Mountain Lion does have some benefits to it. Biggest downside is it is SLOW to startup and shutdown with all of its state-saving ways. Oh well, backups work, and that's the biggest benefit so far.

Thanks to all

I would like to ask for some guidance on these plists. BRumble told me in another thread to come check this out.

My problem is not identical, but just as frustrating. I am using a black 2007 Macbook 2.2 GHZ Core 2 Duo with 128GB Crucial m4 SSD, 4gb RAM, and running 10.6.8. My backups have become slow. The below data is from Time Machine Buddy from an incremental backup moments after a previous backup. Typcially take 20 minutes of high CPU usage to backup thousands of files of small total size. Always runs two passes with high pre-thinning requests.

I have lots (houndreds) of plists in the two mentioned preferences folders, but none that have plist.xxxx. Some of the plists date back to 2007. What is safe to remove and what might actually help? I doubt this is the fix since it is not the thousands of files that TM Buddy is finding. Any other suggestions?

Thanks in advance,

RJ

"Starting standard backup

Backing up to: /Volumes/TIME WARP/Backups.backupdb

No pre-backup thinning needed: 771.0 MB requested (including padding), 99.95 GB available

Copied 16301 files (524 KB) from volume Macintosh HD.

No pre-backup thinning needed: 770.5 MB requested (including padding), 99.93 GB available

Copied 12085 files (71 KB) from volume Macintosh HD.

Starting post-backup thinning

No post-back up thinning needed: no expired backups exist

Backup completed successfully."

gogogut wrote:

. . .

I have lots (houndreds) of plists in the two mentioned preferences folders, but none that have plist.xxxx. Some of the plists date back to 2007. What is safe to remove and what might actually help?

You're probably not having the same problem. 😟

There shouldn't be hundreds of files in the top-level /Library/Preferences folder, as that's for system-wide settings only. There will be more in your user home folder(s), but neither ought to be causing a problem, as few are changed frequently.

Start by using one of the apps in #A2 of Time Machine - Troubleshooting to determine what's actually getting backed-up.

If that doesn't provide a clue, see #D2 in the same link, especially the green box.

Are the lots of small files that are getting backed up actually just folders?

Can you post the output of:

sudo ls -l /.fseventsd

B

gogogut wrote:

I would like to ask for some guidance on these plists. BRumble told me in another thread to come check this out.

My problem is not identical, but just as frustrating. I am using a black 2007 Macbook 2.2 GHZ Core 2 Duo with 128GB Crucial m4 SSD, 4gb RAM, and running 10.6.8. My backups have become slow. The below data is from Time Machine Buddy from an incremental backup moments after a previous backup. Typcially take 20 minutes of high CPU usage to backup thousands of files of small total size. Always runs two passes with high pre-thinning requests.

I have lots (houndreds) of plists in the two mentioned preferences folders, but none that have plist.xxxx. Some of the plists date back to 2007. What is safe to remove and what might actually help? I doubt this is the fix since it is not the thousands of files that TM Buddy is finding. Any other suggestions?

Thanks in advance,

RJ

"Starting standard backup

Backing up to: /Volumes/TIME WARP/Backups.backupdb

No pre-backup thinning needed: 771.0 MB requested (including padding), 99.95 GB available

Copied 16301 files (524 KB) from volume Macintosh HD.

No pre-backup thinning needed: 770.5 MB requested (including padding), 99.93 GB available

Copied 12085 files (71 KB) from volume Macintosh HD.

Starting post-backup thinning

No post-back up thinning needed: no expired backups exist

Backup completed successfully."

Thanks for replying and offering advice.

Can you clarify if I want to be looking for the plists in the LIbrary/Preferences folder in the Hard Drive or the user folder? Hard Drive/Library/Preferences has 78 items, 58 of which are plists dating back to when I purchased the macbook. There are also weird "Document" files with names like "com.apple.alf.plist~orig". There is no ByHost folder.

user/Library/Preferences has 456 items, 396 of which are plists. There are also lots of preferences and files titled something like "cfx#1viKJTD". ByHost folder has 108 items, all but 2 of which are plists. Some have weird names like "com.apple.preference.internet.6119EFA9-05D1-5A52-8891-C448B31E2443.plist". The other 2 files are "cfx#0yWLPMC" and "cfx#M6sHHb6".

I had previously done almost everything on your website. The only things I have not done is uninstall Sophos Antivirus and relaunch the Finder. I do not have the AV set to scan except when I tell it to; plus it isn't working right now since I ran some genermal maintenance with Onyx (I'll figure that one out later or just delete it). I didn't think Finder relaunch would matter since I restart my computer all the time. Should I relaunch it anyways? I am also not sure how to determine if my external drive has a spin down feature. This drive used to work perfectly so I think not.

Here is the latest Time Machine Buddy data from a 12-minute incremental backup moments after another backup:

"Starting standard backup

Backing up to: /Volumes/TIME WARP/Backups.backupdb

No pre-backup thinning needed: 770.4 MB requested (including padding), 99.82 GB available

Copied 12203 files (141 KB) from volume Macintosh HD.

No pre-backup thinning needed: 770.5 MB requested (including padding), 99.80 GB available

Copied 12203 files (161 KB) from volume Macintosh HD.

Starting post-backup thinning

No post-back up thinning needed: no expired backups exist

Backup completed successfully."

I don't get the regularity and high number of files, small size, 2 passes, and slow process.

Backup Loupe said:

109.89 kb backed up.

MacHD = 93.48kB

.exclusions.plist = 13.35kB

.Backup.log = 3.06kB

.com.apple.TMCheckpoint = 0kB

Within MacHD:

Users = 61.45kB --> user folder has .DS_Store = 39.94kB and Library with .DS_Store = 21.51kB

.DS_Store = 15.36kB

Library = 15.36kB --> with .DS_Store = 15.36kB

private = 1.30kB --> with "var" folder --> with db folder and samba folder

-db folder has .TimeMachine.Results.plist = 1.07kB

-samba folder has brose.dat = 231B

I do not know how to run Terminal commands, but will follow your advice if you have suggestions.

Thanks again,

RJ

That terminal command really easy.

Go into your Utilities folder (sub folder of your main Applications folder) and double click Terminal

Or click on your desktop then from the Go menu, select Utilities

Into the window that opens, type sudo ls -l /.fseventsd

you will need to enter your password.

copy and paste the output here

B