UPDATED: Serious Time Machine bug on Mountain Lion

An impressive list! My thoughts:

All the cache (and similar items) down to WebpageIcons: I am very surprised by these. I think they are new to this thread. I would expect all of them to be excluded from Time Machine by files/folder extended attributes on each item. On my computer, at the temrinal prompt, I type the first line below and it returns the following two.

xattr /Users/john/Library/Safari/WebpageIcons.db

com.apple.metadata:com_apple_backup_excludeItem

com.apple.quarantine

Has anyone else seen these (or similar) in the ExcludeByPath key?

PlayReady is added by the installation of MS Silverlight. I think this is benign. But many who have had our TM problem also have installed Silverlight and have this entry.

/Users/Shared/adi I have this and I think is put there by the Adobe updater program - Can someone else confirm? I am not aware of it causing problems.

//Users/Shared/adi looks very odd - it not a proper path.

The others (from /Applications down to /usr) the symptom of our problem. For some people they appear and stay (which is very bad as they never get a full backup). For others (like me) they appear in the list for one backup and then disapear again - so one TM backup misses them, but they get backed up in the following one.

So, in conclusion, your plist is unhappy. I suggest you:

1. check the extended attributes of the caches etc for the com_apple_backup_excludeItem attribute. If it is not there then your whole system seems a bit suspect.

2. delete the plist and let it be recreated as the default and add any exclusions you want via the System Preferences.

3. See how the next TM backup behaves.

Best of luck.

The attributes of the caches etc seem fine.

I've deleted the plist: it was re-created on boot as nice & clean; the next backup included Applications etc; in the 20 hours since the .plist has not had the Exclude section created, so all is now well.

That is, until the next time it happens! The only thing to do is check the .plist, perhaps monthly. Tedious.

I've referred this to Apple & am waiting for an engineer to investigate & respond: I'll post any response here.

AlWil wrote:

...noticed that if you highlight the .plist in Finder, the preview opens showing the HTML and it's reasonably easy to look through that image (although no Find etc). Does anyone know what process opens the file in that way for viewing? It's not Preview.

A ".plist" is simply the extension Apple uses for an XML file containing a list of 'keys' and usually a 'dictionary' and sometimes other data. It is in plain text, just like HTML. I suspect that Safari (or whatever you have set as your default web browser) is what is opening the file. Of course, you can't edit it in that method. If you right-click (control-click) the file, you should get a Contextual Menu with several "Open..." menu items. The OS will look at the file and offer several app choices, one of which should be TextEdit. There are other methods of choosing an app, but that's probably the easiest.

The critical thing to remember, if you plan on editing the plist, is that you save your changes by forcing the app to use its "Plain text" option. If the app doesn't have or you can find that method, just Quit the app and DO NOT SAVE the file. Secondly, do not allow the app to change or add any suffix to the file; it should remain as originally named: "SomeAppsPreferencesFile.plist"

I can open a .plist in TextEdit, but line breaks are stripped out, which makes it very difficult to follow. Safari, Chrome & Pages will not open a .plist.

I don't want to edit it: that would be dangerous and it's much easier to force the OS to create a new clean .plist.

All I want to do is view it in a reasonably easy to understand way: the best I've found is the image of the file in Finder when viewed in columns. Line breaks are retained. So what app can open a .plist in the same way as it's seen in Finder?

May I correct a comment I made earlier.

Many of you have noticed /Users/Shared/adi in the excludeByPath key. This is put there by iBooks (why I don't know). This link has the detective work which discovered the iBooks connection. http://galvanist.com/post/66313155502/users-shared-adi

You may want to check your preferences in TextEdit.

To ensure you don't actually edit/change anything, use the "Format->Prevent editing" menu item. Reduces those "Oops" moments! 😮

You may want to 'play' with the following settings:

Open and Save tab: UN-check the two 'Display...' settings under the 'When Opening a File:' section

In the "Plain Text File Encoding" section, leave the 'Opening files:' and "Saving files:' choices as "Automatic".

Probably check the "Preserve white space" checkbox, also.

I'm not sure what setting would be stripping the line breaks, possibly one of the above. Not sure why that would be particulaly useful, except maybe when looking at a Microsoft file which tend to use line breaks AND carriage returns.

You might also consider installing the free Text Wrangler text editor from Bare Bones.

Just to chime in, the same overall problem hit me. I just realised it when I tried to migrate data to a new MBP. It does not recognise the TM backups, hence I checked for the usual suspects, and they are (not) here. No applications, no system files etc. So far the standard procedures do not help. I think I will revert to manual backup every day. Seems like this has started with me upgrading to Mavericks. First the Calendar is not working correctly (on the old MBP), now this ... (which is even worse).

Regards

I have a very similar problem where I have over 400 paths in ExcludeByPath in the TM plist. Most of these are build directories, so I assume Xcode added them, however it somehow also added my root development dir at /Users/myUserName/src which excluded ALL OF MY SRC DEVELOPMENT 😟

Naturally, I trusted TM and didn't find this out until my laptop died (liquid spill) and I did a restore on a loaner machine. I started using TM back in Snow Leopard, so I imagine if the TM bug was in Mountain Lion, it carried over in the plist when I upgraded to Mavericks in the summer.

As per the #D10a, one of my exclude paths is "/Library/Application Support/Microsoft/PlayReady" so it must be related, although my root dirs (/Applications, etc) backed up fine.

Looks like I gotta pony up big money to get the data in that folder recovered. 😟

I can provide the plist if needed.

Update: I backed up the plist and deleted it. It was regenerated after I visited the TM gui ... including the ExcludeByPath & IncludeByPath entries.

Ah ok. Removing the plist and restarting gives me a clean plist now.

My faith in Time Machine has been severely dented, especially since having fixed the plist, it has now hung several times over the last few weeks, preventing the Mac from shutting down.

Has anyone any recommendations for alternative backup software?

Hi,

I had this issue and I was covered by Applecare so I can hopefully shed a bit more light on this, but unfortunately the source of the problem is still unknown.

Three things first of all:

1. First of all, thanks to Pondini for his detailed web page on this subject and solution for patching the issue.

2. Apologies if this has been covered already but I don't have time to read eleven pages of previous posts.

3. I had this issue on a Mavericks machine as well as a Mountain Lion machine. But... on that machine I recently upgraded from Mountain Lion and for all I know the problem was there before I upgraded so it may be possible that Mavericks fixes the issue (if you fix using Pondini's instructions first).

So... bit of a long story but I'll start with Apple's response:

- The cause of the problem is unknown and therefore could potentially re-occur after resetting the plist. This is rather unsettling and the only conclusion I can come to is that we have no choice but to monitor Time Machine on a regular basis to check it hasn't re-occurred. If I find the time to write an automated script I'll post details here.

- The person I spoke to said it was not a known issue. He also recommended that anyone with Applecare experiencing the issue should call Apple because they do not pay attention to these forums and reporting via the support channels is the only way to bring it to their attention.

- There are several potential sources of the issue e.g. a corrupt file, a backup that didn't complete normally, a rougue software installer, and a couple of other ideas that I don't remember. I think these were mostly guesses though.

- The issue might be fixed in the latest version of Mavericks. I'd be very interested to know if anyone with Mavericks has experienced this (and knows for sure the problem didn't exist before upgrading).

- To give Apple the best chance of finding the source of the fault you need to call them and speak to a senior engineer as soon as the problem occurs. Note this is different from as soon as you notice the problem. This is mostly unworkable without using a script to execute the backup and check the backup immediately (but I intend to write such a script). If you call them they will send you a file to scan the state of your machine and all this has to be done before you make any other changes to your system after the issue occurs. This is also problematic as it took me two weeks after reporting the problem to finally speak to someone qualified to talk about it. I'm not sure how interested Apple are in this but because of the very small time-window for them to investigate, it might be worth calling them even if you don't have Applecare.

Sorry - I said I was going to start with Apple's response but I've interspersed their response with my own thoughts! If you do speak to them it is important to insist on speaking a senior engineer. Their first level of support just told me to upgrade to Mavericks and reset my Time Capsule to factory settings.

Luckily we had enough machines in the house to prove it was not a hardware issue and that upgrading to Mavericks does not fix the issue:

Machine OS Backup Device Missing Backup Files?

iMac Mountain Lion Time Capsule (Ethernet) Yes

iMac Mavericks Time Capsule (Wi-Fi) Yes

MacBook Pro Mountain Lion Time Capsule (Wi-Fi) NO

MacBook Pro Mountain Lion USB Hard Drive Yes

I emailed Apple before and after plist and exlusion files from all three affected machines and system profile of the machine under warranty but apparently these were of no use because as I said above they need the state of the machine at the point in time where the system folders get excluded from the backups.

If this truely was an unknow issue until I reported it, I think it is unlikely that we will ever see a fix for this for Mountain Lion.

If I ever get around to writing some scripts to mitigage the damage, I will update this post.

J

Message was edited by: koagul8or.

Thanks, J, that's useful information.

I can verify that the still occurs in Mavericks, though the symptoms are a bit different. Three times I reset Time Machine (twice with a new, clean backup disk) and each time Time Machine failed after a few weeks. In the most recent failures the missing files were in the ~/Library/Preferences folder, not the top-level system folders mentioned in the original report. I am not sure Pondini's scripts will find these failures. I ran "tmutil compare -n" and inspected the output manually to track these errors down.

It is interesting that the errors occurred in a folder where files change frequently; perhaps the bug is triggered when files are changing in the middle of a backup?

I also verified (using fs_usage) that once Time Machine drops a file it will not back up that file on subsequent runs. When it visits a file on the source disk it checks to see if a copy is present in the most recent backup. If not it skips the file. Presumably this only occurs on older files; files created since the last backup *do* get backed up.

Given the repeated failures of Time Machine and the way it propagates errors to later backups I decided it was a multi-level disaster and ditched it. I now pull daily full-disk backups using Chronosync. There are other alternatives to Time Machine, though none as convenient when it comes to restoring older files.

I don't have Apple Care. Even if I did I'm not sure how one is supposed to detect an error which could occur on any hourly backup and then immediately get the attention of a senior engineer at Apple. I really don't see how anyone can actually act on the advice you were given.

This IS a known issue with Apple. I have Apple Care and opened Case #443758601 way back in May 2013. The senior advisor who worked with me was John Chappell. For almost 8 months I kept calling him back for an update on the issue and his primary response was " The engineers said it would be fixed in the next update". Well, 3 to 4 Mountain Lion updates later and now Mavericks, there is still no fix and you were told that this is NOT a known issue with Apple.

I wish you luck. Maybe your advisor will have a bit more pull with the Gods of the Apple engineers and finally get the issue resolved. I for one, gave up and finally utilized Pondini's solution and my Time Machine has been working fine ever since. I am still on Mountain Lion and will most likely stay with it until I find out that this issue is resolved in Mavericks. I am also doing regular clones using SuperDuper.

I look forward to reading future posts concerning your progress with your case #.

beldenfox wrote:

I can verify that the still occurs in Mavericks...

Can I ask if your Mavericks install was a fresh install or an upgrade from an older version of OS X? With my own Mavericks machine it was an upgrade and so impossible to tell if the issue existed before I upgraded. If that's the case, there's still hope that the issue does not occur on Mavericks if backups are working before you upgrade.

beldenfox wrote:

I don't have Apple Care. Even if I did I'm not sure how one is supposed to detect an error which could occur on any hourly backup and then immediately get the attention of a senior engineer at Apple. I really don't see how anyone can actually act on the advice you were given.

Agreed. Not impossible but practically impossible for most people. That said, some people I know only backup occasionally on when they remember to plug in their backup USB drive so it's a bit more feasible there but that kind of person is probably not going to remember to check backups before and after doing that! 🙂

That said, IF Apple are unable to reproduce the issue themselves, capturing the machine state at the time of the issue is about the best you can do really. I'm still planning to disable the automatic UI Time Machine backups and trigger them from a script that checks the before and after state.

It's kind of low on my list of priorities right now though. Will keep all posted if I get around to it. The more people running a script like that the better I guess. Even then, there's no guarantee it would pinpoint the problem or that Apple would actually act upon the information. Maybe it's 'cos I'm a software dev myself that trying to track it down just seems like the thing to do! 🙂

J

rxbudp wrote:

This IS a known issue with Apple. I have Apple Care and opened Case #443758601 way back in May 2013. The senior advisor who worked with me was John Chappell. For almost 8 months I kept calling him back for an update on the issue and his primary response was " The engineers said it would be fixed in the next update". Well, 3 to 4 Mountain Lion updates later and now Mavericks, there is still no fix and you were told that this is NOT a known issue with Apple.

Can't say I'm particularly shocked to hear this. 🙂 Given the number of posts on the subject I was surprised to be told that I was the first person. Maybe it's taken out of context. Perhaps he meant his team in Ireland were not aware of it. Maybe he didn't use the correct search terms when/if he searched their database.

For the record, my case number is #583231959. I don't expect to hear anymore about it unless I can recreate it. I suppose one of my older machines I could try to break by unplugging the backup disk mid-backup or killing the process or something but I like to think Apple might have already tried that themselves.

rxbudp wrote:

I look forward to reading future posts concerning your progress with your case #.

You'll be the first to hear if I discover anything. To be honest I'm resigned to just manually checking once a week. Even if I wrote a script to do the checking then I'd have to worry that there were no errors in my script! ")

J