UPDATED: Serious Time Machine bug on Mountain Lion

Anyone affected who'd like to help, either by calling AppleCare or filing a Bug Report (we can provide details and instructions), feel free to reply either here or as noted in #D10 of Time Machine - Troubleshooting.

jmrz wrote:

It's definitely not the TC's hardware. I have the same problem occuring on both the TC and the brand new ReadyNAS.

I was replying to cyrus_b_uk.

Note that since he's clearly having a different problem, I asked the Hosts to split his posts and my replies to a new thread. Unfortunately, your second reply got moved there, too.

How do you go about repairing the sparse bundle? I'd be interested to see if that fixes the problem.

See #A5 in Time Machine - Troubleshooting.

It's not clear from your other post -- were the top-level Applications, Libaray, and System folders backed-up or were they skipped entirely?

jmrz wrote:

. . .

Navigating through to the latest backup via terminal shows that the backup contains the User accounts (minus all but two/three applications in their respective applications folders)

If there were individual Applications folders inside user home folders, do you mean they weren't backed-up correctly?

and a bunch of other folders (hidden in finder) e.g. cores, opt, etc, tmp, var.

How about private? The bug addressed here skips it (that's why the backups can't be used with Setup Assistant or Migration Assistant).

I can only see Library folders under the user accounts through terminal (these are the ones that increased in size dramatically when coping via terminal).

That's probably the permissions issue -- the Finder won't show the size of things you don't have permission to read.

I can't see any Applications outside of the user accounts.

It does sound like you have/had this bug.

Unfortunately, you also don't have the evidence (it was in the top-level Library folder).

You might want to look back further in your backups -- you may have the Applications folder from before the problem started. If so, you can recover some "simple" apps by just copying from it. "Complex" apps, however (typically the ones that come with their own installers), also put files in various places in the top-level Library folder, and won't work properly, if at all, without them. Unless you can find and copy them, too, you'll have to reinstall those apps from the original discs or downloads. Most likely, any serial numbers or purchase keys will be gone, too, along with any Internet plug-ins, widgets, etc., installed for all users.

Once you get as much recovered as you can, and set up Time Machine again, run a backup and see if those top-level files get backed-up. If not, report it to Apple before doing anything else (ie, while you still have the evidence). If you're covered by AppleCare, that's the call to make. If not, and assuming you don't really want to pay them $50 or $60 for the call, you can file a detailed Bug Report. In either case, I'll be happy to show you how and suggest what to say.

Don't hesitate to ask for clarification, etc., and keep us posted.

jmrz wrote:

The individual applications folders inside the user directories were not backed up correctly. I had three user accounts on that computer and each only had two or three applications in the backup (all non-apple ones).

That's odd, but I've never tested much with Apps in user folders.

Were there Apple apps in those folders? Any pattern to what was skipped? I'd expect TM to back up anything it finds there.

In terminal, the folders showing in the latest backup folder are: Documents, Network, User Guides and Information, Users, Volumes, cores, etc, lost+found, mach_kernal, opt, tmp and var. There is definitely no private folder.

That's clearly this bug, then. 😟

I don't understand how the Library folder for one user can go from 40gb to 400gb though. The drive it came from was only 500gb full and there's more than 100gb of data outside of that Library folder. I just grabbed my fonts and keychains and deleted the rest of it, space was then restored.

Also odd, and doesn't seem related. Could be some sort of directory problem, I suppose. No way to tell now, of course.

I'm no longer covered by applecare (this machine is nearly a year out of it's applecare, hence the dual hdd upgrade), but I am more than happy to send in a bug report.

Cool. Keep an eye on your backups, to be sure those top-level folders do in fact get backed-up. For some folks, they're backed-up for a while, then skipped, then backed-up again, causing a large backup (15-20 GB or more). If it happens, post back and I'll give you info on how to report a bug and run a diagnostic package to attach to it, and also how to fix it (at least temporarily).

It doesn't seem to affect a large number of users, and seems connected to 3rd-party apps somehow, but we don't really know what's causing it. Something is changing the "plist" file where TM's instructions are kept, telling it to skip those folders. The way it's done is not something you can do via the TM Preferences window, and isn't shown there, so the only way to tell is to look at the backups.

Thankfully I was savvy enough to figure out how to get to my data, but I feel incredibly sorry for anyone who has had this problem and were not able to get around it.

Yup. 😟 I helped a couple of users do just that -- one very inexperienced, but she was also very determined -- and isn't inexperienced anymore!

Keep us posted.

jmrz wrote:

. .

Issue seems to have resolved itself after a fresh install of OSX. Latest backup has Applications, Library, System and Users now.

Yes, that got you a fresh copy of the preferences file: /Library/Preferences/com.apple.TimeMachine.plist

Some app or process, we don't know what, is changing it to exclude the top-level folders.

If it happens again, please let us know before doing anything else, so we can get you to either call AppleCare or file a Bug Report, so Apple can try to figure out what's causing this.

Thanks.

miakt wrote:

. . .

Is there any way I can retrieve the 4 months work that is missing, even manually? Are the missing files somewhere in Time Machine?

Yes, you should be able to recover most of it -- especially your home folder(s).

But you have lost any apps, Internet plug-ins, and Widgets installed for all users since March 12. You've also lost any serial numbers / purchase keys for paid apps. And most "complex" apps (typically the ones that came with their own installers) have lost their support files, so may not work properly, or at all, and will have to be reinstalled from the original discs or downloaded again.

Also printer drivers, system-wide settings and configuration, and various other things in the top-level Library folder.

I had a total false sense of security since I've succesfully restored from Time Machine in the past and it looked for all intents an purposes to be doing its job. This is a disaster.

Yup. 😟

Shouldn't Apple notify people that this is an issue since there is no way to know it is occuring until it is too late, and the consequences are potentially catastrophic?

I'm not sure they've identified it as a problem; I've not had a great deal of luck getting folks to report it. (It doesn't help much once your drive has failed, as the evidence is gone.) And since I've not figured out how to replicate it, I can't report it myself.

Even when they do identify such a thing, they never warn anyone. The most they do is provide a fix via an update.

Ok, end of rant. Here's what you need to do:

First, do a full system restore from the most recent full backup in March (see Time Machine - Frequently Asked Question #14 if you're not sure how to do that).

Then you'll need to restore the home folder(s) from the most recent backup, via the TM browser (the "Star Wars" display). That's a bit of a problem, though, since you can't restore the home folder for the user account you're running from, since it's in use.

So, if you have two user accounts, log on to one. If you don't have a second user account, create one via System Prefs > Users & Groups, log off your normal account and onto the new one.

Either select your internal HD in a Finder sidebar, or press Shift+Cmd+C to show your Computer, then select your internal HD from the center part of the screen.

Under your internal HD should be a Users folder; inside that should be the home folders for each of your User Accounts (plus the special Shared folder), something like this (with the Finder in List View):

Select the home folder for the other account (the one you're not logged-on to, click the Restore button at the lower-right, authenticate, and when you get a prompt about how there's already a folder there, select Replace.

Then log on to the restored user account and make sure everything's there. Repeat for the other home folder.

With the exceptions noted above, that should get you mostly recovered.

Don't hesitate to post back if you have questions or problems, and let us know how it goes.

Message was edited by: Pondini

There aren't really any error logs (but most of your logs are gone anyway, since they're not backed-up).

What's happening is, some process (OSX or 3rd-party, we don't know) is putting specific exclusions in the Time Machine preferences file to exclude the top-level Applications, Library, System, and some hidden folders. That file is in Library, so is now gone. The way it's done cannot be done via the TM Preferences window, and aren't shown there, either, so the only way you know things aren't getting backed-up is to look at the top level of your HD in your backups.

If you're covered by AppleCare and want to rattle them (800-275-2273 in the U.S), by all means do so. They won't have much to work with, but don't hesitate to refer them to this thread (there are others) and/or my web page: http://pondini.org/TM/D10.html

I have the evidence (copies of the file) from a number of users, and would be glad to speak with Apple about it, either by phone or e-mail, but since I cannot replicate the problem, and am not covered by AppleCare any more myself, they're unlikely to want to do that.

miakt wrote:

. . .

With Recovery software, I was able to find the TM preference files from later than the most complete back-up, and indeed, System, Library and Apps had been excluded from backup after March 12. I can send you this file, Pondini if you think it would be helpful.

Won't hurt (send to the address on my Contact page and please identify yourself if your e-mail address isn't a clue), but the main place it needs to go is Apple. If you're covered by AppleCare, please give them a call as in my previous post. Not much they can do now, but the more reports they get, with that preference file, the better the chances they'll recognize the problem and perhaps find the cause.

Is it possibly related to the ML update that came out that month, since several users seem to date the problem from around then?

It doesn't look like it -- some of the failures seem to have happened last November. But we stil have no clue what's triggering it.

1. When my hard drive failed, it had nearly 800GB of data on it. The new one has less than 500GB. I can't figure out what I'm missing. I compared the sizes of the current and backed up Sys, Lib, and Apps folders and the difference is only a couple of GB. There are also a lot of photos because I didn't back up iPhoto, but I don't think it's 300GB worth.

It's possible that was system work files, caches, logs, trash, etc., in addition to your photos. That's usually only a few GBs, but can be quite large in some circumstances. No way to tell now, of course. 😟

2. What am I supposed to do about Time Machine now? It seems my back-up is corrupt as well as incomplete. How do I know if it's the folder or the external drive it is on that is damaged? Should I make a new complete back up to this drive? Should I delete the old back-ups? It seems I can still pull anything off of there that the Mac isn't currently using. In the future, how can I tell if the back-up is corrupt before it is too late? I've used TM in the past to do a full restore and it worked perfectly, so now I don't know if I can trust it at all.

I'd recommend a couple of things:

• Start making "secondary" backups to a different drive, with a different app. Just as no hardware works forever, no app is perfect, either. That's always prudent, but even more so if there's the slightest doubt about your primary backups. See the green box in Time Machine - Frequently Asked Question #27 for some suggestions.

• If you're sure you don't need anything from the (partial) old backups, your best bet might be to reformat the TM drive (perhaps taking the time to zero-it out) and start fresh, just to eliminate anything that might be left over -- either a data or directory problem Disk Utility couldn't find, and reduce the possibility there's a hardware problem (if the zero-out fails, you know there's a problem; if it succeeds, the drive is likely ok, but there's never any certainty). Then periodically check your backups, by verifying that the Applications, Library, System, etc., folders are there, until Apple provides a fix.