UPDATED: Serious Time Machine bug on Mountain Lion

I have this bug as well. Only discovered it after purchasing a new imac and trying to use migration assistant to tranfer over everthing from my time machine. I have no idea how long my time machine backup has been corrupted but fortunately was able to migrate directly from the old mac using a twisted pair cable. If something had happened to the old mac I would be out of luck as this was my only backup.

It is not at all obvious if you have this bug. You only find out after you try and migrate and then it is probably to late. Time machine appears to operate normally and there is a large backup file on the time machine hard drive. I would strongly suggest to run migration assistant to see if it can see your backup on your time machine or run the script at pondini.org. I suspect that a lot of people have this bug but have no idea they do.

I contacted apple care this morning and told them all about this, gave them the website pondini.org, and offered them the time machine backup that I ran. The only thing I can't do yet is run timemachine on my new mac yet as I don't have the disk space but I do wonder if I migrated over the bug

I've had this same issue and may have found a cause. In every backup on my Time Machine drive AFTER the latest "complete backup" I found this file under the root backup (Backups.backupdb > [ComputerName] > SomeRecentBackupDate): CoreSyncInstall.log

The contents are this:

-kill

Using default path, invalid path given.

Using Install Path:

/Applications/Utilities/Adobe Creative Cloud/

Killing application

I believe Adobe Creative Cloud is killing Time Machine Backups.

What's intersting is that in each backup directory AFTER I installed Creative Cloud apps, this file exists and the other directories needed for a "complete" backup do not. Does anybody else have this file and similar symptoms?

I don't have Creative Cloud, but my symptoms are different; while I get 'excluded' backups from time to time, they are always followed by full backups. So my problem is not really that I can't restore from a backup (unless I happened to need to just after an 'excluded' one), it's that I keep running out of space on my backup drive because of all the 20GB backup sessions. My issue seems to be related to interaction between installd (triggered by Sophos AV updates) and backupd, and I'm working with Apple on that.

Thanks to the starter and community, my TM now shows a valid copy to use from recovery boot.

What I did was a confused .plist file deletion, several reboots and here we are.

Just, thanks.

I too have been hit by this twice since I upgraded to Mountain Lion in September, on a laptop that is only turned on a few hours a week.

There may be two situations here:

• Time Machine inexplicably does one enormous backup (with the symptom that it initially says the backup will be small, but the size keeps on climbing and climbing). After that things are back to normal. Investigation shows that the backup was huge because the previous backup excluded /Applications etc.
• Time Machine stops backing up /Applications etc. permamently.

My issue is the first one.

As previously noted, what is happening is a timing issue:

1. Time Machine starts a backup. backupd is now running.
2. An install starts.
3. installer > installd > PackageKit: suspends backupd
4. SecurityServer: Succeeded authorizing right 'com.apple.ServiceManagement.daemons.modify'.
5. PackageKit > install_monitor: temporarily excludes /Applications, /Library, etc.
6. Install runs, but so does...
7. backupd continues to run. It reads the Time Machine plist, which has the system folders excluded
8. Install completes
9. install_monitor re-includes /Applications, etc.
10. PackageKit releases backupd, which triggers the com.apple.ServiceManagement.daemons.modify right grant again.
11. backupd continues to run, which removes /Applications etc from the backup.

Observations:

• I believe when PackageKit suspends backupd, it is actually just using launchctl or an equivalent API to suspend the Time Machine schedule in launchd, as evidenced by the particular sandbox right it wants. It is clear it isn't actually suspending a running backupd task.
• This problem isn't connected to any particular third party application. If you look through the install log, the same pattern of install_monitor temporary excludes occurs on many (if not all) standard installs. I've even seen it on Apple installs.
• But since it is a timing issue, it is more likely to occur in certain cirumstances. For example, consider the Sophos AntiVirus update, which a) runs frequently, b) may run soon after the computer is awaken from sleep, just like Time Machine, c) and so may become aligned closely with the Time Machine backup time, d) takes long enough to install that the window where the folder paths is excluded is long enough to catch Time Machine (we're talking just seconds, though. On my machine the window of exposure was just 23 seconds), and e) hits a lot of folders during the install, which brings up point:
• One question is what does it take for Time Machine to a) notice that the folder paths are missing, and b) notice that they have returned. Consider that Time Machine usually needs an update in the fsevent log to get it to analyze a folder. Perhaps the fact that Sophos is installing to many if not all of the excluded paths contributes to the issue
• I too don't understand why PackageKit is excluding the paths. If it suspends backupd, then why bother changing the Time Machine rules?

I think you're probably right that installd doesn't actually suspend backupd if it is running - it just tries to block backupd from running. Which is bad programming.

I understand that installd doesn't want backupd to copy files while installd is replacing them. You run the risk that you end up with a backup that contains half of the old version of a package and half of the later version that installd is putting in place. If you restored from that backup, you'd probably have a non-functional app. But changing the Exclude list is a clumsy way to do it. Either installd needs to properly suspend backupd, or it needs to wait until backupd completes, or it needs to specifically exclude the subfolders it is actually changing. Or it needs to have some sort of inter-process communication with backupd so that both apps stay out of each others way.

Dear all,

the TM on my iMac was a mess for a while, working and stopping at random time.

Even the celebrated Pondini tricks could not solve the problem -- they worked sometimes, yet not deterministically.

A little bit desperate, I updated to Mavericks as soon as it was available.

Great, the TM works regularly. No accidents since Mavericks was available.

Encouraging, yet still not conclusive.

Should I get in trouble again, I'll show up.

Very best,

I haven't reported back in a while, but I am still working with Apple Support on my Time Machine situation. There's no way to recover lost applications, but I'm hoping something else can be done to make me a happy customer. (I'll let you know.) I was successful at getting them to replace my 2011 MacBook Pro. It only took took four repair attempts, four months, three new logic boards and two new hard drives to get them to do it, but they did. I returned the new MBP they gave me and got an iMac.

topherez, I use Creative Cloud, as well. I wouldn't doubt it if Adobe's syncing setup is somehow involved. That's been a less-than-stellar rollout across the board, in my opinion. That's difficult for me to say, too, because I like Adobe almost as much as I like Apple.

I missed my Apple Support rep's call today, but I'll come back and post whatever news I have after we talk, if there is any.

Howdy all,

This may be a resolution to the issue here…it worked for me, so I am sharing in case this works for others.

In trying to migrate my older MacBookPro to a new MacBookPro, I was having the same issue that many folks are having where the Migration Assistant doesn’t see the external USB drive with my Time Machine backup. After reading through the posts by a user, Pondini, I discovered that I had the “bug” that was causing my automatic backups to exclude the Applications, System and Library folders. For those of you not familiar with this “bug”, it is described by user Pondini here: http://pondini.org/TM/D10.html. I performed the following steps which seemed to resolve the backup issue and subsequently allowed the Migration Assistant to “see” my external USB backup drive and perform the data migration:

1. I performed the “Full Reset of Time Machine” as described by user “Pondini” here: http://pondini.org/TM/A4.html. I first deleted the file: /Library/Preferences/com.apple.TimeMachine.plist. NOTE: I DID NOT IMMEDIATELY PERFORM A NEW TIME MACHINE BACKUP AT THIS POINT…I did the next steps first.

2. I quit ALL applications - including any that might be running in the background…like Intego’s Virus Barrier software that continuously runs “Real-Time Scanning” of my system. I believe that the Virus Barrier software was interfering with the Time Machine backup process and causing the Applications, System and Library folders to somehow be excluded from my backups. However, I cannot say this definitively. But turning off Virus Barrier seemed to solve the issue for me. Is anyone else with this issue running Intego’s Virus Barrier? Or perhaps something similar in the background?

3. Next, I did a fresh Time Machine backup selecting the same external USB drive I had been using. When this subsequent backup was complete, the Application, System and Library folders were there.

4. I turned on my new MacBookPro (running Mavericks), and started the Migration Assistant application. DO NOT PLUG IN THE EXTERNAL DRIVE YET. I selected the option to migrate from an existing Time Machine backup. Once at the screen with the spinning grey wheel that says: “Looking for sources…”, THEN I plugged in the external USB drive, turned it on and it was immediately discovered. I hit “Continue” and it transferred everything to the new MacBookPro as it was supposed to.

5. I have run subsequent Time Machine backups from my new MacBookPro running Mavericks and have not seen the exclusion issue reoccur, but I’ll keep an eye on it.

Hope this helps. Thanks to all for their input on this issue…especially user “Pondoni” - who I understand passed away before this issue was resolved.

Message was edited by: jaystad

There certainly appears to be a connection between AV programs and this problem - not because of their anti-virus scanning activity, but because they have regular (often hourly) updates that are installed using the Apple installd app. It's the conflict between installd and backupd that causes either a temporary or permanent set of exclusions to be written into the Time Machine .plist file.

I now have this bug, but I didn't have it yesterday and I know precisely when it happened:

After spending some days cleaning and tidying the iMac, using EtreCheck mainly but also Pondini's check applet, I noticed an error with some stuff failing to load after a period of some inactivity, possibly including sleep mode.

On 9 November just before 07:00 (NB - all times GMT) I woke the machine from sleep and with nothing running I ran EtreCheck. The output shows load errors, as expected, and nothing else problematical. Extract from EtreCheck:

Kernel Extensions:

com.sophos.kext.sav (8.0.14)

Problem System Launch Daemons:

[failed] com.apple.wdhelper.plist

Problem System Launch Agents:

[failed] com.apple.accountsd.plist

[failed] com.apple.AirPlayUIAgent.plist

[failed] com.apple.printtool.agent.plist

Time Machine:

Skip System Files: NO

Mobile backups: OFF

Auto backup: YES

Volumes being backed up:

Silver iMac HD: Disk size: 751.32 GB Disk used: 476.49 GB

Destinations:

WD Elements TM [Local] (Last used)

Total size: 2 TB

Total number of backups: 62

Oldest backup: 2013-02-04 23:03:41 +0000

Last backup: 2013-11-08 17:25:16 +0000

Size of backup disk: Adequate

Backup size 2 TB > (Disk used 476.49 GB X 3)

I rebooted the machine then ran EtreCheck again. Sure enough, the load failures were gone, but I then noticed Time Machine errors. Extract from EtreCheck:

Problem System Launch Daemons:

Problem System Launch Agents:

Launch Daemons:

Time Machine:

Skip System Files: NO

Mobile backups: OFF

Auto backup: YES

Volumes being backed up:

Silver iMac HD: Disk size: 751.32 GB Disk used: 473.03 GB

Destinations:

WD Elements TM [Local] (Last used)

Total size: 2 TB

Total number of backups: 62

Oldest backup: 2013-02-04 23:03:41 +0000

Last backup: 2013-11-08 17:25:16 +0000

Size of backup disk: Adequate

Backup size 2 TB > (Disk used 473.03 GB X 3)

/sbin excluded from backup!

/usr excluded from backup!

/System excluded from backup!

/bin excluded from backup!

/private excluded from backup!

/Library excluded from backup!

/Applications excluded from backup!

I had read about these errors (system files not being backed up) over the last few days and had satisfied myself that this machine was not infected. Actions included running Pondini's applet, which showed 'clean'.

I ran the applet again and it showed 'bitten'. I then checked in Time Machine. The back up was correct at 17:25 on 8 November but flawed this morning.

The computer had been used for nothing since about 17:00 8 November until I woke it this morning; all I did then was run EtreCheck and reboot. This cycle had been operated many times over the last week while checking for Mavericks errors, cleaning up the machine and rolling back to Mountain Lion from a Carbon Copy clone.

So the error was introduced between 06:53 and 07:04 on 9 November when essentially nothing was being done with the computer. On reading this thread and Pondini's web site again, the only possible culprit I can think of it Sophos anti-virus. Quote Pondini: "In addition, similar behavior seems to occur when installing Sophos (and perhaps other) anti-virus updates, which apparently can happen hourly."

I then tried to report the bug to Apple as directed by Pondini, but the bug reporting process has a bug, so I reported that instead!

I post this in the hope that it might help diagnosis. I'd like to add that I was sorry to learn of Pondini's death about the same time that I learned of him and began to learn so much from him. RIP.

Message was edited by: STCav to add time zone info.

To my horror I recently found that my /Applications folder was not being backed up since August. Thanks to this thread and a few other similar ones I found a long list of ExcludedByPath in my com.apple.TimeMachine.plist.

Apart from saying thanks to all those contributors I wanted to adda few comments that may help the detective work. I'm running Mavericks now - but at the time this went 'pear shaped' I was probably running Mountain Lion.

a) I have been running Sophos anti-virus - I recently uninstalled it as I was trying to sort out an App Store install problem and Sophos was clouding the issue with all its log entries. These installation problems were why I spotted the broken backup issue.

b) The excluded paths mentioned a whole load of project folders I had been using with XCode (Applescript, Objective-C and iPad projects) as well as /Library, /Applications, /System, /bin, /sbin, /private, /usr - a total of 40+ folders.

c) The excusions you add manually go into the SkipPaths section not the ExcludeByPath section

d) Deleting the plist file doesn't always work as there seems to be a cached version somewhere and so the old one keeps being resurrected. Make sure the Preferences Panel is closed, backups are off and the backup device is unmounted (possibly even disconnected) before deleting the file - and then restart immediately after. On restart I ignored the offer of the system automatically starting time machine backups to my mounted external disk and selected the disk manually from System Preferences - then started a backup. It has found all the old backup points and is now doing a 80GB backup - presumably of all the stuff that has changed in all those folders not backup up since August.

e) Other than these mysterious exclusions, the other files have been backed up fine, which is part the reason I've not spotted it until now

Fingers crossed!

I came across this discussion after discovering today that /Applications on my new rMBP is not backed up by Time Machine (to Time Capsule). rMBP runs Mavericks and I used Migration Assistant to move everything from an older MBP.

I have other issues with Time Machine. For example, I don't see backups of individual emails in Mail app.

I'm trying now the solution of deleting the plist. After deleting the plist and re-starting backup, Time Machine has new backups of about 75GB. Hopefully, that would solve the problem.

I have an issue that is superficially similar, but different in a key aspect. I posted here: https://discussions.apple.com/thread/5670458?answerId=24141585022#24141585022&ac_cid=op123456#24141585.

The major differences are that I'm only missing the hidden system folder like /usr and /sbin. I still see /Applications, /System and /Library. I also don't see the ExcludeByPath key in my plist.

Looks like I may have this issue but it's far too late to do anything now.

I was messing around with terminal commands this morning (I know I know) trying to get dropbox to work like it should (automatically syncing from documents). Any way, long story short I messed up a symlink or something and deleted the documents folder periodically.

Not a problem I thought, I'll just restore it from a backup. The latest one available was from October, I thought it was strange but because I could see the latest backups in time machine I thought I could select the latest backup after restoring or something.

Turns out I was wrong.......

Absolutely infuriating that a backup system fails to function and doesn't even tell you.