Yosemite and Time Machine

In summary - IMHO at least one BUG - in Yosemite the space calculations for Time Machine don't always give the right answer, so TM may delete backups it doesn't need to or erroneously report there isn't enough space.

If you look at the console log you'll see that first thing TM does is calculate how much space it MAY need. In Yosemite this is sometimes way off, by orders of magnitude. E.g a couple of days ago TM stated that it needed hundreds of GB to backup my rig, in the end it used tens of MB.

Now, if the (incorrect) calculation is more than the current free space it will start deleting backups until there is sufficient space. However, it doesn't delete the latest backup so you can end up with the size of that plus the incorrect calculation being larger than the size of the target disk. In this case you get the message about there not being enough space.

With my external drives, I always get the wrong answer being calculated when TM runs following a mount of a drive that was dismounted the last time TM ran.

Now I've thought about it a bit more, as far as the effect of dismounting/mounting external disks goes, I now think its deliberate. Not right but probably still deliberate.

It seems to me that its an 'optimisation' introduced in Yosemite so that Time Machine can start 'faster'. Rather than do the deep scan which was performed in Mavericks, just assume that all the content has changed and get on with it.

Simple example shows why this algorithm can never reliably work. You have 2TB of data to back up with a flat 10% per month rate of change, half of which are new files. You'd like at least a years worth of backups so you set TM disk at 4GB and back everything up. You run for two months without ever dismounting the disk so you are now using c2.4GB on the TM disk and 2.2GB on the source.

You dismount the disk, TM runs, you remount the disk having changed nothing, TM runs and the 'fast' calculation is that it needs 2.2GB (plus padding) rather than the virtually zero thats actually required. However, there is only c1.6 GB available - it deletes 0.2 GB from TM by thing backups (the other 0.2GB is the new files) - still not enough - you have lost all but the latest backup and are 'Out of space' even though only a trivially few bytes are actually required, those for the new folder. There is now no option to erase the TM disk and start again.

Why do I think this - its exactly what happens when you remove something from the exclusions list and the 'deep scan' messages have disappeared from the console logs.

Now I'd love Apple to either contradict, in which case they should fix the bug or to revert to an algorithm that works all the time.

Send Apple feedback. They won't answer, but at least will know there is a problem. If enough people send feedback, it may get the problem solved sooner.

Feedback

Thanks, did that days ago.

Have to say though that my experience is that unless its a suggestion on how to make something thinner they tend to ignore it.😉

I'm experiencing the same exact issue: if I disconnect my external USB drive and reconnect it the next morning, Time Capsule then reports "not enough space".

Following advices on forums, I've deleted all files on the Time Capsule and started over: the initial back (~570GB) took a while (3 days over ethernet!) but it worked. For the next few hours, incremental backups happened fine too.

Then I disconnected the USB drive again. This morning I plugged it back. And the issue reappeared: "not enough space" :-(

The issue started after I upgraded to Yosemite.

TC was working fine before. I've had that external USB drive for several years and never encountered that issue. There is no reason to want to do a full backup again of that USB drive. My TC is 1TB, my Mac has 70GB of data and the USB drive has 500GB. TC has 420GB of free space. I have not added any new file to the USB drive. The only thing I've done is disconnect/reconnect (with a few hours in between).

I've also reported the issue to Apple. Let's hope they fix this soon or that a workaround exists (to for TM to do a full scan).

Backups are critical and part of why OSX is a great OS.

I basically can't use TC anymore, at least not with an external drive (I can't guaranty it stays connected 100% of the time... especially since my Mac is a laptop).

I'm convinced they rewrote it completely for Yosemite.

• Users requirements for a backup system - Security
• Apples design criteria for the new version - Raw speed at any cost
• What utter rubbish is that

/End Rant

It seems that you only get this behaviour if Time Machine runs while the disk is dismounted.

A few possibilities:

1. You may not have to start again (again) as the last backup should still be on the TC. Turn Time Machine Off., Use the Time Machine options to exclude the USB drive BY FOLDERS (probably top layer or second down depending on how you have the drive organised). Run Time Machine. Remove one of the folders from the exclusions, run Time Machine again, should only backup any delta in the newly enable folder. Repeat until the whole drive is being backed up. Key is that when you re-enable a folder that the total size contents including any sub-folder is less than the remaining space in the TC. Just to reiterate its the total size rather than the anticipated delta as its the total size that Time Machine is using in the calculation.
2. Assuming it works, you could do this every time you mount the drive !! You might be able to write a script for this (beyond my capabilities)
3. If it is the case that this occurs if Time Machine runs when the drive is dismounted then you could turn off Time Machine before you dismount the drive. Turn it back on when the drive is remounted. Of course you won't be backing up the changes on the built-in drive for that time. (see last line of rant above)

PS, all the sizes in the example I wrote should be TB, doesn't make sense otherwise

I've tried possibility #1 and if failed:

I excluded a top folder on that drive that is 280GB. That is enough for in theory make TM think there is enough room left on the TC.

TM did run a backup but it announced that it was backuping 260GB, which is basically the size of all other data to backup on that USB drive.

In other words, TM was doing a "full backup" of that drive :-( (because I'm positive there can't be 260GB of changes coming from elsewhere, my Mac's hard drive is way smaller than that).

I cancelled that backup, I can't let my TC get filled with redundant full backups.

My solution for now is to no longer backup that external drive with TM. I'm going to look into alternatives, like rsync. But it's far from ideal (to say the least). Hopefully, if Apple did a full rewrite, it's full of issues and they'll end up fixing enough of them for my backups to work again (soon?).

This is very frustrating because backups are essential. That's what had make OSX great all those years: it came with a fully functional backup solution that is easy to use (which is also important otherwise people rapidly stop backuping things).

How long did you let the backup run for? Reason I ask is that for me its been the calculation thats wrong; not the subsequent backup.

If it was a very short time before you aborted it, might be worth trying again and looking at the Console log as it starts. Run Console from Applications\Utilities and filter on backupd (box in top right) and you should see something like the below. Note I've numbered the lines so I can reference them so 1., 2., etc won't appear in the log.

In this example ScratchUse had been dismounted while a Time Machine backup ran, was mounted again with 'Music and Video' mounted the whole time. The log is the next occasion that Time Machine ran. You can see the overall calculation includes the c45GB thats on ScratchUse (and already backed-up) but it only wrote 5.4MB of the 55GB. The number that was reported in Preferences and on right-click of the menu icon was the 55GB. After that it was all fine next occasion Time Machine ran.

What you want to ideally want to see is a small number in brackets the first time your external appears in the log (i.e like line 2). If all the numbers like those in lines 1. to 3. are small then the backup should run normally and the corresponding values like those in lines 6. to 8. should be small as well. If the number on your equivalent of line 2. is large then there is something else going on.

1. 27/10/2014 17:41:53.780 com.apple.backupd[1147]: Will copy (5.4 MB) from Macintosh HD
2. 27/10/2014 17:41:53.791 com.apple.backupd[1147]: Will copy (Zero KB) from ScratchUse
3. 27/10/2014 17:41:53.792 com.apple.backupd[1147]: Will copy (Zero KB) from Music and Video
4. 27/10/2014 17:41:53.793 com.apple.backupd[1147]: Found 108 files (45.51 GB) needing backup
5. 27/10/2014 17:41:53.803 com.apple.backupd[1147]: 55.87 GB required (including padding), 1.98 TB available
6. 27/10/2014 17:41:56.041 com.apple.backupd[1147]: Copied 91 items (5.4 MB) from volume Macintosh HD. Linked 1851.
7. 27/10/2014 17:41:59.404 com.apple.backupd[1147]: Copied 3 items (234 bytes) from volume ScratchUse. Linked 4.
8. 27/10/2014 17:42:00.224 com.apple.backupd[1147]: Copied 1 items (Zero KB) from volume Music and Video. Linked 6.
9. 27/10/2014 17:42:01.881 com.apple.backupd[1147]: Created new backup: 2014-10-27-174201

Good catch, thank you for insisting. I've tried again and the backup worked and ended when saying "130MB of 246GB".

2014-11-05 12:11:17.125 PM com.apple.backupd[9996]: Will copy (Zero KB) from Macintosh HD

2014-11-05 12:11:24.525 PM com.apple.backupd[9996]: Will copy (Zero KB) from MEDIA

2014-11-05 12:11:24.673 PM com.apple.backupd[9996]: Found 323220 files (223.93 GB) needing backup

2014-11-05 12:11:25.879 PM com.apple.backupd[9996]: 246.75 GB required (including padding), 426.13 GB available

2014-11-05 12:13:48.954 PM com.apple.backupd[9996]: Copied 1886 items (43.1 MB) from volume Macintosh HD. Linked 8555.

2014-11-05 12:15:49.173 PM com.apple.backupd[9996]: Copied 67 items (87.1 MB) from volume MEDIA. Linked 1549.

2014-11-05 12:16:19.981 PM com.apple.backupd[9996]: Created new backup: 2014-11-05-121619

So this is good news, at least I can on a regular basis "re-sync" my external drive... Althought it's time consuming and I'm sure I'll probably drop the ball down the road. Let's hope for a fix before that happens.

What's puzzling from those logs is that we see 3 very different numbers!

1) "Will copy" (which was zero BK in my case)

2) Found 223GB needing backup (completely off)

3) What really happened: 130MB total were backed up

I'm not sure where the "will copy" numbers come from... Indeed, the crazy part if (2), especially since it's the number used to assess whether or not there is enough room to perform the backup. Sounds like really bad logic. Who does not use external drives, especially when SSD remain so expensive? Also who would let an external drive always connected?

It is really, really, really,..., really bad logic. I change my earlier opinion - its beyond utter rubbish.

Basically what seems to have happened is:

In Mavericks

1. When a disk was re-mounted after missing a backup

• (A) in the first phase, while its doing the calculation it did a deep scan of the drive and got precise figures to calculate the space requirement. I've seen this take better part of an hour.
• (B) in the second phase, while doing the copies, it did a recalculation of files that had changed while it was running and included these in that backup

2. When you removed something from the exclusions in prefereces

• (A) Presumption is you excluded it, nothing will ever have been backed up. So in the space calculation include the full size of the folder (and sub-folders); ignoring whether any files had already been backed up. A very quick calculation but could be wrong
• (B) when doing the backup, work out which files actually need copying.

In Yosemite

1, When a disk is is remounted

• Seems always to report Zero in the will copy line, however
• Use algorithm 2A above to get to the total space requirement (i.e. full folder size)
• Very quick calculation as it doesn't need to scan the drive so it can start backing up files earlier but,
• Almost guaranteed to be wrong
• Way-way out if the disk has already been backed up (about 223GB wrong in your example)
• Execute as per 2B which is why it only copied 87MB in your example
• Speed again, ought to be faster as you only need to scan each folder once (during copy) rather than one and a bit times (once for scan plus one more time for anything copied)

2. When you remove something from the list of exclusions

Do as it dd in Mavericks

3. Apple removed 1B, so it no longer does the rescan towards the end of the backup. Finishes quicker but now there are files that have changed but aren't backed up until next time TM runs

For what its worth this is the list of bugs I've experienced in Time Machine:

• This discussion - On remount of disk, uses full size (including data already backed-up and not needing backup) rather than delta
• Backs up partial files if large copy is running in parallel with the backup and the copy doesn't complete while Time Machine is doing something else (used to continue running and backup the whole file)
• Hit or miss whether you can see backup history through Time Machine UI
• Deletes backups <24 hours old
• Time of next backup in Preferences bears little relation to time it runs (can be +- 30 minutes either way)
• Skips backups randomly (used to produce message if skipped because already backed-up in previous 10 minutes)
• Unpredictable behaviour messages in log

New but IMHO useless feature

Thins backups in reverse order so deletes the latest backup first

All newly introduced with Yosemite.

By the time you gate the 'Out of space' message it has deleted all but your latest backup

Provide you still have enough space you could now remove the exclusion and backup that up as well. Will be similarly wrong message...

"So this is good news, at least I can on a regular basis "re-sync" my external drive... Althought it's time consuming and I'm sure I'll probably drop the ball down the road."

Will help you if:

• You do the exclusion before you dismount the disk
• Turn Time Machine off before you remount the disk and check the exclusions before you Turn Time Machine back on

So the solution of excluding some large folders to re-add them later does not seem to work as well as expected.

As reported previously, by exlusing a 280GB folder, I was able to have the backup of my external drive go through (minus that excluded folder), in an incremental way (only 87MB really needed to be backed up on that external drive). That's were I left things in my previous comment.

Since then, I have removed that 280GB folder from the exclude list. Since the TC has plenty of space (425GB), I was hoping things to work like previously: TM would say "I will backup 280GB" but would end up backup a less lot (pretty much nothing since I have not made any change to that folder in the last few days).

But this is not what happened. It started to backup and is now at 32GB out of 280GB, 3000 files (out of 6000) were backed up already... I'm cancelling it as it will fill my TC for nothing.

In conclusion, removing exclusions and re-adding them later seems to confuse TM and make it think it does not have any previous backup for that non-longer-excluded folder, even if it was backed up previously (before being added to the exclude list). Maybe it's normal and expected behavior?

All in all, I'm stucked and am giving up the hope of backuping that 280GB folder for now :-(

Rats... looks like its unreliable as I did have it work once

Means I have to be ultra cautious as with one drive now back to being past the point if no return if I had same happen. Could be yet another seven day rebuild.

Wasn't normal behaviour and if it is now meant to be then you will never be able to run Time Machine safely for external disks.

For example, to keep third party products out of the picture, predicted behaviour would be:

• You forget to turn Time Machine off
• You run Disk Utility and then use Verify or Repair disk
• Disk Utility dismounts the disk
• Time Machine runs during Disk Utility execution
• Disk Utility remounts the disk
• Time Machine runs
• Bang...

If Apple does monitor these threads then someone ought to be deeply ashamed

Unfortunately I don't think they do. They have our money already. My problem is a little bit different, but also involves time machine. Time machine is apparently backing up to my backup drive, but I can't seem to access these backups. On my first Yosemite install, I chose to install over Mavericks. Big mistake, ran like molasses. On my 2nd install, I chose to clean install, and use migration assistant. Little faster, but still slow. My third install I clean installed, and didn't use migration assistant, and it runs a lot faster, but I cannot access time machine. W/O time machine the machine is pretty useless. I was planning on carefully moving them myself manually after the install was finished, because I probably brought something back that slowed down my machine, when I used migration assistant. I have documents, and apps on there, that I've been using for the past 5 years. Time machine is still backing up, since I still have a backup drive wt backups from today, but Time machine is not in the dock itself. When I choose Time Machine in System Preferences, it just brings me to either the On or Off screen wt Select Disc, and options. It is On, and the Backup Disc is selected. Also in my Backup Drive, I can view backups done within the past hour, but I don't see the Time Machine icon in the dock, or desktop. When I choose Time Machine from my apps, it opens a completely empty Time Machine. Completely confused ..

Not something I've ever come across, but suggest having a read through this https://developer.apple.com/library/mac/documentation/Darwin/Reference/Manpages/ man8/tmutil.8.html

especially the text under associate disk and inheritdisk.

Note though its the 'Mavericks' version; can't see that Apple updated them anywhere. The pages on Unix.com seem identical at first glance