Best practices for storage and backups on internal and external drives

First, to the ExFAT being the problem, I would suggest the drive is just physically going bad. ExFAT is perfectly acceptable as a format where both Mac and PC can read/write. You may have overreacted a bit by abandoning ExFAT for HFS, but you made your choice and will adjust to it.

This is how I operate my backups:

I use CCC (CarbonCopyClone, $40, bombich.com). It copies the boot sectors, disk mapping, Recovery Partition (these three are "invisible" to DiskUtility mostly) as well as the system/data partitions to "other" disks. These other disks can be internal for clone-to-test-new-OSX in multi-disk systems, or external for backup purposes. The beauty is that you can boot an external clone in a system crash, then CCC will re-copy the active external clone onto a new/fixed internal disk.

Encryption is not something I know, but ... if just the contents of the system/data partition is scrambled, it *might* be OK to clone-back from encrypted-partition to partition-as-encrypted. If you are cloning back to internal from external the external partition is active, and copying the "encrypted map" while small adjustments are made while running could be bad. Copying changing files on my unencrypted partitions does not cause issues because they are individual files.

As an alternative, you could have one system drive that only has CCC loaded, leave the encrypted clone and encrypted internal idle and just copy the entire map with the "third system" active.

I personally believe that any good Mac home backup plan has at least these types of backups:

1) Local TimeMachine backups (either to desktop attached storage such as an external drive, or to a Time Capsule)

You can't beat TimeMachine for a real-time backup that allows rolling back when necessary

2) Bootable clones of your boot and external drives (as steve359 suggests)

At a minimum, you should clone your boot drive before any major OS update just in case something goes wrong and you need to recover quickly. Either CCC or SuperDuper! work very well.

3) Off-site Backups are a NECESSITY that many people forget completely. If you have a disaster or theft at your house will it take your Mac and all backups with it?

By far the best modern solution to off-site backups are cloud backup services like Backblaze or CrashPlan.

Lastly, when devising a backup scheme I have found one very important consideration. It is very easy to completely backup both internal drives and directly attached (e.g. FireWire, USB3, Thunderbolt) external drives using TimeMachine and cloud backup services. However, it is very difficult to fully backup a local NAS to either of these. Therefore, you need to consider this carefully if you are considering a NAS for storage (remembering that RAID is NOT backup).

Good luck. It takes a lot of careful consideration of your particular storage needs to feel confident in your backup strategy. But, if you have TimeMachine, cloned drives, and offsite backups, you will be better off than 99.9% of all home users.

TimeMachine has the advantage of "background processing". That means it can quietly move unused files that have changed in the last hour. But it is not "directly bootable" as an active system.

Clones are bootable copies, complete with boot sectors and copies of the Recovery Partitions. But I like to have the system "running, but quiet" to get the best copies of all files.

As to "offsite" ... you can accomplish that by rotating sets of clones to a friend's house, in case of fire or theft. Online backups take longer to load back down, while physical clones only need at worst a short drive.

I still think another needs to chime in about "encrypted partitions". My partitions are not encrypted, so the directory structure can be copied file-by-file, while "encryption" tends to "map" the partition contents and you cannot see the directory structure from the outside of the encryption program. This keeps things safe, but if the "map" is corrupted you lose the entire partition. My files can obviously be read if the disk is mounted as a "generic external" which is a drawback.

My concern is that you may need a third "CCC only" copy of an OSX so that both clone-target and internal-disk are both completely-idle when making the clone so that the "map" is not corrupted. But I may be wrong.

Keep it simple. Time Machine to an encrypted volume, either directly-attached or via Time Capsule.

We all get busy, and many of us don't run manual backups as often as we should. Any backups beat no backups. Recent backups beat old backups. Current backups beat recent backups.

Time Machine is built in, works, can be encrypted, and there are folks here that can usually help if something goes sideways. Yes, once in a while TM gets itself tangled or a TM target disk itself gets corrupted or fails, so either performing an occasional Time Capsule archive or having multiple TM disk or Time Capsule targets can help.

Off-site is nice to have, once basic backups are sorted. If the laptop comes and goes to an office or other location, then there's a potential TM offsite backup, if you can configure a disk or a Time Capsule there. It's also possible to rotate TM backups, though disks do tend to get bumped or dropped.

If your network bandwidth is commensurate with your particular volume of data, uploading some or all or just the most critical data can be appropriate, too. A service such as SpiderOak targets folks fond of encryption, for instance. Or get your own hosting and your own storage, AES-encrypt the data, and upload it yourself — if you wanted to write or acquire the necessary scripts, and schedule the scripts to run. (Again, manually triggered backups are only any good when they are reliably used.)

Disk clones don't get you any depth of backup. You get one copy. If your clone had the bad or corrupt file, you're toast. They're also wasteful of disk space, unless you go to a non-bootable compressed-format archive of some sort. (Then you get to figure out how you'll recovery that, if your OS X boot disk is corrupted, or your Mac gets lost or stolen or irreparably damaged.)

It's quite correct that TM isn't directly bootable, but having a local bootable USB key disk is trivial to set up and use, and many of the Mac systems now support Internet Recovery. OS X recovery mechanisms, as well as OS X migrations and upgrades can all source their input data from Time Machine backups. With third-party backup tools or with clones using dd or Disk Utility or other tools, you'll want to determine how to recover that data, and then use it to perform the install or upgrade or migration.

All backups do occasionally fail, and not enough folks test their own recovery path.

If you're not exchanging data with other systems via external device via the particular device, I'd not use FAT or ExFAT. I'd use GPT-partitioned HFS+-formatted disks; native-format OS X disk volumes.

Encrypted internal disks via FileVault 2 and via encrypted external backups — assuming the encryption keys are sufficiently robustly chosen, and are not forgotten — beats having unencrypted backups, particularly if the Mac or the disk gets stolen or (as all hardware eventually becomes) scrapped.

All storage devices eventually and inevitably fail. Application and system upgrades fail. Volumes get corrupted. Keys get forgotten. Mistakes happen. Etc.

Having any backups beats having no backups.

celio-likes-apples wrote:

As for CCC, what's the difference from Time Machine?

CCC is not built into OS X, you have to configure it for your local requirements, and various of the Apple tools don't directly interoperate with it. CCC works, and I know of a number of folks that use it.

This latter also allows a full system recovery after booting with OS X System Utilities (Cmd+R), right?

Yes, TM allows full system recovery as well as migration to a new or different system, or a migration into a freshly-installed OS X boot disk. The TM backups are not directly bootable. As for restoring or migrating a TM backup, the use of the recovery partition of or a bootable copy of OS X on a flash drive or other secondary disk is sufficient. If I want or need a directly bootable backup, CCC could be used, but Disk Utility can build a bootable backup, too. (I'll generally use a direct, bootable copy when I either need the speed — TM is slower — and/or when I don't need the depth of backups that's provided by TM.)

TimeMachine needs OSX to be re-installed to properly re-lay the TM contents. A TM backup is not by itself bootable. "Yes" is the answer to your second question.

CCC makes a bootable clone of a drive that can let you evaluate a system that does not boot on internal ... is the screen working?, do the USB ports work?, is the internal drive readable?, etc. Clones usually have the CCC loaded on it because clones are copies of the cloned drive, so can "clone themselves back onto" the internal drive with minimal or no loss (depending on how much changed between "clone/backup" and "now".

Here is work from a former member (now deceased) of ASC who knew TM better than Apple probably does: http://pondini.org/OSX/Home.html

Some never use TM because it needs OSX reinstalled first, other see nothing wrong with it and will not pay $40 for CCC. As long as one backs up files, they are not wrong.

Local snapshots usually go to the TM drive, when connected. When the drive is not attached they accumulate. Read Mr. Pondini's excellent documentation.

CCC will replace old versions of files with new versions, saving old versions in separate folder on that drive. CCC incremental drives can seem to take forever ("looking for files to backup") but will only copy changed files.

Internet recovery of files, in my non-experienced opinion, still require the internet and can take longer to recover than attached physical drives. But that is just my opinion.

Plea

Any backup tool — when there's a software problem, whether due to failed hardware or a misconfiguration — can fail. Somebody that filled their disk because of Time Machine is something well worth considering. Are you likely to ignore backup errors? If you are, then consider that you can ignore cloned backups, too. It's trivial to end up with no backups. Which is what you appear to have now, and your data is at risk. Right now.

Yes, with backups, there are many choices and many options.

The only absolutely wrong choice for your critical data is having no backups of your critical data.

Time Machine works fine. Once in a while, it can get corrupted. Any backup can. Having more than one TM backup volume can help with that. Folks with laptops can use a backup at home and a backup at the office, which gets you two copies, and an off-site copy.

Incremental backups do not exist with classic disk clones. Cloned disks have no depth of backups, unless you have multiple target backups. And if you have multiple backup targets, you probably want multiple disks — backup disks can get corrupted, and can fail, just like Time Machine backups. There are ways to do incremental backups — Time Machine is one example — but those are not usually directly bootable.

"Cloud services" — what used to be called a service bureau or a hosted provider — are what they provide — there are a zillion different offerings here, ranging from storage where you dump a zip of a clone, to a server that you use rsync or some other remote backup tools, or cases where you use some cloning tool and simply mail the backup disk or even backup tape around. You get to figure that out for yourself, your data, your budget, and some consideration around how good you are at following a backup schedule.

When looking at hosted storage, also calculate how much time it will take to perform a network backup or a network restore. Some Internet links will take a very long time, for a sufficiently large volume of data. In other words, compare the size of your disk(s) with the speed of your network link, and round down (possibly round down substantially)...

Also consider what your state of mind will be when your system fails or your system is lost or damaged, and you have to get your data restored. Do you want to map out the process, or just see if it works... This is why most folks recommend testing your recovery processing, before you need it.

Time Machine supports remote backups to OS X Server as a NAS, and TM also supports remote backups to an Apple Time Capsule.

In general: the only wrong answer is no backups. Get something going now.

To reinforce something MR. Hoffman said ...

"Also consider what your state of mind will be when your system fails or your system is lost or damaged, and you have to get your data restored. Do you want to map out the process, or just see if it works... This is why most folks recommend testing your recovery processing, before you need it."

You do not want to be "looking for the notes someone gave me" when your system just crashed and you need the system restored safely and quickly. Know and even test, if you can, your recovery procedures.

Performing remote incremental copies with any backup is a difficult problem, if you want it to work reliability.

The remote copy is inherently slower than any local link, and that means remote copies are much more prone to glitches and outages than local links, and the bandwidth limitations mean that more "clever" solutions can be chosen by the developers. Recovering from a glitch is non-trivial, too. The most certain way is to recopy everything. With a more "clever" solution, the developer then has to detect and recover from interruptions and corruptions that can arise within the data transfers, or errors that can arise on the local system, or errors that can arise with the remote backup.

Terms like "cloud" and "best practices" tend to obfuscate the local requirements and expectations, unfortunately. They don't make the issues of network glitches, network bandwidth nor data integrity or data security requirements go away, either.

Get backups going. At least one backup, performed automatically. Anything beyond that is site-specific.