celio-likes-apples

Q: Best practices for storage and backups on internal and external drives

Hello,

 

I have a Mac laptop and I would like your advice on how to organise file storage on the internal and external drives and how the optimise the backup plan.

 

As of now, my file storage organisation is as follows:

 

I keep my most important files on the internal encrypted SSD drive and less critical files on an external encrypted drive (ExFAT formatted). This external drive has been giving errors so I bought a new LaCia 2 Tb drive which I formatted in Mac OS journaled with encryption instead of ExFAT, to replace it.

 

For backups, I don't use Time Machine, but I use an app that synchronizes files between two drives. Using this app, I first sync the critical document files on the internal SSD to the external drive, and then I sync the entire external drive to another external drive.

 

My question is, should I keep this file storage organisation or should I, for instance just move all my files, critical and non critical to the internal SSD (I could make enough room for that) and then backup up the entire Mac with Time Machine? Or there is another optimal way to organise the files within this scenario?

 

The second question is, if it's best to keep the storage split organisation as is (critical on internal, less critical on external) what would be the best way to backup everything?

 

Thanks in advance for all contributions.

MacBook Pro with Retina display, OS X El Capitan (10.11.2)

Posted on Dec 17, 2015 8:35 PM

Close

Q: Best practices for storage and backups on internal and external drives

  • All replies
  • Helpful answers

  • by steve359,Helpful

    steve359 steve359 Dec 26, 2015 11:18 AM in response to celio-likes-apples
    Level 6 (14,032 points)
    Dec 26, 2015 11:18 AM in response to celio-likes-apples

    First, to the ExFAT being the problem, I would suggest the drive is just physically going bad.  ExFAT is perfectly acceptable as a format where both Mac and PC can read/write.  You may have overreacted a bit by abandoning ExFAT for HFS, but you made your choice and will adjust to it.

     

    This is how I operate my backups:

     

    I use CCC (CarbonCopyClone, $40, bombich.com).  It copies the boot sectors, disk mapping, Recovery Partition (these three are "invisible" to DiskUtility mostly) as well as the system/data partitions to "other" disks.  These other disks can be internal for clone-to-test-new-OSX in multi-disk systems, or external for backup purposes.  The beauty is that you can boot an external clone in a system crash, then CCC will re-copy the active external clone onto a new/fixed internal disk.

     

    Encryption is not something I know, but ... if just the contents of the system/data partition is scrambled, it *might* be OK to clone-back from encrypted-partition to partition-as-encrypted.  If you are cloning back to internal from external the external partition is active, and copying the "encrypted map" while small adjustments are made while running could be bad.  Copying changing files on my unencrypted partitions does not cause issues because they are individual files.

     

    As an alternative, you could have one system drive that only has CCC loaded, leave the encrypted clone and encrypted internal idle and just copy the entire map with the "third system" active.

  • by FishingAddict,Helpful

    FishingAddict FishingAddict Dec 26, 2015 11:23 AM in response to celio-likes-apples
    Level 4 (1,586 points)
    Mac OS X
    Dec 26, 2015 11:23 AM in response to celio-likes-apples

    I personally believe that any good Mac home backup plan has at least these types of backups:

     

    1) Local TimeMachine backups (either to desktop attached storage such as an external drive, or to a Time Capsule)

    You can't beat TimeMachine for a real-time backup that allows rolling back when necessary

     

    2) Bootable clones of your boot and external drives (as steve359 suggests)

    At a minimum, you should clone your boot drive before any major OS update just in case something goes wrong and you need to recover quickly.  Either CCC or SuperDuper! work very well.

     

    3) Off-site Backups are a NECESSITY that many people forget completely.  If you have a disaster or theft at your house will it take your Mac and all backups with it?

    By far the best modern solution to off-site backups are cloud backup services like Backblaze or CrashPlan.

     

    Lastly, when devising a backup scheme I have found one very important consideration.  It is very easy to completely backup both internal drives and directly attached (e.g. FireWire, USB3, Thunderbolt) external drives using TimeMachine and cloud backup services.  However, it is very difficult to fully backup a local NAS to either of these.  Therefore, you need to consider this carefully if you are considering a NAS for storage (remembering that RAID is NOT backup).

     

    Good luck.  It takes a lot of careful consideration of your particular storage needs to feel confident in your backup strategy.  But, if you have TimeMachine, cloned drives, and offsite backups, you will be better off than 99.9% of all home users.

  • by steve359,

    steve359 steve359 Dec 17, 2015 10:43 PM in response to celio-likes-apples
    Level 6 (14,032 points)
    Dec 17, 2015 10:43 PM in response to celio-likes-apples

    TimeMachine has the advantage of "background processing".  That means it can quietly move unused files that have changed in the last hour.  But it is not "directly bootable" as an active system.

     

    Clones are bootable copies, complete with boot sectors and copies of the Recovery Partitions.  But I like to have the system "running, but quiet" to get the best copies of all files.

     

    As to "offsite" ... you can accomplish that by rotating sets of clones to a friend's house, in case of fire or theft.  Online backups take longer to load back down, while physical clones only need at worst a short drive.

     

    I still think another needs to chime in about "encrypted partitions".  My partitions are not encrypted, so the directory structure can be copied file-by-file, while "encryption" tends to "map" the partition contents and you cannot see the directory structure from the outside of the encryption program.  This keeps things safe, but if the "map" is corrupted you lose the entire partition.  My files can obviously be read if the disk is mounted as a "generic external" which is a drawback.

     

    My concern is that you may need a third "CCC only" copy of an OSX so that both clone-target and internal-disk are both completely-idle when making the clone so that the "map" is not corrupted.  But I may be wrong.

  • by MrHoffman,Solvedanswer

    MrHoffman MrHoffman Dec 19, 2015 11:28 AM in response to celio-likes-apples
    Level 6 (15,637 points)
    Mac OS X
    Dec 19, 2015 11:28 AM in response to celio-likes-apples

    Keep it simple.  Time Machine to an encrypted volume, either directly-attached or via Time Capsule. 

     

    We all get busy, and many of us don't run manual backups as often as we should.  Any backups beat no backups.   Recent backups beat old backups.   Current backups beat recent backups.

     

    Time Machine is built in, works, can be encrypted, and there are folks here that can usually help if something goes sideways.   Yes, once in a while TM gets itself tangled or a TM target disk itself gets corrupted or fails, so either performing an occasional Time Capsule archive or having multiple TM disk or Time Capsule targets can help.


    Off-site is nice to have, once basic backups are sorted.   If the laptop comes and goes to an office or other location, then there's a potential TM offsite backup, if you can configure a disk or a Time Capsule there.  It's also possible to rotate TM backups, though disks do tend to get bumped or dropped.


    If your network bandwidth is commensurate with your particular volume of data, uploading some or all or just the most critical data can be appropriate, too.   A service such as SpiderOak targets folks fond of encryption, for instance.   Or get your own hosting and your own storage, AES-encrypt the data, and upload it yourself — if you wanted to write or acquire the necessary scripts, and schedule the scripts to run.   (Again, manually triggered backups are only any good when they are reliably used.)

     

    Disk clones don't get you any depth of backup.  You get one copy.   If your clone had the bad or corrupt file, you're toast.   They're also wasteful of disk space, unless you go to a non-bootable compressed-format archive of some sort.   (Then you get to figure out how you'll recovery that, if your OS X boot disk is corrupted, or your Mac gets lost or stolen or irreparably damaged.)

     

    It's quite correct that TM isn't directly bootable, but having a local bootable USB key disk is trivial to set up and use, and many of the Mac systems now support Internet Recovery.   OS X recovery mechanisms, as well as OS X migrations and upgrades can all source their input data from Time Machine backups.   With third-party backup tools or with clones using dd or Disk Utility or other tools, you'll want to determine how to recover that data, and then use it to perform the install or upgrade or migration.

     

    All backups do occasionally fail, and not enough folks test their own recovery path.


    If you're not exchanging data with other systems via external device via the particular device, I'd not use FAT or ExFAT.    I'd use GPT-partitioned HFS+-formatted disks; native-format OS X disk volumes.

     

    Encrypted internal disks via FileVault 2 and via encrypted external backups — assuming the encryption keys are sufficiently robustly chosen, and are not forgotten — beats having unencrypted backups, particularly if the Mac or the disk gets stolen or (as all hardware eventually becomes) scrapped.

     

    All storage devices eventually and inevitably fail.   Application and system upgrades fail.   Volumes get corrupted.   Keys get forgotten.   Mistakes happen.  Etc.

     

    Having any backups beats having no backups. 

  • by celio-likes-apples,

    celio-likes-apples celio-likes-apples Dec 26, 2015 11:22 AM in response to steve359
    Level 1 (0 points)
    Dec 26, 2015 11:22 AM in response to steve359

    Hello Steve,

     

    Thanks for answering. As for CCC, what's the difference from Time Machine? This latter also allows a full system recovery after booting with OS X System Utilities (Cmd+R), right?

  • by MrHoffman,

    MrHoffman MrHoffman Dec 26, 2015 11:48 AM in response to celio-likes-apples
    Level 6 (15,637 points)
    Mac OS X
    Dec 26, 2015 11:48 AM in response to celio-likes-apples

    celio-likes-apples wrote:

     

    As for CCC, what's the difference from Time Machine?

     

    CCC is not built into OS X, you have to configure it for your local requirements, and various of the Apple tools don't directly interoperate with it.   CCC works, and I know of a number of folks that use it.

     

    This latter also allows a full system recovery after booting with OS X System Utilities (Cmd+R), right?

     

    Yes, TM allows full system recovery as well as migration to a new or different system, or a migration into a freshly-installed OS X boot disk. The TM backups are not directly bootable.   As for restoring or migrating a TM backup, the use of the recovery partition of or a bootable copy of OS X on a flash drive or other secondary disk is sufficient.    If I want or need a directly bootable backup, CCC could be used, but Disk Utility can build a bootable backup, too.   (I'll generally use a direct, bootable copy when I either need the speed — TM is slower — and/or when I don't need the depth of backups that's provided by TM.)

  • by steve359,

    steve359 steve359 Dec 26, 2015 11:52 AM in response to celio-likes-apples
    Level 6 (14,032 points)
    Dec 26, 2015 11:52 AM in response to celio-likes-apples

    TimeMachine needs OSX to be re-installed to properly re-lay the TM contents.  A TM backup is not by itself bootable.  "Yes" is the answer to your second question.

     

    CCC makes a bootable clone of a drive that can let you evaluate a system that does not boot on internal ... is the screen working?, do the USB ports work?, is the internal drive readable?, etc.  Clones usually have the CCC loaded on it because clones are copies of the cloned drive, so can "clone themselves back onto" the internal drive with minimal or no loss (depending on how much changed between "clone/backup" and "now".

     

    Here is work from a former member (now deceased) of ASC who knew TM better than Apple probably does: http://pondini.org/OSX/Home.html

     

    Some never use TM because it needs OSX reinstalled first, other see nothing wrong with it and will not pay $40 for CCC.  As long as one backs up files, they are not wrong.

  • by celio-likes-apples,

    celio-likes-apples celio-likes-apples Dec 26, 2015 12:20 PM in response to FishingAddict
    Level 1 (0 points)
    Dec 26, 2015 12:20 PM in response to FishingAddict

    Hello FishingAddict,

     

    Thank you for your answer, I have never really thought about online backups but you made look into it.

     

    Regarding your answers, I have further questions:

     

    1) Local TimeMachine backups (either to desktop attached storage such as an external drive, or to a Time Capsule)

    You can't beat TimeMachine for a real-time backup that allows rolling back when necessary

    I have never used Time Machine on my Macs but I have seen someone that had Time Machine enabled, and he was complaining about running out of free disk space not knowing what was taking up his free space. When looking into his problem, I found out that it was the local snapshots of Time Machine that were taking several gigabytes of disk space. This was the primary reason I never wanted to use Time Machine. I did manage to disable his Time Machine's local snapshots using complicated terminal commands. I believe local snapshots are useful in a business storage server environment, but for home computing where disk space comes at a premium, I'm not sure if this functionalty justifies it's use of disk space. By the way, my friend's Mac was running Mountain Lion at the time, I don't know if Time Machine has improved since.

     

    2) Bootable clones of your boot and external drives (as steve359 suggests)

    At a minimum, you should clone your boot drive before any major OS update just in case something goes wrong and you need to recover quickly.  Either CCC or SuperDuper! work very well.

     

    Yes, drive cloning seems a good idea at first. But does it support incremental cloning? Or you have to dump the entire drive(s) each time you create a clone? I ask this because backups are most useful when they are as recent as possible.

     

    3) Off-site Backups are a NECESSITY that many people forget completely.  If you have a disaster or theft at your house will it take your Mac and all backups with it?

    By far the best modern solution to off-site backups are cloud backup services like Backblaze or CrashPlan.

     

    I'm presently looking into this, thanks for the suggestion. It seems that both cost about 5$/mo for unlimited backup storage, which one do you use? Do they allow for easy file finding and retrieving from other devices?

     

    Lastly, when devising a backup scheme I have found one very important consideration.  It is very easy to completely backup both internal drives and directly attached (e.g. FireWire, USB3, Thunderbolt) external drives using TimeMachine and cloud backup services.  However, it is very difficult to fully backup a local NAS to either of these.  Therefore, you need to consider this carefully if you are considering a NAS for storage (remembering that RAID is NOT backup).

     

    Good luck.  It takes a lot of careful consideration of your particular storage needs to feel confident in your backup strategy.  But, if you have TimeMachine, cloned drives, and offsite backups, you will be better off than 99.9% of all home users.


    Yes, as NAS is not directly attached it makes sense that Time Machine isn't aware of them.


    Thanks for all the answers.


  • by steve359,

    steve359 steve359 Dec 26, 2015 12:26 PM in response to celio-likes-apples
    Level 6 (14,032 points)
    Dec 26, 2015 12:26 PM in response to celio-likes-apples

    Local snapshots usually go to the TM drive, when connected.  When the drive is not attached they accumulate.  Read Mr. Pondini's excellent documentation.

     

    CCC will replace old versions of files with new versions, saving old versions in separate folder on that drive.  CCC incremental drives can seem to take forever ("looking for files to backup") but will only copy changed files.

     

    Internet recovery of files, in my non-experienced opinion, still require the internet and can take longer to recover than attached physical drives.  But that is just my opinion.

  • by MrHoffman,

    MrHoffman MrHoffman Dec 26, 2015 12:47 PM in response to celio-likes-apples
    Level 6 (15,637 points)
    Mac OS X
    Dec 26, 2015 12:47 PM in response to celio-likes-apples

    Plea

  • by MrHoffman,

    MrHoffman MrHoffman Dec 26, 2015 1:03 PM in response to MrHoffman
    Level 6 (15,637 points)
    Mac OS X
    Dec 26, 2015 1:03 PM in response to MrHoffman

    Any backup tool — when there's a software problem, whether due to failed hardware or a misconfiguration — can fail.   Somebody that filled their disk because of Time Machine is something well worth considering.   Are you likely to ignore backup errors?   If you are, then consider that you can ignore cloned backups, too.   It's trivial to end up with no backups.   Which is what you appear to have now, and your data is at risk.   Right now.

     

    Yes, with backups, there are many choices and many options.


    The only absolutely wrong choice for your critical data is having no backups of your critical data.

     

    Time Machine works fine.   Once in a while, it can get corrupted.   Any backup can.   Having more than one TM backup volume can help with that.   Folks with laptops can use a backup at home and a backup at the office, which gets you two copies, and an off-site copy.

     

    Incremental backups do not exist with classic disk clones.   Cloned disks have no depth of backups, unless you have multiple target backups.    And if you have multiple backup targets, you probably want multiple disks — backup disks can get corrupted, and can fail, just like Time Machine backups.  There are ways to do incremental backups — Time Machine is one example — but those are not usually directly bootable.

     

    "Cloud services" — what used to be called a service bureau or a hosted provider — are what they provide — there are a zillion different offerings here, ranging from storage where you dump a zip of a clone, to a server that you use rsync or some other remote backup tools, or cases where you use some cloning tool and simply mail the backup disk or even backup tape around.   You get to figure that out for yourself, your data, your budget, and some consideration around how good you are at following a backup schedule.

     

    When looking at hosted storage, also calculate how much time it will take to perform a network backup or a network restore.   Some Internet links will take a very long time, for a sufficiently large volume of data.   In other words, compare the size of your disk(s) with the speed of your network link, and round down (possibly round down substantially)...

     

    Also consider what your state of mind will be when your system fails or your system is lost or damaged, and you have to get your data restored.   Do you want to map out the process, or just see if it works...   This is why most folks recommend testing your recovery processing, before you need it.

     

    Time Machine supports remote backups to OS X Server as a NAS, and TM also supports remote backups to an Apple Time Capsule.

     

    In general: the only wrong answer is no backups.   Get something going now.

  • by steve359,

    steve359 steve359 Dec 26, 2015 1:24 PM in response to celio-likes-apples
    Level 6 (14,032 points)
    Dec 26, 2015 1:24 PM in response to celio-likes-apples

    To reinforce something MR. Hoffman said ...

     

    "Also consider what your state of mind will be when your system fails or your system is lost or damaged, and you have to get your data restored.   Do you want to map out the process, or just see if it works...   This is why most folks recommend testing your recovery processing, before you need it."

     

    You do not want to be "looking for the notes someone gave me" when your system just crashed and you need the system restored safely and quickly.  Know and even test, if you can, your recovery procedures.

  • by celio-likes-apples,

    celio-likes-apples celio-likes-apples Dec 26, 2015 5:08 PM in response to steve359
    Level 1 (0 points)
    Dec 26, 2015 5:08 PM in response to steve359

    Hello,

     

    I've just read the documentation about Local Snapshots on the Pondini.org site and TM by default makes local and external backups, even when the external drive/Time Capsule is available. That is true for laptops, for desktops, local snapshots are disabled by default.

     

    http://pondini.org/TM/30.html

     

    I can see the CCC's worth when diagnosing a Mac that doesn't boot. Usually we have little options, either trying "fixing" the drive with Disk Utility after booting into OS X Utilities, or reinstall everything and lose data, if no backups are available. So yes, have a running clone copy of OS X on an external drive is useful, specially if you do technical support for Macs.

     

    Yes, in order to restore files from a cloud backup site you need an Internet connection, and a fast one, which is not available in all countries. However, I think the purpose of cloud backups is to provide an additional layer of protection over your external backups.

     

    As MrHoffman said, having multiples TM disks or Time Capsules at different locations (one at home, another at work) is useful as for adding offsite backups, however that makes you wondering if the file you want to restore is in your home or work Time Machine.

     

    I like the multi-tier backup principle, for example:

     

    Mac -> External Disk 1 -> External Disk 2 -> Cloud Storage

     

    But as far as I know, if you use Time Machine you can't backup its contents to the cloud, such as:

     

    Mac -> External Disk 1 -> Time Machine -> Cloud Storage

     

    If that is the case, I wish cloud services would natively support acting as a TM destination, that would simplify things.

  • by MrHoffman,

    MrHoffman MrHoffman Dec 27, 2015 11:07 AM in response to celio-likes-apples
    Level 6 (15,637 points)
    Mac OS X
    Dec 27, 2015 11:07 AM in response to celio-likes-apples

    Performing remote incremental copies with any backup is a difficult problem, if you want it to work reliability.


    The remote copy is inherently slower than any local link, and that means remote copies are much more prone to glitches and outages than local links, and the bandwidth limitations mean that more "clever" solutions can be chosen by the developers.   Recovering from a glitch is non-trivial, too.   The most certain way is to recopy everything.   With a more "clever" solution, the developer then has to detect and recover from interruptions and corruptions that can arise within the data transfers, or errors that can arise on the local system, or errors that can arise with the remote backup.

     

    Terms like "cloud" and "best practices" tend to obfuscate the local requirements and expectations, unfortunately.   They don't make the issues of network glitches, network bandwidth nor data integrity or data security requirements go away, either.

     

    Get backups going.  At least one backup, performed automatically.   Anything beyond that is site-specific.