6 Replies Latest reply: Jun 28, 2012 3:50 PM by Shuttleworth125
wombat2007 Level 1 Level 1 (5 points)

I'm a photographer and reaching a library that is pushing 1T. I don't see a performance isue as much as the backups I run are painful - even the vaults. I back on two other network drives as well, so I could be out of commision for a day.

What are the pros doing out there ? Are you using referenced or masters ? Do you keep one big library or smaller contextual libs and why ? Are there any really good tutorials on heavy workflow? I have decided to use three different managed libraries, one for Archival, Current, and Professional. I can then run Vaults on all three seperatly on an external drive.

I am curious about other folks experience and the workflows they've settled on //

 

JJ


iMac, Mac OS X (10.7.4), 32 GB DDR3, Triple TB Display
  • Shuttleworth125 Level 2 Level 2 (415 points)

    Ours is a split library, current jobs are managed, old jobs and personal work are referenced. A new library is done for each year.

     

    The referenced files are backed up hourly to an external using ChronoSync. The external also has our vaults on. The external is regularly swapped for  one offsite.

     

    The whole lot is also backed up using time machine.

  • SierraDragon Level 4 Level 4 (2,695 points)

    IMO

     

    • Multiple Libraries are self-defeating to the concept of an images database because they prevent global searches like keyword searches. Only for very specific and unusual reasons (like specific-client security of some kind) should there be more than one Library. Creating multiple Libraries for organizational or drive space purposes is misguided thinking.

     

    • Large Managed Libraries are a bad idea for the reasons you are experiencing. They can be done well, but only by true mass storage experts with sophsticated large-drives setups.

     

    • The Library on an internal drive with Masters/originals referenced on external drives is preferable for all but a small Library.

     

    From a previous post of mine:

    ---------------------------------------

    The Library with its Previews lives on the internal drive and is always accessible. Masters live on external drives. The Library is backed up via Vaults and originals are backed up to redundant locations using the Finder before import into Aperture.

     

    Personally I have images managed on the internal SSD until editing is complete then convert to Referenced-Masters.

     

    -------------------

     

    Database Physics

     

    Aperture is designed to bite into small chunks at a time, so TBs of data do not bother the app itself. However, handling super-large batches of data like 1.5 TB on consumer hardware tends to be problematic.

     

    Slower speeds seem to exacerbate handling large data chunks.

     

    IMO referenced Masters make far more sense than building huge managed-Masters Libraries. With referenced Masters one has no need to copy a 1.5 TB sized file. A I find that even (2011 MBP) copying 5-15 GB-sized batches of RAW/JPEG files copying fails with some frequency, enough so that I always verify the copy. Failures copying 1500 GB to a drive as a single file should be expected based on my experience.

     

    • Hard disk speed. Drives slow as they fill so making a drive more full (which managed Masters always does) will slow down drive operation.

     

    • Database size. Larger databases are by definition more prone to "issues" than smaller databases are.

     

    • Vaults. Larger Library means larger Vaults, and Vaults are an incremental repetitive backup process, so again larger Vaults are by definition more prone to "issues" than smaller Vaults are. One-time backup of Referenced Masters (each file small, unlike a huge managed-Masters DB) is neither incremental nor ongoing; which is by definition a more stable process.

     

    Managed-Masters Libraries can work, but they cannot avoid the basic database physics.

     

    Note that whether managed or referenced, original images should be separately backed up prior to import into Aperture or any other images management application. IMO after backing up each batch of original images importing that batch into Aperture as a new Project by reference makes by far the most sense. Building a huge managed Library or splitting into multiple smaller Libraries is less logical.

     

    HTH

     

    -Allen

  • SierraDragon Level 4 Level 4 (2,695 points)

    Just a reminder that hard disk drives slow as they fill. A good guideline is to not exceed ~70% full; even less full is preferable for speed.

     

    -Allen

  • wombat2007 Level 1 Level 1 (5 points)

    Shuttle -

    Sounds like an interesting solution. I like the idea of seperating current from archival. However ti does seem like quite a bit of maintenance.

    Why would you back up non-current jobs every hour ( or were you referrring to Time Machine ? )

    I also see you're backing all your Libraries up on Vaults and taking a drive off-site. You can also do this automatically with ChronoSynch. No shuttling drives back and forth, as you're inevitably going to lose info that way.

    The bottomline for me with your approach, is I just dont trust TimeMachine. It's fine as an emergency backup, but I don't count on it.

    How do you respond to Sierra's process ?

  • wombat2007 Level 1 Level 1 (5 points)

    Allen

    Thanks so much for the detalied reply - this was exactly what I was looking for. I do have some clarifying questions.

    I still need to 'seperate' into buckets, at least for sanity ( lets say ARCHIVAL, PERSONAL , JOBS )

    Per your database discussion, are you suggesting I simply do this as subdirectories in one Library ?

    Also, I'm still a bit unclear on why you copy RAW masters onto a redundant drive and then import. If I import off my cam into Aperture, the referenced masters will be saved on that external drive anyway, correct ?

    My understanding with referenced vs managed is that the 'Masters' are held in one place ( an external 12T ) and the 'Previews', "Meta' and 'Adjustments' are all held somewhere else ( you're suggesting keeping locally on SSD for speed ? )

    I'll wait for your response and then outline my setup a little more to make sure I'm covering my bases.

     

    Thanks

     

    JJ

  • Shuttleworth125 Level 2 Level 2 (415 points)

    It's actually very low maintenance. I try to keep it simple as it's my wife's photography business, I just look after the computer side of it.

     

     

    Chronosync looks at the referenced masters every hour and copies any changes across to the backup. It doesn't do anything with the archived masters as its an incremental backup. I don't use it to back up the library as I didn't like the idea of software messing about inside the library while it was being used. I also had problems with backing up libraries as copies, it's not automatic or incremental and you can never be sure which copy is most up date. At least with a vault you know when it was last updated, and that it is a backup not a 'real' library.

     

    Vaults are on the external and offsite drive because the a very easy to use. Although we rarely use the old libraries we update the vaults anyway so they reflect any slight changes.

     

    We aren't just relying on time machine, we have vaults as well, if a recovery using time machine failed, we could rebuild from a vault. In the event of fire or theft we have all the masters and the vaults in an offsite location.

     

    I forgot to mention that we also back up masters on import as Allen suggested, though I admit we do use aperture to do this rather than by finder copy which is the method Allen recommends.  This is done so you have a backup before it has been through any management software which could potentially cause damage to or lose the files.