Apple Event: May 7th at 7 am PT

Looks like no one’s replied in a while. To start the conversation again, simply ask a new question.

Call for Referenced vs. Managed Definitive Opinions

Hi all...

I would like to solicit opinions from experienced Aperture users with large, mature libraries, referenced or managed.

My goal is to develop a singular and final well-considered opinion on the pros and cons of each method. Please no conjecture, only facts that you are able to discuss and results that are real and repeatable.

First I will explain my current personal position on this discussion and why. Next I will layout my current understanding of the pros and cons of each method. Hopefully facts will arise out of this exchange of ideas that are not currently in my understanding.

While I do remember an uproar that occurred during Aperture v1 with people worried about the wisdom of entrusting their photos to the Aperture managed library system. The reasoning, if I recall correctly was that people were concerned with the idea of some closed, proprietary monolithic file format that they could not be easily accessed outside of Aperture. They made their dissatisfaction with the managed method clear, and expressed a desire to maintain their files themselves, in Finder-readable format, and access them via reference in Aperture.

There was no discussion or defense of the attributes of managed method from Apple, or an explanation that the managed Library was not a proprietary format, just a delivery of the referenced method in response to user complaints.

To be honest, when the feature of referencing masters became available, I read all the info from Apple and the user feedback. I then realized that to convert from managed to referenced for me at that juncture was too long of a task, with a payback that was of dubious value to me at the time, so I just remained with the Aperture managed Library System.

I have not had any real problems using the managed method. I have had my share of file issues like missing previews or permissions problems, but the fixes have all come through Aperture's rebuild system. I have not experienced an unreadable or unusable Library, although I have read about those who have had this problem. It is my opinion that the victims of these problems did not maintain their data as I do, and the problems arose out of heavy fragmentation finally culminating in file/directory mismatch and ultimately file/Library corruption.

I hope everyone by now understands that the Managed Aperture Library is not a closed, proprietary, monolithic system. It is simply a container holding a full hierarchy of Finder-readable files, with your masters organized in their own folder labeled "Masters" in a simple day-date-time format.

The Aperture Managed Library is nothing but a FOLDER, with special attributes. Instead of simply opening in response to a double-click, this folder initiates the FOLDER ACTION of launching its connected application, and instructing said application to read the contents of the folder.

This folder action is bypassed if the user right-clicks on the Aperture Library, selects "Open Package Contents" from the menu, and Viola! the folder opens, revealing the entire contents of the Aperture Library in Finder readable form. Now you obviously do not want to move or delete anything within the managed folder (after all, you did ask Aperture to manage it), just copy files out of there if you need them.

To my mind, at this point in the discussion, pure managed Library operation offers the following:

PROS
1) Single control location for all my Aperture data. I cannot lose any pieces as I could in referenced operation.

2) Simple, single operation to copy, defrag and maintain.

3) Vaults system within Aperture provide me a simple way of making fast archive backups to multiple locations.

4) Vaults also provide a safety net, trapping deleted masters in the event I did not want to have deleted them.

CONS
1) Masters are not Finder-accessible by other applications.

2) User cannot choose the storage hierarchy design of the master files

3) User cannot maintain a library on a relatively small storage medium, like a laptop for easy portability, and access to Library previews for slideshows and Media Browser use in other apps.

Pure referenced Library operation offers the following:

PROS
1) User can maintain a Library on a laptop, and organize images without being connected to large storage for the master files, with the exception of editing.

2) User gets to determine where and how master files are stored and maintained on disk(s).

3) Very large files, like long format video can be stored offline, and only brought online when needed by Aperture, preventing Aperture library from getting excessively large.

CONS
1) Users are completely responsible for maintaining storage of the Masters. Master files can get misplaced or lost if user makes a mistake in managing them.

2) All files are not in a single location making defragmentation, copying and backup of the media they are stored on more complex.

3) Masters are subject to editing by other programs, possibly changing their content, and therefore changing work done in Aperture, possibly rendering work done in Aperture useless.

The preceding points are my understanding of the two methods at this point in time.

For me, a managed Library is the way to go because it is self-contained. I cannot risk having Masters changed or lost through simple human error. In Aperture, the Master is the basis of all image manipulation. Aperture does not edit the master, but treats if like an original negative, only adding adjustment recipes referencing its content. If the master itself was to get manipulated by another program, the reference Aperture counts on as being unchanged would be invalidated, and the previous adjustment recipes within Aperture would be wrong.
In a Managed system, it is on the same media, and can be maintained simply. Complex storage schemes are prone to human error. In managed, all copies of the Library contain all the data required. Backups and media maintenance are greatly simplified.

I did some experiments taking referenced masters and storing them into a read-only disk image format. Obviously, this requires a bit more work upfront that the user would have to handle themselves, since Aperture knows nothing of creating this kind of storage. It also would require the user maintain the editable hierarchies of masters so they can be added to at any time. There is also a significant amount of time spent in Disk Utility, writing these massive read-only images, and the user would have to open the image files to make the masters available to Aperture when they needed to import or edit, which is also very time consuming.

This said, this method solved 2 problems for me:

1) The masters themselves can never be edited, moved individually, lost, or corrupted as they are read-only, so it removes the problem of possible destructive editing or deletion.

2) The building of a read-only volume in a disk image makes an unfragmented copy of the Master's data. They will never run the risk of fragmentation because the volume is read-only.

While that did make a case for referencing, it did not provide a performance increase over Managed, and actually took a great deal more effort to do than the maintenance of the Managed Libraries, that typically occur unattended at night.

As far as retrieving masters from a managed library, it was simple given the day-date-time hierarchy employed by Aperture internally. Once on the date, I just selected the file(s) by name, and copied them out, no problem.

OK, now I would like to hear from others who have have opinions regarding either method of storage, especially if they have found advantages or disadvantages of which I am not as yet aware.

Sincerely,

K.J Doyle

MBP 17" Glossy HiRes 2.6 6GB RAM, NVIDIA 8600 GT Video w 512MB, Mac OS X (10.6.2), 30" Cinema Display and External eSATA RAID for Library

Posted on Mar 7, 2010 6:44 PM

Reply
142 replies

Mar 8, 2010 11:54 PM in response to Matthew Bergsma

Matthew Bergsma wrote:
I think you would notice a performance boost by splitting it up, especially if you are using a macbook with 2 1tb firewire drives.

But here's what I would do:
I'd use File->Relocate masters to move the images outside of the library into a subfolder structure of your choosing on your externals, and then I would put your library on your internal drive.

This way you can do a whole bunch of work even while you are not connected to the externals. Just set your preview size to something reasonable, and turn off automatically generate previews, and your library size will be fairly small. (For reference, my library of 30k images All referenced is about 16GB with faces on and previews set to quality 6, 1920x1920 max size).

Funny thing - Look at the "Aperture speedup tip" on http://xlr8yourmac.com/ tonight... Too bad this could all be avoided if people understood some details about the disk overhead required by aperture.



Thanks. I have a MBP core 2 duo.
Using and external 2TB drive (2x1TB enclosed) SATA connected to my MBP via eSata (thanks to info from Kevin Doyle).

Currently Ap3 Lib is on external but you mention placing AP3 Lib on local MBP drive and just referencing images from the external.

I was thinking along the lines of library on external referencing RAW files on the other external.
I'm currently running a test right now. So far it seems faster, but next will be a test using the AP3 library on the MBP drive and referencing that way.

Thanks for the additional info.

Mar 9, 2010 12:09 AM in response to Kevin J. Doyle

I used and would only use referenced file management. i want my files available to other programs. I'm quite able to manage my files, prefer my own file heirarchy and so only want the Aperture library to contain the non-destructive metadata. Whehatn I first started using macs, I played around with iPhoto (back in Tiger). I very quickly became frustrated and alarmed at the library system for that program. I decided there that I would not use any image editor that did not let me have easy access to my original files. That'a a bottom line for me.

Mar 9, 2010 6:24 AM in response to Kevin J. Doyle

I have been using a managed Library since the Aperture beta release. It is currently about 330G with around 32K images. Part of my workflow is to rigorously "prune" dupes and obvious mistakes, and I don't do a lot of work with Books or the Light Table. I am using a dedicated 1TB drive on the eSata bus of the MacPro, and have multiple external FW800 drives for Vaults - one in the studio, one in a bank vault off site. When I need to access the Library from my MBP, I either use my local N wireless network or export projects from the Library to a temp Library on the laptop. I do the latter when I travel and need to work on edits, or for reviews with clients away from my studio. It all works well for me, is fast and very little trouble to maintain. Haven't had a crash yet, but am ready if one happens.

With that background of my workflow, I chose the managed option for the primary reason that I did NOT want the hassle of maintaining a hierarchy, worrying about fragmentation, tracking where things are stored, etc. I want Aperture and the db to work for ME, not the other way around. I am a photographer (have been a techie in another life, tho) and want to focus on my art and not the plumbing. Fortunately, using plug-ins and the excellent editing tools native to Ap (even better now in A3), I don't need to give other stand-alone apps access to my images - frankly, I don't want anyone messing with my RAW files.

So, I "drank the Kool-Aid" and have benefitted for a number of years. I'm pleased with my disk space utilization (even tho a 1TB eSata drive may be had for less than $100 now), and the speed has not ever been an issue. I rebuild at least quarterly and am scrupulous about the Vault backup process.

Good luck with YOUR decision...thanks for listening to my 2¢...maybe some help.

cheers,
david

Mar 9, 2010 7:30 AM in response to sailor

For those who want a separate "external" folder hierarchy of their RAWs in case other apps need to access them, you can import to a managed library and also save a backup to disk using whatever folder hierarchy you want - all using the import command.

As for the managed library, I've discovered that if you delete some albums and move AP's folders around (doing this in AP), then in the "Masters" folder of the library package, you'll discover the masters moved in odd subfolders. For example, I imported a set of top level folders from disk (named "1996", ... "2010"), and later in AP I moved images around from one project to another and rearranged albums and "folders" (because I didn't choose "folders to projects" for some of the imported years as I was trying to do, so AP's folders/projects/albums no longer matched the source folder structure on disk). The result is that in the managed package, masters from 1996 and other years are now buried in 2008. For this reason, I cannot rely on the folder structure in AP's managed package to parallel the folder hierarchy of the external original image files and AP's folder/project/album hierarchy that I see in the library panel of AP.

At this point, I don't care what the "Masters" folder structure looks like in the managed package, since I have a set of duplicate RAWs in a folder hierarchy that any other app or Finder can find. Plus I use vaults for the managed library and perform backups of the external drive that stores the RAWs to in-house and off-site locations.

But at this point I'm still experimenting with managed vs. referenced.

Mar 9, 2010 7:48 AM in response to sailor

Quote: "I chose the managed option for the primary reason that I did NOT want the hassle of maintaining a hierarchy, worrying about fragmentation, tracking where things are stored, etc. I want Aperture and the db to work for ME, not the other way around. I am a photographer (have been a techie in another life, tho) and want to focus on my art and not the plumbing."

Note: This is a common misconception about referenced files. You can still have aperture do all these things, the only difference is that the "Root" folder the files go in is not an .aplibrary file. You just setup the import dialog to use a "subfolder" - I.E. "YYYY/MM/DD" and all your masters get stored there on import.

I have mine set to simply "YYYY/Project Name" so that it is easy for me to archive individual projects or years if needed. The benefit with referenced files is the Avoidance of fragmentation if your masters go on a different volume than the library.

Mar 9, 2010 8:26 AM in response to Kevin J. Doyle

Here is what I do:

I have my photos (95% raws, 5% jpegs) on my iMac's drive, in the Pictures folder. When I import from my SD card, I ask Aperture to create a folder in a given "main" folder and to create a project of the same name (but that's me and doesn't change anything performances wise, it's just nice that AP can do all by itself).

My Aperture library is on an external LaCie Q2 1TB drive connected with FW800. FW800 is actually faster than a physical hard drive can go and according to reviews, the difference of speed between eSata and FW800 is negligible (see: http://www.macworld.com/article/133840/2008/06/lacied2quadradisk.html). No need to buy an eSata adapter if you ask me, put that money in a lens.

That way, when Aperture is working with the library, it makes the LaCie drive work, whereas the program itself can use my main drive for its needs. I noticed a performance boost when I did this set-up.

I use iDefrag to defrag my library once in a while.

Now, why referenced and not managed master? I really like Aperture, I find it superior to rival products in many regards. But will this last for 5, 10 years? I hope so, but I can't be sure. Will I ever switch? I don't think so, but I can't be sure. So I keep my images out and accessible to other programs (even if I could export them with Aperture, but it might take time I don't want to waste)

I also think it boosts performances to use referenced instead of managed file (due to disk access, as mentioned before), but it is just my observation. Since Aperture can actually manage referenced files, the arguments around "using managed files is good for me since I don't have to care about organizing the files" are pretty weak.

Aperture is more flexible than anything else, this very thread proves it 🙂

Message was edited by: Manusnake

Mar 9, 2010 3:59 PM in response to Kevin J. Doyle

I use the managed model but prior to Aperture I was using a referenced style system with Extensis Portfolio and I had developed a 'mature' library and back up system on my own. Despite my prior investment in time and several things I had to give up the advantages of the managed model appealed to me.
Pros
1. The vault system provides better back up than the combination of system wide back up and archive to removable media.
2. You can keep several projects in the working phase. When I managed my own hierarchy, I was really only able to actively work on the most recent images. Once something hit the archives, I would be unlikely to ever edit it without essentially reimporting it as new.
Cons
1. The sanctity of "The Master" is, in my opinion, overrated. In the early days of my image library, the majority of the images were TIFF created from scanning film. These need routine editing for dust and scratches, color balance and levels, straightening and so on and so forth. All of this needs to be done once and then the image is good to go. If you did this editing in Aperture, it would really slow down the opening of the image. If you use the external editor in Aperture to do this then you end up using twice as much space.
2. Along the same lines - it is recommended to store the metadata right in the same file as the image. If the metadata database ever is destroyed then at least the metadata is safe in the file.
3. The version metaphor breaks down in some fairly common image processing tasks, for example: The stitched panoramas (which file should they be a version of?) or the images turned into posters or cards that contain multiple layers - these seem like new creations. In these cases once all the editing is complete - you need to decide if you want to put the image back into the library and what to do with all the saved intermediate edits. These are the kinds of questions that you are not supposed to have when you use the managed system.

Mar 9, 2010 5:05 PM in response to Kevin J. Doyle

I've been using multiple Managed libraries since 1.0 When a library gets over 50GB or so I generally start a new one, and archive the old one.

Pros:
* Relatively small files to backup (compared to some people's multi TB libraries).
* More library portability between multiple computers because of the moderate size.

Cons:
* Sometimes I have to go hunting through multiple libraries to find older images.
* The vault backup thing was hideously slow and opaque (I didn't trust it)... so I copied vaults to external drives instead of using vaults.

With Aperture 3.0 and the growing number of libraries I was trying to keep track of I decided to merge all my managed libraries into one master referenced library.

Now my library lives on an internal RAID 0 array, and all my referenced images live on external firewire 800 drives (with a sync job copying them to a NAS).

For working remotely on other computers I'm experimenting with exporting/merging projects to a small "working" libraries on my laptops.

Pro:
* Having everything in one place is great.
* Much higher degree of confidence in my backup strategy.
* The vault system makes more sense (and runs a lot faster) when it isn't backing up masters.

Con:
* Faces takes f-o-r-e-v-e-r to do anything on a 100,000+ image library.

Mar 9, 2010 7:30 PM in response to Kevin J. Doyle

I don't know of any advantages or disadvantages that you haven't mentioned in your post. I agree with your evaluation of the relative merits of Managed vs. Referenced.

I'm an early adopter of Aperture, have been using it since the first version when Managed was the only way to go. Since then I've never needed to access my photos through the finder. There has never been a situation where having a Referenced library would have been helpful. It's just completely a non-issue.

Mar 9, 2010 10:10 PM in response to Kevin J. Doyle

Long time Photoshop user, new Aperture 3 user; Because I still use ACR/Photoshop quite a bit and Canon DPP occasionally I prefer to have images available to any of the three apps. That should also result in minimal fragmentations since Aperture and Adobe add nothing to the original file and regardless of the size of Adobe's and/or Aperture's XMP file no additional space is required beyond the initial XMP creation.

Having worked on my Aperture installation (way too) steady for a couple of weeks I would say that while A3's results are excellent and exciting its library's behavior and stability is still not ready for prime time. I'm sure not putting any of my photos in that thing! Plus, I'd rather try to manage a 25GB referenced library than a 1TB managed one.

Right now I'm trying to get a vault to work, should be a simple job. Vaults have been working all week but suddenly both vaults for my main library quit at about 80% or so finished and I haven't figured out how to rebuild a Vault (thought the libraries have been rebuilt far too many times). I've made a new one, maybe it will work. That's one of the reasons I don't want my images managed.

For me it's referenced photos all the way. In fact one of the reasons I purchased A3 was that it can manage my digital assets without swallowing them.

Mar 10, 2010 1:00 AM in response to Matthew Bergsma

Matthew Bergsma wrote:
On a single drive, the combination of simultaneous reading and writing for 8-10 tiny metadata files and 1 large RAW file is a recipe for fragmentation. Especially with RAW files over 20MB, because those won't get automatically defragmented by the OS.

This is what I stated in my first post (fragmentation on a single drive). On separate drives this isn't an issue, which is why there is a performance benefit because you are removing the storage bottleneck of simultaneously reading a large block raw file at the same time writing to small block library files.

The problem is, that with a managed library you CAN'T gain the speed benefit of separate volumes. Anyone who has done server tuning for database hosting would agree with this storage speed analysis, it isn't really a mystery.

A lot of people think that a RAID will speed things up for them here, but the RAW files themselves don't need a RAID - it's the library itself that needs fast writing and reading. This file structure benefits more from an SSD than it does from a RAID.

The masters themselves are small enough, and are only read 1-3 files at a time, so a RAID isn't necessary for speed with the masters.

I prefer to update my vault frequently, and enjoy the fact that it goes very fast since the RAW files aren't going into it!!! I also like the ability to take my entire library (about 15-20GB for 35k images) onto a laptop with me for offline access of the projects. (Although with aperture 3's new library merge/sync feature this isn't as much of an issue).

However, putting all your masters and your library on one 2tb volume sure is a case of putting all your eggs in one basket - Sure you can back it up, but that takes TIME.

It's nice to be able to distribute 4-5 copies of a referenced library onto various mediums. (Fits well within Bluray limitations).

Additionally, when it comes time to "cleanup" or archive at the end of a year, its very easy to pull the master files offline and leave your library and projects intact.

The real kicker is that with a "referenced" library, you don't have to change anything about your workflow. You just setup the import dialog to store the Masters in a project subfolder hierarchy of your choosing, and it is pretty much identical to the managed library structure, except that it is in a folder of your choosing (and more importantly) a volume of your choosing.

I listed the only two "Pros" for a managed library I could think of, I'd be really curious if there are any I missed.


Hi Matthew,

OK, I actually had to do some shooting the past couple of days, lol but I am back for a bit...thanks for all the good input, everyone.

ALthough I think I am onto something good here. I did a Relocate Masters to an empty 4TB striped pair, from 3 libraries, and then I merged them. I have almost 3TB of masters on the 4TB striped pair with zero fragmentation. Using Aperture I imported more RAWs into the new setup, and again, zero fragmentation on the masters.

I played with this setup all evening, and as long as I do not delete anything, and of course I have no background processes running like Spotlight, the 3TB of data is on long contiguous piece, no fragments, no changes....perfect. From this I can only assume as long as I do not manipulate anything on the Masters volume from the Finder, it would continue to add files in a contiguous piece until the drive was full. This is a big find here, as this dedicated storage would only require a SmartUpdate to its backup volume, done immediately after any importing was done, to maintain an equally contiguous backup. Even the oldest rotating backup disk would work the same way. This means all storage of Master files could be placed into service immediately, with no downtime, and absolutely identical storage integrity. The worst case delay would be to swap the drives into the array, and relaunch...could’t take more than a minute or so...WOW.

From reading some of the other responses, one puzzled comment is that I still have no idea why people would want to access these Finder-readable masters with another program. Obviously, in Aperture’s non-destructive editing the Master file is (and must remain) the sacrosanct source of the adjustment “recipe” of any version. If one was to modify the source, you would certainly change and possibly destroy the recipe. As you are aware, it is only after Aperture has “baked a version” by export has the recipe been applied to the Master and written out to a file.

Obviously, it is an advantage to have Finder access to the masters....but to copy them, not to modify them. If I could have my wish here, I would like to see Aperture store referenced masters in a read-only hierarchy to prevent this, with Aperture being the only file that can write to this designated volume, locking it back to read-only after adding to it. That way, even programs that do destructive editing can access the Aperture Masters SAFELY. They would have to do a Save as... in order to to anything with the Masters, and they could not store their modified version in the Aperture hierarchy, either.

I have not tried deleting large blocks of referenced masters and then importing more yet, although I will assume that this would lead to some fragmentation going forward. The good news is that this would be uncomplicated fragmentation, solved by a simple copy operation to return to contiguous file status, similar to what I am doing now on a daily basis.

OK, ‘nuff said about the masters storage for the moment, I have to test more to see the results of deletion to gather what occurs.

As far as the masterless Library now, the problem I am running into is that the size of the Previews/Thumbnails vary widely, I believe making the Masterless library significantly larger than it need be. Since these libraries came from 3 different people within my group, let’s assume they did not have their Preferences set identically, to begin with. Before I merged them, my former 500GB managed library was now 63GB for 44,589 images (mostly D2X and D3, about 5% D3X), about 61GB of that is Previews/Thumbnails, making the average image consume approx 1.4MB each. Is there some guideline size you have found to be optimum?...and what is the easiest way to get there? The 3 merged libraries are 188GB, masters being about 3TB. About half the Masters storage is taken up by about 11,000 medium format files, most are 130MB each.

While I don’t want to be asking for too much trouble here, do you have a clever solution to clear out the ones that are too large, and what do you recommend for a nominal size?

I have the library on a 256GB volume, with a backup copy out in the array. It is backed up at the same time as the masters via script.

I have ordered a Crucial 256GB SATA III SSD to use for the Library, but won’t get it for a couple of weeks, as it is back-ordered. This latest SSD is benchmarking at 350MB/sec reads in a SATA III host, although I don’t expect to see more than about 200-220MB/sec into the MBP on a single eSATA channel, the other channel would remain into a 5 bay array. Now if I could get the Library to be under 100GB, I could have room to put this into the laptop to use as a boot drive as well, and get to keep all 10 bays externally.

Of course, with an SSD since there is almost negligible latency, but maintaining sequential reads is still 3 times as fast as random. I would still want to do a copy onto and back from an array stripe, although perhaps less frequently than the nightly ritual now. Random reads on an SSD is about as fast as sequential on a good drive.

Anyway, with this volume of files, trimming the Library size should go a longway to increasing possible performance, I look forward to your comments.


Sincerely,



K.J. Doyle

Mar 10, 2010 6:30 AM in response to Kevin J. Doyle

I'm trying to condense and summarize, in my mind, the many issues and points made in this thread. I'm leaning strongly towards referenced masters. (In the past I always used referenced and am now experimenting with both referenced and managed - I'm talking AP 3)

Am I correct that with masters on a FW800 drive, there should be enough speed for that, even as the number of masters increase going forward.

Also, I've concluded that having the referenced library needs special consideration - placing it on the internal drive for example, and defragging it often. However, would it make sense to get an eSata drive (not necessarily RAID 0) and have the library there (still with the masters on the FW800)? This also assumes the eSata would have to be defragged often. It seems some of the information here says it's the library that is prone to fragmentation and uses frequent read/write.

Call for Referenced vs. Managed Definitive Opinions

Welcome to Apple Support Community
A forum where Apple customers help each other with their products. Get started with your Apple ID.