Skip navigation
This discussion is archived

Call for Referenced vs. Managed Definitive Opinions

26341 Views 142 Replies Latest reply: Apr 27, 2010 4:05 AM by nikonshooter2010 RSS
1 2 3 ... 10 Previous Next
Kevin J. Doyle Level 1 Level 1 (90 points)
Currently Being Moderated
Mar 7, 2010 6:44 PM
Hi all...

I would like to solicit opinions from experienced Aperture users with large, mature libraries, referenced or managed.

My goal is to develop a singular and final well-considered opinion on the pros and cons of each method. Please no conjecture, only facts that you are able to discuss and results that are real and repeatable.

First I will explain my current personal position on this discussion and why. Next I will layout my current understanding of the pros and cons of each method. Hopefully facts will arise out of this exchange of ideas that are not currently in my understanding.

While I do remember an uproar that occurred during Aperture v1 with people worried about the wisdom of entrusting their photos to the Aperture managed library system. The reasoning, if I recall correctly was that people were concerned with the idea of some closed, proprietary monolithic file format that they could not be easily accessed outside of Aperture. They made their dissatisfaction with the managed method clear, and expressed a desire to maintain their files themselves, in Finder-readable format, and access them via reference in Aperture.

There was no discussion or defense of the attributes of managed method from Apple, or an explanation that the managed Library was not a proprietary format, just a delivery of the referenced method in response to user complaints.

To be honest, when the feature of referencing masters became available, I read all the info from Apple and the user feedback. I then realized that to convert from managed to referenced for me at that juncture was too long of a task, with a payback that was of dubious value to me at the time, so I just remained with the Aperture managed Library System.

I have not had any real problems using the managed method. I have had my share of file issues like missing previews or permissions problems, but the fixes have all come through Aperture's rebuild system. I have not experienced an unreadable or unusable Library, although I have read about those who have had this problem. It is my opinion that the victims of these problems did not maintain their data as I do, and the problems arose out of heavy fragmentation finally culminating in file/directory mismatch and ultimately file/Library corruption.

I hope everyone by now understands that the Managed Aperture Library is not a closed, proprietary, monolithic system. It is simply a container holding a full hierarchy of Finder-readable files, with your masters organized in their own folder labeled "Masters" in a simple day-date-time format.

The Aperture Managed Library is nothing but a FOLDER, with special attributes. Instead of simply opening in response to a double-click, this folder initiates the FOLDER ACTION of launching its connected application, and instructing said application to read the contents of the folder.

This folder action is bypassed if the user right-clicks on the Aperture Library, selects "Open Package Contents" from the menu, and Viola! the folder opens, revealing the entire contents of the Aperture Library in Finder readable form. Now you obviously do not want to move or delete anything within the managed folder (after all, you did ask Aperture to manage it), just copy files out of there if you need them.

To my mind, at this point in the discussion, pure managed Library operation offers the following:

PROS
1) Single control location for all my Aperture data. I cannot lose any pieces as I could in referenced operation.

2) Simple, single operation to copy, defrag and maintain.

3) Vaults system within Aperture provide me a simple way of making fast archive backups to multiple locations.

4) Vaults also provide a safety net, trapping deleted masters in the event I did not want to have deleted them.

CONS
1) Masters are not Finder-accessible by other applications.

2) User cannot choose the storage hierarchy design of the master files

3) User cannot maintain a library on a relatively small storage medium, like a laptop for easy portability, and access to Library previews for slideshows and Media Browser use in other apps.

Pure referenced Library operation offers the following:

PROS
1) User can maintain a Library on a laptop, and organize images without being connected to large storage for the master files, with the exception of editing.

2) User gets to determine where and how master files are stored and maintained on disk(s).

3) Very large files, like long format video can be stored offline, and only brought online when needed by Aperture, preventing Aperture library from getting excessively large.

CONS
1) Users are completely responsible for maintaining storage of the Masters. Master files can get misplaced or lost if user makes a mistake in managing them.

2) All files are not in a single location making defragmentation, copying and backup of the media they are stored on more complex.

3) Masters are subject to editing by other programs, possibly changing their content, and therefore changing work done in Aperture, possibly rendering work done in Aperture useless.

The preceding points are my understanding of the two methods at this point in time.

For me, a managed Library is the way to go because it is self-contained. I cannot risk having Masters changed or lost through simple human error. In Aperture, the Master is the basis of all image manipulation. Aperture does not edit the master, but treats if like an original negative, only adding adjustment recipes referencing its content. If the master itself was to get manipulated by another program, the reference Aperture counts on as being unchanged would be invalidated, and the previous adjustment recipes within Aperture would be wrong.
In a Managed system, it is on the same media, and can be maintained simply. Complex storage schemes are prone to human error. In managed, all copies of the Library contain all the data required. Backups and media maintenance are greatly simplified.

I did some experiments taking referenced masters and storing them into a read-only disk image format. Obviously, this requires a bit more work upfront that the user would have to handle themselves, since Aperture knows nothing of creating this kind of storage. It also would require the user maintain the editable hierarchies of masters so they can be added to at any time. There is also a significant amount of time spent in Disk Utility, writing these massive read-only images, and the user would have to open the image files to make the masters available to Aperture when they needed to import or edit, which is also very time consuming.

This said, this method solved 2 problems for me:

1) The masters themselves can never be edited, moved individually, lost, or corrupted as they are read-only, so it removes the problem of possible destructive editing or deletion.

2) The building of a read-only volume in a disk image makes an unfragmented copy of the Master's data. They will never run the risk of fragmentation because the volume is read-only.

While that did make a case for referencing, it did not provide a performance increase over Managed, and actually took a great deal more effort to do than the maintenance of the Managed Libraries, that typically occur unattended at night.

As far as retrieving masters from a managed library, it was simple given the day-date-time hierarchy employed by Aperture internally. Once on the date, I just selected the file(s) by name, and copied them out, no problem.

OK, now I would like to hear from others who have have opinions regarding either method of storage, especially if they have found advantages or disadvantages of which I am not as yet aware.

Sincerely,

K.J Doyle
MBP 17" Glossy HiRes 2.6 6GB RAM, NVIDIA 8600 GT Video w 512MB, Mac OS X (10.6.2), 30" Cinema Display and External eSATA RAID for Library
  • Matthew Bergsma Level 3 Level 3 (600 points)
    Haha, Kevin, you know where I stand

    Referenced.

    Managed offers only two "pros" and they are:
    1. Extra protection from accidentally mucking around in the finder and removing images.
    2. The masters get backed up in the vault.

    The cons are much more numerous -

    Slow library operation, slow vault updates, fragmentation on a single drive for 20MB+ images, Eventual "storage ceiling" on one drive, and the biggest reason: A 700GB library is just not maintainable. Many people encountered this trying to upgrade to version 3 -
    Mac pro 3.2x8, macbook pro 2.4, Mac OS X (10.6.2)
  • William Lloyd Level 6 Level 6 (19,205 points)
    IMO this "fragmentation" thing is a ruse. There should be no differences in performance between a referenced and managed library, unless the drive the files themselves on differ.

    I've used both (primarily managed images) and have found no performance increases for referenced. I frankly like the ability to use a vault with managed images.

    True, with a 700 GB image it can get difficult to use a managed library, but it's not impossible if you have a 2 TB drive dedicated to the task. I would never let a drive get more than 85% full for performance reasons, so with a library that large I'd use a 2 TB drive. Thankfully, they've been available for about a year now. It's true that updating an Aperture 2.x to 3.x library that's 700 GB in size would be painful. My library is only 250 GB and I have a 1 GB mirrored drive that holds it; upgrading was no problem at all.
    8-Core Mac Pro, Mac OS X (10.6.2)
  • Abbstrack Level 1 Level 1 (35 points)
    I don't really see a much of a difference in either method, obviously discounting the x-factor being the ability to screw up your library by messing with it on your own in the referenced model. So really that makes this more a matter of personal choice. For me, I prefer referenced. That way, my libraries stay relatively small, and are nimble enough for me to backup/copy pretty easily. I have 1.6TB of images referenced onto a 2TB drive, and trying to copy a managed library or libaries holding those files would be very time consuming. Instead I have them split up into 3 libraries, ranging in size from ~50GB, to ~6GB, and it becomes very easy for me to make nightly copies of my libraries to external drives.

    With the 2TB drive, I don't use it for anything else, so it's simply a container/locker. I dont have to worry about performance, fragmentation, etc.. it's a dedicated image drive. And I can trust myself as someone who knows the workings of the system/software not to go into the files there unless i absolutely have to. And when I do, the referenced model allows me to arrange them so specifically that I know exactly where I need to go.

    The other part of this is that Aperture 3's new method of allowing projects to be exported as libraries is that much more of a bonus for me with this setup. If i'm going on the road for a week, and working on 4 or 5 projects while travelling, I can simply export them out, consolidate the masters into those projects, make my edits on the road, then merge when i return. I dont have to think about having my entire library on me, possibly losing senstive data, etc...

    Again, it seems more of a matter of personal choice. I think referenced works best for my setup.

    Message was edited by: Abbstrack
    iMac 24in, Mac OS X (10.6.2), 2.8Ghz Core Duo 4GB ram NVIDIA GeForce 8800 GS, Macbook Pro 2.53Ghz 4GB ram
  • Matthew Bergsma Level 3 Level 3 (600 points)
    On a single drive, the combination of simultaneous reading and writing for 8-10 tiny metadata files and 1 large RAW file is a recipe for fragmentation. Especially with RAW files over 20MB, because those won't get automatically defragmented by the OS.

    This is what I stated in my first post (fragmentation on a single drive). On separate drives this isn't an issue, which is why there is a performance benefit because you are removing the storage bottleneck of simultaneously reading a large block raw file at the same time writing to small block library files.

    The problem is, that with a managed library you CAN'T gain the speed benefit of separate volumes. Anyone who has done server tuning for database hosting would agree with this storage speed analysis, it isn't really a mystery.

    A lot of people think that a RAID will speed things up for them here, but the RAW files themselves don't need a RAID - it's the library itself that needs fast writing and reading. This file structure benefits more from an SSD than it does from a RAID.

    The masters themselves are small enough, and are only read 1-3 files at a time, so a RAID isn't necessary for speed with the masters.

    I prefer to update my vault frequently, and enjoy the fact that it goes very fast since the RAW files aren't going into it!!! I also like the ability to take my entire library (about 15-20GB for 35k images) onto a laptop with me for offline access of the projects. (Although with aperture 3's new library merge/sync feature this isn't as much of an issue).

    However, putting all your masters and your library on one 2tb volume sure is a case of putting all your eggs in one basket - Sure you can back it up, but that takes TIME.

    It's nice to be able to distribute 4-5 copies of a referenced library onto various mediums. (Fits well within Bluray limitations).

    Additionally, when it comes time to "cleanup" or archive at the end of a year, its very easy to pull the master files offline and leave your library and projects intact.

    The real kicker is that with a "referenced" library, you don't have to change anything about your workflow. You just setup the import dialog to store the Masters in a project subfolder hierarchy of your choosing, and it is pretty much identical to the managed library structure, except that it is in a folder of your choosing (and more importantly) a volume of your choosing.

    I listed the only two "Pros" for a managed library I could think of, I'd be really curious if there are any I missed.
    Mac pro 3.2x8, macbook pro 2.4, Mac OS X (10.6.2)
  • Matthew Bergsma Level 3 Level 3 (600 points)
    On one drive, referenced vs managed speeds will be no different. On separate drives, one designed for fast small block read/write, and one designed for fast sustained read speed, All sorts of library operations are considerably faster:

    1. Importing images is noticeably faster, as the entire structure for each image is built in the library, it isn't bottlenecking the volume with attempts to copy the master simultaneously. In fact, I was somewhat shocked at how fast it is with the new aperture 3 import routine... As in "Wait, did that really just finish?"

    2. Here's the kicker. Fragmentation affects one thing in the aperture library the most - the dang AP.thumbnails and AP.minies files. Since these are Giant BLOB-like files of every thumbnail in the project, they are continuously updated while a file is being imported and adjusted, they get very fragmented (you can confirm this in idefrag). Putting the masters on a separate volume helps mitigate this tremendously.

    3. General library loading operations are faster. Switching between projects/albums, Loading thumbnails while simultaneously loading the selected raw image...

    4. Generating previews doesn't slow the storage system to a crawl, especially while applying adjustments, and thumbnails get reprocessed.

    Now, if you have a fast enough RAID, the difference might not be very noticeable, but for a mac pro user with just three internal drives - you can get comparable to RAID performance just by optimizing your setup to account for the needs of each asset - the library needs fast random small block read/write speed, and the masters need fast sustained large block read speed.
    Mac pro 3.2x8, macbook pro 2.4, Mac OS X (10.6.2)
  • Abbstrack Level 1 Level 1 (35 points)
    I would be very interested if you would run the free demo of iDefrag on that disk to look at the number (or lack thereof), fragments.


    based on iDefrag, 82,806 total files on the drive (all referenced files), 866 of which are fragmented.

    Of note is that in the past I have tried running aperture off this drive (i no longer do, i partitioned my internal HD and created a space for my Aperture libraries), and I just recently turned spotlight indexing off, so its possible those are more to blame for the fragmented files.

    At any rate, I'm not a DBA, but those numbers would seem to be pretty positive.

    Message was edited by: Abbstrack
    iMac 24in, Mac OS X (10.6.2), 2.8Ghz Core Duo 4GB ram NVIDIA GeForce 8800 GS, Macbook Pro 2.53Ghz 4GB ram
  • Falcon01 Calculating status...
    Matthew Bergsma wrote:
    On one drive, referenced vs managed speeds will be no different. On separate drives, one designed for fast small block read/write, and one designed for fast sustained read speed, All sorts of library operations are considerably faster:

    1. Importing images is noticeably faster, as the entire structure for each image is built in the library, it isn't bottlenecking the volume with attempts to copy the master simultaneously. In fact, I was somewhat shocked at how fast it is with the new aperture 3 import routine... As in "Wait, did that really just finish?"

    2. Here's the kicker. Fragmentation affects one thing in the aperture library the most - the dang AP.thumbnails and AP.minies files. Since these are Giant BLOB-like files of every thumbnail in the project, they are continuously updated while a file is being imported and adjusted, they get very fragmented (you can confirm this in idefrag). Putting the masters on a separate volume helps mitigate this tremendously.

    3. General library loading operations are faster. Switching between projects/albums, Loading thumbnails while simultaneously loading the selected raw image...

    4. Generating previews doesn't slow the storage system to a crawl, especially while applying adjustments, and thumbnails get reprocessed.

    Now, if you have a fast enough RAID, the difference might not be very noticeable, but for a mac pro user with just three internal drives - you can get comparable to RAID performance just by optimizing your setup to account for the needs of each asset - the library needs fast random small block read/write speed, and the masters need fast sustained large block read speed.




    In my case I have a 2TB external (enclosure has 2x1TB drives NOT RAID) so having the AP3 Library on one drive and the original RAW files on the second enclosed drive referenced, this will theoretically give me better performance?

    I'm trying the managed route right now but may consider referenced.

    I will then have ANOTHER 2TB external to backup the one mentioned above.
    MBP 2.5Ghz-4gigs ram, Mac OS X (10.6.2), 20" ACD, LaCie storage, OWC Mercury drives
  • gjsmith Calculating status...
    I have a single managed Library, currently about 6 TB on a 5-disk eSata array. I selected the managed option for many of the reasons you listed, Kevin.

    I will probably convert to a referenced system, however, because of these issues:
    1. Speed. Or lack thereof. The Library takes forever to open, even though Aperture itself opens quite quickly. Rebuild is an 8 to 12 hour ordeal.
    2. Scalability. I'm nearing the point where I need an additional 10TB of space. Under the managed option, I would have to create, maintain, and work with two physical libraries. I'm not certain I could stomach that.
    3. Vault inoperative. Around the 3 TB level, vault ceased to function. Apple support couldn't figure out the problem, so the "solution" was for me to maintain a copy of my Library on a separate array.

    Gordon
  • nrci Calculating status...
    For my $.02

    Absolutely referenced. I have a library just shy of 1TB which used to run on my main hd when it was smaller. Terrible, terrible, terrible performance. The only benefit was that I could edit any image when ever I wanted.

    Moved all to referenced and on it's own Raid drive. (most) problems solved. It is the only way to go. Aperture is really not designed to manage its own files. It's basically iPoto.

    Apple should simply admit as such and enable off-line editing.

    Make the move, you'll be glad you did.
    n
    MBpro 3.06Ghz 8GB RAM, iMac G5 needs rebuilding, Mac OS X (10.6.2)
  • Matthew Bergsma Level 3 Level 3 (600 points)
    Currently Being Moderated
    Mar 8, 2010 11:27 PM (in response to Falcon01)
    I think you would notice a performance boost by splitting it up, especially if you are using a macbook with 2 1tb firewire drives.

    But here's what I would do:
    I'd use File->Relocate masters to move the images outside of the library into a subfolder structure of your choosing on your externals, and then I would put your library on your internal drive.

    This way you can do a whole bunch of work even while you are not connected to the externals. Just set your preview size to something reasonable, and turn off automatically generate previews, and your library size will be fairly small. (For reference, my library of 30k images All referenced is about 16GB with faces on and previews set to quality 6, 1920x1920 max size).

    Funny thing - Look at the "Aperture speedup tip" on http://xlr8yourmac.com/ tonight... Too bad this could all be avoided if people understood some details about the disk overhead required by aperture.
    Mac pro 3.2x8, macbook pro 2.4, Mac OS X (10.6.2)
1 2 3 ... 10 Previous Next

Actions

More Like This

  • Retrieving data ...

Bookmarked By (2)

Legend

  • This solved my question - 10 points
  • This helped me - 5 points
This site contains user submitted content, comments and opinions and is for informational purposes only. Apple disclaims any and all liability for the acts, omissions and conduct of any third parties in connection with or related to your use of the site. All postings and use of the content on this site are subject to the Apple Support Communities Terms of Use.