Looks like no one’s replied in a while. To start the conversation again, simply ask a new question.

Call for Referenced vs. Managed Definitive Opinions

Hi all...

I would like to solicit opinions from experienced Aperture users with large, mature libraries, referenced or managed.

My goal is to develop a singular and final well-considered opinion on the pros and cons of each method. Please no conjecture, only facts that you are able to discuss and results that are real and repeatable.

First I will explain my current personal position on this discussion and why. Next I will layout my current understanding of the pros and cons of each method. Hopefully facts will arise out of this exchange of ideas that are not currently in my understanding.

While I do remember an uproar that occurred during Aperture v1 with people worried about the wisdom of entrusting their photos to the Aperture managed library system. The reasoning, if I recall correctly was that people were concerned with the idea of some closed, proprietary monolithic file format that they could not be easily accessed outside of Aperture. They made their dissatisfaction with the managed method clear, and expressed a desire to maintain their files themselves, in Finder-readable format, and access them via reference in Aperture.

There was no discussion or defense of the attributes of managed method from Apple, or an explanation that the managed Library was not a proprietary format, just a delivery of the referenced method in response to user complaints.

To be honest, when the feature of referencing masters became available, I read all the info from Apple and the user feedback. I then realized that to convert from managed to referenced for me at that juncture was too long of a task, with a payback that was of dubious value to me at the time, so I just remained with the Aperture managed Library System.

I have not had any real problems using the managed method. I have had my share of file issues like missing previews or permissions problems, but the fixes have all come through Aperture's rebuild system. I have not experienced an unreadable or unusable Library, although I have read about those who have had this problem. It is my opinion that the victims of these problems did not maintain their data as I do, and the problems arose out of heavy fragmentation finally culminating in file/directory mismatch and ultimately file/Library corruption.

I hope everyone by now understands that the Managed Aperture Library is not a closed, proprietary, monolithic system. It is simply a container holding a full hierarchy of Finder-readable files, with your masters organized in their own folder labeled "Masters" in a simple day-date-time format.

The Aperture Managed Library is nothing but a FOLDER, with special attributes. Instead of simply opening in response to a double-click, this folder initiates the FOLDER ACTION of launching its connected application, and instructing said application to read the contents of the folder.

This folder action is bypassed if the user right-clicks on the Aperture Library, selects "Open Package Contents" from the menu, and Viola! the folder opens, revealing the entire contents of the Aperture Library in Finder readable form. Now you obviously do not want to move or delete anything within the managed folder (after all, you did ask Aperture to manage it), just copy files out of there if you need them.

To my mind, at this point in the discussion, pure managed Library operation offers the following:

PROS
1) Single control location for all my Aperture data. I cannot lose any pieces as I could in referenced operation.

2) Simple, single operation to copy, defrag and maintain.

3) Vaults system within Aperture provide me a simple way of making fast archive backups to multiple locations.

4) Vaults also provide a safety net, trapping deleted masters in the event I did not want to have deleted them.

CONS
1) Masters are not Finder-accessible by other applications.

2) User cannot choose the storage hierarchy design of the master files

3) User cannot maintain a library on a relatively small storage medium, like a laptop for easy portability, and access to Library previews for slideshows and Media Browser use in other apps.

Pure referenced Library operation offers the following:

PROS
1) User can maintain a Library on a laptop, and organize images without being connected to large storage for the master files, with the exception of editing.

2) User gets to determine where and how master files are stored and maintained on disk(s).

3) Very large files, like long format video can be stored offline, and only brought online when needed by Aperture, preventing Aperture library from getting excessively large.

CONS
1) Users are completely responsible for maintaining storage of the Masters. Master files can get misplaced or lost if user makes a mistake in managing them.

2) All files are not in a single location making defragmentation, copying and backup of the media they are stored on more complex.

3) Masters are subject to editing by other programs, possibly changing their content, and therefore changing work done in Aperture, possibly rendering work done in Aperture useless.

The preceding points are my understanding of the two methods at this point in time.

For me, a managed Library is the way to go because it is self-contained. I cannot risk having Masters changed or lost through simple human error. In Aperture, the Master is the basis of all image manipulation. Aperture does not edit the master, but treats if like an original negative, only adding adjustment recipes referencing its content. If the master itself was to get manipulated by another program, the reference Aperture counts on as being unchanged would be invalidated, and the previous adjustment recipes within Aperture would be wrong.
In a Managed system, it is on the same media, and can be maintained simply. Complex storage schemes are prone to human error. In managed, all copies of the Library contain all the data required. Backups and media maintenance are greatly simplified.

I did some experiments taking referenced masters and storing them into a read-only disk image format. Obviously, this requires a bit more work upfront that the user would have to handle themselves, since Aperture knows nothing of creating this kind of storage. It also would require the user maintain the editable hierarchies of masters so they can be added to at any time. There is also a significant amount of time spent in Disk Utility, writing these massive read-only images, and the user would have to open the image files to make the masters available to Aperture when they needed to import or edit, which is also very time consuming.

This said, this method solved 2 problems for me:

1) The masters themselves can never be edited, moved individually, lost, or corrupted as they are read-only, so it removes the problem of possible destructive editing or deletion.

2) The building of a read-only volume in a disk image makes an unfragmented copy of the Master's data. They will never run the risk of fragmentation because the volume is read-only.

While that did make a case for referencing, it did not provide a performance increase over Managed, and actually took a great deal more effort to do than the maintenance of the Managed Libraries, that typically occur unattended at night.

As far as retrieving masters from a managed library, it was simple given the day-date-time hierarchy employed by Aperture internally. Once on the date, I just selected the file(s) by name, and copied them out, no problem.

OK, now I would like to hear from others who have have opinions regarding either method of storage, especially if they have found advantages or disadvantages of which I am not as yet aware.

Sincerely,

K.J Doyle

MBP 17" Glossy HiRes 2.6 6GB RAM, NVIDIA 8600 GT Video w 512MB, Mac OS X (10.6.2), 30" Cinema Display and External eSATA RAID for Library

Posted on Mar 7, 2010 6:44 PM

Reply
142 replies

Mar 7, 2010 7:22 PM in response to Kevin J. Doyle

Haha, Kevin, you know where I stand 🙂

Referenced.

Managed offers only two "pros" and they are:
1. Extra protection from accidentally mucking around in the finder and removing images.
2. The masters get backed up in the vault.

The cons are much more numerous -

Slow library operation, slow vault updates, fragmentation on a single drive for 20MB+ images, Eventual "storage ceiling" on one drive, and the biggest reason: A 700GB library is just not maintainable. Many people encountered this trying to upgrade to version 3 -

Mar 7, 2010 7:44 PM in response to Matthew Bergsma

IMO this "fragmentation" thing is a ruse. There should be no differences in performance between a referenced and managed library, unless the drive the files themselves on differ.

I've used both (primarily managed images) and have found no performance increases for referenced. I frankly like the ability to use a vault with managed images.

True, with a 700 GB image it can get difficult to use a managed library, but it's not impossible if you have a 2 TB drive dedicated to the task. I would never let a drive get more than 85% full for performance reasons, so with a library that large I'd use a 2 TB drive. Thankfully, they've been available for about a year now. It's true that updating an Aperture 2.x to 3.x library that's 700 GB in size would be painful. My library is only 250 GB and I have a 1 GB mirrored drive that holds it; upgrading was no problem at all.

Mar 7, 2010 8:51 PM in response to Kevin J. Doyle

I don't really see a much of a difference in either method, obviously discounting the x-factor being the ability to screw up your library by messing with it on your own in the referenced model. So really that makes this more a matter of personal choice. For me, I prefer referenced. That way, my libraries stay relatively small, and are nimble enough for me to backup/copy pretty easily. I have 1.6TB of images referenced onto a 2TB drive, and trying to copy a managed library or libaries holding those files would be very time consuming. Instead I have them split up into 3 libraries, ranging in size from ~50GB, to ~6GB, and it becomes very easy for me to make nightly copies of my libraries to external drives.

With the 2TB drive, I don't use it for anything else, so it's simply a container/locker. I dont have to worry about performance, fragmentation, etc.. it's a dedicated image drive. And I can trust myself as someone who knows the workings of the system/software not to go into the files there unless i absolutely have to. And when I do, the referenced model allows me to arrange them so specifically that I know exactly where I need to go.

The other part of this is that Aperture 3's new method of allowing projects to be exported as libraries is that much more of a bonus for me with this setup. If i'm going on the road for a week, and working on 4 or 5 projects while travelling, I can simply export them out, consolidate the masters into those projects, make my edits on the road, then merge when i return. I dont have to think about having my entire library on me, possibly losing senstive data, etc...

Again, it seems more of a matter of personal choice. I think referenced works best for my setup.

Message was edited by: Abbstrack

Mar 7, 2010 9:18 PM in response to Matthew Bergsma

Matthew Bergsma wrote:
Haha, Kevin, you know where I stand 🙂

Referenced.

Managed offers only two "pros" and they are:
1. Extra protection from accidentally mucking around in the finder and removing images.
2. The masters get backed up in the vault.

The cons are much more numerous -

Slow library operation, slow vault updates, fragmentation on a single drive for 20MB+ images, Eventual "storage ceiling" on one drive, and the biggest reason: A 700GB library is just not maintainable. Many people encountered this trying to upgrade to version 3 -


Hi Matthew,

Yes. I do know where you stand...and because you have much more time in referenced than I do, your input is really valuable to expanding my knowledge in this topic. I want to clarify a couple of things if I may...

"Slow library operation..."

I have heard this from many, and I want to see the difference. However, this one has me stumped at the moment. SInce the A3 upgrade I have played with this performance issue in referenced quite a bit and have not as yet been able to see any speed differences. I did a number of batch operations on 500 files or so, no significant difference. I did a 500 jpeg version export from both setups with a 500GB library and got very similar times. I was able to speed it up if I put the referenced masters on a faster disk than the library, but I then just would place the managed all on the faster disk and get the same execution times. Under what circumstances have you seen faster operation out of referenced?

As far as the program is concerned, the library file structure of both systems are identical. All Aperture libraries are the same internal folder hierarchy, the only difference ref v. man. is that the folder(s) that hold the masters can be anywhere, on any disk or offline. In man the masters folder is a folder in the Library folder, stored in a year-date-time format.

The question I have in performance is that unless you select an image, and make it connect with the master, like the system would in batching something or exporting...ref and man are exactly the same, the library is exactly the same files both systems are only looking at the library without the masters, previews and thumbnails are identical in size and position.

"fragmentation on a single drive for 20MB+ images"

Fragmentation effects all files and both systems, and if managed on a regular basis, neither scheme sees that problem.

"Eventual "storage ceiling" on one drive"

True, except with this many files this size either system should use an array for performance reasons, so the limit is not an issue.

"and the biggest reason: A 700GB library is just not maintainable. "

OK, we have 4 libraries here larger than 700GB, one being 1.4TB and they are maintained without issue using the techniques I have described, and all upgraded successfully to A3 in the same weekend. All run on duplicated RAID 0 pairs in an eSATA array.

One of the beauties of Aperture 3 is we can also have hybrid libraries, where some images are managed and others referenced. I can see referencing any video files that are of considerable length, because it is not efficient to store such data in the Library, and if it is something like uncompressed HD, it needs to be on a fast array that can do more that 220 MB/sec to playback without stuttering, so I can see that being referenced for sure. I have not tried this yet, but I wonder if A3 maintains a thumbnail Quicktime copy of a hi-res file like that so you can edit it with the real media offline.

Sincerely,

K.J. Doyle

Mar 7, 2010 9:31 PM in response to William Lloyd

William Lloyd wrote:
IMO this "fragmentation" thing is a ruse. There should be no differences in performance between a referenced and managed library, unless the drive the files themselves on differ.

I've used both (primarily managed images) and have found no performance increases for referenced. I frankly like the ability to use a vault with managed images.

True, with a 700 GB image it can get difficult to use a managed library, but it's not impossible if you have a 2 TB drive dedicated to the task. I would never let a drive get more than 85% full for performance reasons, so with a library that large I'd use a 2 TB drive. Thankfully, they've been available for about a year now. It's true that updating an Aperture 2.x to 3.x library that's 700 GB in size would be painful. My library is only 250 GB and I have a 1 GB mirrored drive that holds it; upgrading was no problem at all.


Hi William,

As I said above I have not been able to setup a scenario where I could see performance differences, either. I am hoping through this polling process to see if there is such a setup to benchmark, showing one faster than the other, as better performance is always a good reason to look at doing things a certain way.

Fragmentation, however is no ruse...it is absolutely measurable, and correcting it results in solid repeatable benchmarks on all systems. Kept in check it maintains, or in the case of so many who had fragmentation problems trying to upgrade to A3...when corrected reclaims the performance lost. It is a disk based phenomenon, caused by app operation and the os disk subsystem management.

Thanks for your input,

Sincerely,


K.J. Doyle

Mar 7, 2010 9:45 PM in response to Abbstrack

Abbstrack wrote:
I don't really see a much of a difference in either method, obviously discounting the x-factor being the ability to screw up your library by messing with it on your own in the referenced model. So really that makes this more a matter of personal choice. For me, I prefer referenced. That way, my libraries stay relatively small, and are nimble enough for me to backup/copy pretty easily. I have 1.6TB of images referenced onto a 2TB drive, and trying to copy a managed library or libaries holding those files would be very time consuming. Instead I have them split up into 3 libraries, ranging in size from ~50GB, to ~6GB, and it becomes very easy for me to make nightly copies of my libraries to external drives.

With the 2TB drive, I don't use it for anything else, so it's simply a container/locker. I dont have to worry about performance, fragmentation, etc.. it's a dedicated image drive. And I can trust myself as someone who knows the workings of the system/software not to go into the files there unless i absolutely have to. And when I do, the referenced model allows me to arrange them so specifically that I know exactly where I need to go.

The other part of this is that Aperture 3's new method of allowing projects to be exported as libraries is that much more of a bonus for me with this setup. If i'm going on the road for a week, and working on 4 or 5 projects while travelling, I can simply export them out, consolidate the masters into those projects, make my edits on the road, then merge when i return. I dont have to think about having my entire library on me, possibly losing senstive data, etc...

Again, it seems more of a matter of personal choice. I think referenced works best for my setup.

Message was edited by: Abbstrack


Hi there,

You just gave me a reason for faster operation (over time) with referenced masters...providing they use your setup...

"With the 2TB drive, I don't use it for anything else, so it's simply a container/locker. I dont have to worry about performance, fragmentation, etc.. it's a dedicated image drive. "

OK, as long as that drive is PURE image dedication...meaning only adding new images, never duping, or moving within the drive, and no stupid Spotlight indexing or any other interloping background process app writing to the disk, this drive should theoretically not fragment over time.

I would be very interested if you would run the free demo of iDefrag on that disk to look at the number (or lack thereof), fragments.

What I am saying is while this would be no faster doing things Aperture tasks vs. a freshly loaded managed Library, if neither is maintained for fragmentation, the Managed library will slow down over time MUCH faster than will this setup.

Of course a referenced library where the masters are not stored on a dedicated drive would have no such advantage....or if you were to put your Library on the same volume as the Masters, your advantage would be lost.

Plus, having all masters on one volume make them easy to backup and hard to lose.

Thanks for your input,

Sincerely,

K.J. Doyle

Mar 7, 2010 11:17 PM in response to William Lloyd

On a single drive, the combination of simultaneous reading and writing for 8-10 tiny metadata files and 1 large RAW file is a recipe for fragmentation. Especially with RAW files over 20MB, because those won't get automatically defragmented by the OS.

This is what I stated in my first post (fragmentation on a single drive). On separate drives this isn't an issue, which is why there is a performance benefit because you are removing the storage bottleneck of simultaneously reading a large block raw file at the same time writing to small block library files.

The problem is, that with a managed library you CAN'T gain the speed benefit of separate volumes. Anyone who has done server tuning for database hosting would agree with this storage speed analysis, it isn't really a mystery.

A lot of people think that a RAID will speed things up for them here, but the RAW files themselves don't need a RAID - it's the library itself that needs fast writing and reading. This file structure benefits more from an SSD than it does from a RAID.

The masters themselves are small enough, and are only read 1-3 files at a time, so a RAID isn't necessary for speed with the masters.

I prefer to update my vault frequently, and enjoy the fact that it goes very fast since the RAW files aren't going into it!!! I also like the ability to take my entire library (about 15-20GB for 35k images) onto a laptop with me for offline access of the projects. (Although with aperture 3's new library merge/sync feature this isn't as much of an issue).

However, putting all your masters and your library on one 2tb volume sure is a case of putting all your eggs in one basket - Sure you can back it up, but that takes TIME.

It's nice to be able to distribute 4-5 copies of a referenced library onto various mediums. (Fits well within Bluray limitations).

Additionally, when it comes time to "cleanup" or archive at the end of a year, its very easy to pull the master files offline and leave your library and projects intact.

The real kicker is that with a "referenced" library, you don't have to change anything about your workflow. You just setup the import dialog to store the Masters in a project subfolder hierarchy of your choosing, and it is pretty much identical to the managed library structure, except that it is in a folder of your choosing (and more importantly) a volume of your choosing.

I listed the only two "Pros" for a managed library I could think of, I'd be really curious if there are any I missed.

Mar 7, 2010 11:32 PM in response to Kevin J. Doyle

On one drive, referenced vs managed speeds will be no different. On separate drives, one designed for fast small block read/write, and one designed for fast sustained read speed, All sorts of library operations are considerably faster:

1. Importing images is noticeably faster, as the entire structure for each image is built in the library, it isn't bottlenecking the volume with attempts to copy the master simultaneously. In fact, I was somewhat shocked at how fast it is with the new aperture 3 import routine... As in "Wait, did that really just finish?"

2. Here's the kicker. Fragmentation affects one thing in the aperture library the most - the dang AP.thumbnails and AP.minies files. Since these are Giant BLOB-like files of every thumbnail in the project, they are continuously updated while a file is being imported and adjusted, they get very fragmented (you can confirm this in idefrag). Putting the masters on a separate volume helps mitigate this tremendously.

3. General library loading operations are faster. Switching between projects/albums, Loading thumbnails while simultaneously loading the selected raw image...

4. Generating previews doesn't slow the storage system to a crawl, especially while applying adjustments, and thumbnails get reprocessed.

Now, if you have a fast enough RAID, the difference might not be very noticeable, but for a mac pro user with just three internal drives - you can get comparable to RAID performance just by optimizing your setup to account for the needs of each asset - the library needs fast random small block read/write speed, and the masters need fast sustained large block read speed.

Mar 8, 2010 3:21 PM in response to Kevin J. Doyle

I would be very interested if you would run the free demo of iDefrag on that disk to look at the number (or lack thereof), fragments.


based on iDefrag, 82,806 total files on the drive (all referenced files), 866 of which are fragmented.

Of note is that in the past I have tried running aperture off this drive (i no longer do, i partitioned my internal HD and created a space for my Aperture libraries), and I just recently turned spotlight indexing off, so its possible those are more to blame for the fragmented files.

At any rate, I'm not a DBA, but those numbers would seem to be pretty positive.

Message was edited by: Abbstrack

Mar 8, 2010 3:42 PM in response to Abbstrack

Abbstrack wrote:
I would be very interested if you would run the free demo of iDefrag on that disk to look at the number (or lack thereof), fragments.


based on iDefrag, 82,806 total files on the drive (all referenced files), 866 of which are fragmented.

Of note is that in the past I have tried running aperture off this drive (i no longer do, i partitioned my internal HD and created a space for my Aperture libraries), and I just recently turned spotlight indexing off, so its possible those are more to blame for the fragmented files.

At any rate, I'm not a DBA, but those numbers would seem to be pretty positive.

Message was edited by: Abbstrack


That is a very good number, especially if Spotlight had been running on the volume for any length of time.

Mar 8, 2010 8:52 PM in response to Matthew Bergsma

Matthew Bergsma wrote:
On one drive, referenced vs managed speeds will be no different. On separate drives, one designed for fast small block read/write, and one designed for fast sustained read speed, All sorts of library operations are considerably faster:

1. Importing images is noticeably faster, as the entire structure for each image is built in the library, it isn't bottlenecking the volume with attempts to copy the master simultaneously. In fact, I was somewhat shocked at how fast it is with the new aperture 3 import routine... As in "Wait, did that really just finish?"

2. Here's the kicker. Fragmentation affects one thing in the aperture library the most - the dang AP.thumbnails and AP.minies files. Since these are Giant BLOB-like files of every thumbnail in the project, they are continuously updated while a file is being imported and adjusted, they get very fragmented (you can confirm this in idefrag). Putting the masters on a separate volume helps mitigate this tremendously.

3. General library loading operations are faster. Switching between projects/albums, Loading thumbnails while simultaneously loading the selected raw image...

4. Generating previews doesn't slow the storage system to a crawl, especially while applying adjustments, and thumbnails get reprocessed.

Now, if you have a fast enough RAID, the difference might not be very noticeable, but for a mac pro user with just three internal drives - you can get comparable to RAID performance just by optimizing your setup to account for the needs of each asset - the library needs fast random small block read/write speed, and the masters need fast sustained large block read speed.




In my case I have a 2TB external (enclosure has 2x1TB drives NOT RAID) so having the AP3 Library on one drive and the original RAW files on the second enclosed drive referenced, this will theoretically give me better performance?

I'm trying the managed route right now but may consider referenced.

I will then have ANOTHER 2TB external to backup the one mentioned above.

Mar 8, 2010 9:50 PM in response to Kevin J. Doyle

I have a single managed Library, currently about 6 TB on a 5-disk eSata array. I selected the managed option for many of the reasons you listed, Kevin.

I will probably convert to a referenced system, however, because of these issues:
1. Speed. Or lack thereof. The Library takes forever to open, even though Aperture itself opens quite quickly. Rebuild is an 8 to 12 hour ordeal.
2. Scalability. I'm nearing the point where I need an additional 10TB of space. Under the managed option, I would have to create, maintain, and work with two physical libraries. I'm not certain I could stomach that.
3. Vault inoperative. Around the 3 TB level, vault ceased to function. Apple support couldn't figure out the problem, so the "solution" was for me to maintain a copy of my Library on a separate array.

Gordon

Mar 8, 2010 11:05 PM in response to Kevin J. Doyle

For my $.02

Absolutely referenced. I have a library just shy of 1TB which used to run on my main hd when it was smaller. Terrible, terrible, terrible performance. The only benefit was that I could edit any image when ever I wanted.

Moved all to referenced and on it's own Raid drive. (most) problems solved. It is the only way to go. Aperture is really not designed to manage its own files. It's basically iPoto.

Apple should simply admit as such and enable off-line editing.

Make the move, you'll be glad you did.
n

Mar 8, 2010 11:27 PM in response to Falcon01

I think you would notice a performance boost by splitting it up, especially if you are using a macbook with 2 1tb firewire drives.

But here's what I would do:
I'd use File->Relocate masters to move the images outside of the library into a subfolder structure of your choosing on your externals, and then I would put your library on your internal drive.

This way you can do a whole bunch of work even while you are not connected to the externals. Just set your preview size to something reasonable, and turn off automatically generate previews, and your library size will be fairly small. (For reference, my library of 30k images All referenced is about 16GB with faces on and previews set to quality 6, 1920x1920 max size).

Funny thing - Look at the "Aperture speedup tip" on http://xlr8yourmac.com/ tonight... Too bad this could all be avoided if people understood some details about the disk overhead required by aperture.

Call for Referenced vs. Managed Definitive Opinions

Welcome to Apple Support Community
A forum where Apple customers help each other with their products. Get started with your Apple ID.