Looks like no one’s replied in a while. To start the conversation again, simply ask a new question.

Aperture 3 Library Fragmentation causes problems

Hi All,

I have read a number of the complaints in A3 conversion, and I think this may help solve some of them.

We have 6 systems with Libraries from 500GB to 1.2TB here and converted them all last weekend without serious problems, and all are enjoying superb increases in performance as I write this.

We did not have the conversion problems others have suffered, and I think I know why:

1) Disable Faces - Saves time in conversion, can be done afterwards.

2) DO NOT reprocess masters in the conversion, it can be done as needed once you are running, PLUS - WARNING - it will adversely effect many v2 images, changing exposure and colors, etc. PLUS, the conversion will go MUCH faster, as you have given it less to do. Check out the Apple Knowledgebase piece: "Aperture 3: Discussion about differences in exposure levels with Aperture 3 RAW decoding"
http://support.apple.com/kb/HT3948

3) Use an EMPTY freshly formatted volume that is at least double the size needed.

4) When conversion is complete, COPY your converted library to to another new freshly-formatted volume with at least 40% free space before you use it.

Why?.....

The normal operation of Aperture has always resulted in some disk fragmentation, the larger the libraries the worse the problem. Always working from a copy that has just been made eliminates the great majority of the fragmentation, and ensures it does not become a performance issue.

Conversely, if someone just keeps using the same Library, it will just get more fragmented, and slower and slower until real problems develop. If they are also running out of disk space, then the fragments become fragments as the file system feverishly tries to fit all your data into a shrinking space. All this eats time, and given the size and number of files to deal with it has become a problem that NO AMOUNT of fast hardware will fix entirely. We need to deal with some database best practices....but trust me it works and A3 simply screams on our systems here 2-3 times as fast as A2.

To test this premise that conversion of a large library is going to result in a tremendously fragmented mess on the volume, I reconverted my last Aperture 2 library today from a backup from last week. It is 500GB and it completed conversion to A3 as described above in about 4.5 hours. I then looked at the volume with iDefrag, and over 50% of the file was fragmented. My original A2 file, was not fragmented of course as it had come from a backup. I then opened the converted file in A3, and it began to process previews VERY slowly, with really lousy disk activity read/written numbers reported in Activity monitor.

I stopped the process, quit Aperture3 and made a copy of the file to a fresh volume. I immediately noted the backup was going VERY slowly...I had not noticed this last week as all our first backups were done overnight. As a benchmark, an unfragmented 500GB file will copy in about 2.5 hours and a bit....this backup took over 6 hours! It had a lot of file fragments to assemble to put the copy together, and it all takes time and resources.

When the copy was complete, I opened it again in A3, and the preview processing raced right along. Even before it was complete, the data was snappy and available...when the preview finished, it was a screamer....as our systems are today.

We use iDefrag, by Coriolis Systems to look at the fragmentation on these large files. It is a $30USD utility, and invaluable in providing some reality into why your system is slow. I invite you view you Library and see where your performance is gone. http://www.coriolis-systems.com/iDefrag.php

Apple tries to make all of their programs looks simple and carefree to use...fine...I submit when a database is hundreds of gigs NOTHING is simple and carefree. Performance requires understanding and some simple maintenance. Would you buy a Porsche and not change the oil, run it in a small parking lot and then complain to Porsche about performance? Kinda the same thing...

For those who are interested...

Our basic daily operation has the Aperture library running from a volume that consists of a RAID 0 pair in slots 1 and 2 an 5-bay eSATA array. There is an identical RAID 0 pair that carries a backup of the first one in slots 3-4. There is a rotating single backup mechanism that is used for daily offsite rotation in bay 5.

Every night a full finder copy is made from the day's working RAID 0 pair to the other pair, and to the offsite disk.

The next morning, the operator will verify the backups have occurred without incident, swap the offsite mechanism out and then start the day's work ON THE OTHER RAID 0 PAIR that was the backup of yesterday's data. Why?.....The normal operation of Aperture has always resulted in some disk fragmentation, the larger the libraries the worse the problem. Always working from a copy that has just been made eliminates the great majority of the fragmentation, and ensures it does not become a performance issue. We actually use 3 pairs to have redundant backups, but I won't confuse the basic issue here with that discussion.

Conversely from our methods, if someone just keeps using the same Library every day, it will just get more fragmented, gradually becoming slower and slower until real problems develop. If on top of that they are running out of disk space, then the fragments become fragments as the file system feverishly tries to fit all your data into a shrinking space. All this eats time, and given the size and number of files to deal with it has become a problem that NO AMOUNT of fast hardware will fix entirely.

Given the volume of data we digital photographers keep collecting, we need to take responsibility and deal with some database best practices....but trust me it works and A3 simply screams on our systems here 2-3 times as fast as A2.

Will this fix everything, for everyone???

Of course not...the wide variety of machines and configs make it impossible to predict this. I do feel pretty confident I can reliably make A3 run really fast on our systems here... and to note, the 6 workstations I have discussed are operating on 2008 MBPs (4,1) 6G RAM with 30" displays, not even Mac Pros and we are enjoying excellent throughput.

Hope this helps,

Sincerely,


K.J. Doyle

MBP 17" Glossy HiRes 2.6 6GB RAM, NVIDIA 8600 GT Video w 512MB, Mac OS X (10.6.2), 30" Cinema Display and External eSATA RAID for Library

Posted on Feb 19, 2010 7:43 PM

Reply
164 replies

Feb 19, 2010 8:32 PM in response to Kevin J. Doyle

this is invaluable info, thanks for sharing. unfortunately, for some of us pros with less...resources i'll say...doing it your way is virtually impossible.

that said, i'm going to look into iDefrag...i'm thinking about creating completely new libraries once this 3rd attempt at recovering/reconverting an AP3 library, exporting projects out of my current libraries and importing them into those new libraries as a way of trying to clean them.

Library rebuild has been an immense failure for me, so any new tips are welcomed. much appreciated.

Feb 19, 2010 8:41 PM in response to Kevin J. Doyle

Thanks Kevin for the long post.

I agree wholeheartedly and often use idefrag to defragment the library itself. Particularly the AP.minies files and AP.thumbs files get very fragmented and this can slow the library down a lot.

Your post also brings up another great reason to use referenced masters! The library is a lot easier to maintain and does not fragment as much when its on its own dedicated volume, and the Masters are on their own dedicated volume.

Feb 20, 2010 11:51 AM in response to Kevin J. Doyle

Great post.

I too had no issues and loads of free disk space. I also unchecked the option to have Aperture reprocess images for adjustment brushes (giving the upgrade less to do) but I did - err, inadvertently - run face detection. 6Gb of free RAM on a 12Gb MPro and no other running processes. PLUS, I RAN DISK UTILITY to ensure no allocation errors before commencing (and there were some).

No issues on 5 libraries (all referenced) averaging 30Gb each. BTW, the upgrade makes a copy of the v2 library at the start of the upgrade process - hence minimum of 2x space, but because HDD's are disappointingly slower when > 50% full, the emptier and bigger the drive the better - Laptop owners this is a double whammy if your drive is 5400rpm or lower.

Feb 20, 2010 12:56 PM in response to Kevin J. Doyle

Even as a home user with small library (Libraries), there is something useful here for which thanks. That assumes that I will remember as my Library grows. Having said that and having read many recent Aperture 3 threads of woe, I wonder if there is merit in limiting Library size? A3 makes generating a new Library very easy and switching between them is a breeze. I have just created a second Library on my 24" iMac and all went well. My instinct is to limit library size for now at least and even long term.

Feb 21, 2010 7:56 AM in response to Kevin J. Doyle

as I am attempting to make a copy of my library which now seems to be fully recovered (fingers crossed) i'm noting even making a copy is moving extremely slowly. Is this indicative of said fragmentation?

I tried to copy first to an external via firewire 800, and it's a 48GB library..finder is estimating it would take about 2 hours to copy..then i cancelled out and am now copying to another folder on my internal hard drive. I did just do a clean install a little less than 2 weeks ago, so i can assume my internal hard drive still counts as a freshly formatted volume as well. Finder is still estimating a 2 hour transfer for a 48.57GB file from one folder to another, which obviously seems like quite a long time.

Feb 21, 2010 10:51 AM in response to Abbstrack

Abbstrack wrote:
as I am attempting to make a copy of my library which now seems to be fully recovered (fingers crossed) i'm noting even making a copy is moving extremely slowly. Is this indicative of said fragmentation?

I tried to copy first to an external via firewire 800, and it's a 48GB library..finder is estimating it would take about 2 hours to copy..then i cancelled out and am now copying to another folder on my internal hard drive. I did just do a clean install a little less than 2 weeks ago, so i can assume my internal hard drive still counts as a freshly formatted volume as well. Finder is still estimating a 2 hour transfer for a 48.57GB file from one folder to another, which obviously seems like quite a long time.


EXCELLENT POST...this is a perfect opportunity to clarify what I am talking about...

"I am attempting to make a copy of my library which now seems to be fully recovered (fingers crossed) i'm noting even making a copy is moving extremely slowly. Is this indicative of said fragmentation? "

YES...I can just about guarantee you have a HEAVILY fragmented library. Any library that has just endured some significant manipulation by the program is typically fragmented. A freshly converted v2 to v3 Library will be a ton of fragments. This means the read operation the hard drive has to go through involves repositioning the head again and again to assemble the fragments so they can be written to the target disk space. This takes a lot of time, and gets worse fast with big files.

"I did just do a clean install a little less than 2 weeks ago, so i can assume my internal hard drive still counts as a freshly formatted volume as well."

NOOOOO! NOT EVEN CLOSE! Freshly formatted means RIGHT NOW, TODAY, and without anything else being stored on it or done to it. It also means BLANK, EMPTY OF DATA. This is to ensure that the disk space you are making the copy onto is completely free of data and able to record the library in a contiguous block of data. Making a copy onto another volume that already has a bunch of data on it actually risks fragmenting the file more. Now this FRESHLY FORMATTED volume can also be a new clean partition of another drive, but I recommend using separate mechanisms for safety sake. You can have a number of fresh VOLUMES on the same drive, but if the drive fails you lose them all...not a good strategy.

Also, while we are at it...your internal boot VOLUME is NO PLACE to be storing a large database like your Aperture Library or its backups. The simplest reason is that if you are storing your precious photographic data, it should be on a dedicated medium that does not see any disk activity other than that to operate Aperture. You want to make it easy on yourself to do regular disk maintenance to your critical Library storage mediums, not put your most important file (in our case the Aperture library) on a medium that will guarantee fragmentation because it is being used for everything else your computer does all day long. This is a critical error, and sadly made all the time. So give your Library a safe home all its own, NO ROOMMATES ALLOWED, lol.

In fact, when this copy of your 48GB file completes....ASSUMING it was copied to a disk that gave it empty, contiguous disk space to write to...when you make a copy of that copy (again to a FRESHLY FORMATTED volume), it should copy very fast. Basically if all the hard drive is being asked to do is read data sequentially from the drive, and not have to reposition the heads to get a bunch of fragments it is a FAR more efficient operation.

I would strongly suggest getting a copy of iDefrag, or some other program that shows disk fragmentation. Without such a tool, and dealing with huge files like Aperture libraries, you are operating BLIND. it will open your eyes to this problem and explain many of the issues and slowdowns people are attributing to Aperture. It will show you why we operate from the last night's backup copy of the library everyday. One look at the Libraries with a DeFrag program, and you will see what I mean.

You also will see why you should NEVER use a drive past 70% of is capacity or so. Given daily use, the contents of your boot volume will fragment over time, it is the nature of the beast. The amount of fragmentation from normal activity increases drastically when there is little unused operating space on the drive. Regular defrag activity, and not using your drives to capacity will keep your computer running like the day you unpacked it.

Let me know how it goes...

Sincerely,

K.J. Doyle

Feb 21, 2010 4:26 PM in response to Falcon01

Falcon01 wrote:
Kevin this is some really good info you've been sharing.
I've been contemplating a couple of things here:

1) AP3 library on Boot drive in Pics folder BUT using referenced files on an external FW drive

or Do you think the AP3 library managing the files but running on an external FW800 drive?


GREAT QUESTION....because it gives me a chance to do a bit of a soapbox regarding system architecture and data management and safety. Segregation is the key, here is why...

Apple has set a bad example, IMHO, by shipping computers that mix created content storage with executable files. It does a disservice to both, and promotes fragmentation of your boot drive making it gradually lose performance over time.

20 years ago, no IT professional would EVER purposely colocate these two datatypes on the same drive. Given the relatively (to today) lack of processing power, mistakes like that would kill your performance very quickly. In today's super capable processing computers, small files like word processing or a few photos or songs can be stored without killing the user's performance...still very bad form, but ignored because it is masked by the processing of the computer....and because marketing guys want to make this all seem simpler...fine, it works for the average user.

BUT...

There comes a time when the defrag monster, and the disk maintenance monster will not be constrained, and that is when you have thousands of large files and many gigabytes of storage involved. THEN, because it is magnified by the size and number of files, you start to have bad performance problems.

Judging from the posts in the forum, Aperture users, no all serious digital photographers have collectively reached that point...BUT, not to worry. The strategy employed 20 years ago really works great and soon you are throwing around hundreds of gigs of photos at super speed.

Now in our group we use all laptops...BUT the build on the laptops is sacrosanct. All of the machines have 500GB 7200 Seagate Momentus drives with over 300GB free. No created content save email is left on the machines. I use the term "left" as we will store pics from a tethered shoot in a library during the day, and dump them off the drive at night onto our portable external drive.

On the road, we all carry the Sonnet Fusion A2, which is an eSATA 1TB RAID 0 array that we connect to the MBP via the Sonnet Tempo Pro Express 34 card. It is blazing fast (190Mb/sec), and makes a super PS scratch drive when working in the hotel.

One alternative strategy would be to create a partition on your boot drive that contains ONLY the A3 Library (assuming referenced masters, of course), and was at a minimum, maintained at double the size of that file. I don't like this, because I want my work data on a separate physical mechanism that is not part of the computer for safety....BUT in fairness, it is convenient and if it is own volume would not be subject to the fragmentation it would see if left on the main boot drive with everything else. Also, if you do this remember to shut off Spotlight indexing for that volume as well, it just kills performance in this scenario.

The most important part of having this data (both pics and Library) external is that you can store copies of this irreplaceable, priceless data in another location. For our group, a backup mechanism goes home with each operator at night, and is rotated back in with the fresh overnight backup each morning. An extra copy is made and kept weekly in safe deposit. That way fire or theft will not destroy our work.

One last thing...if you are on a MacPro or MBP with an Express 34 slot you can and should get away from FW800. eSATA is 3-4 times faster and not subject to the FW800 issues. I would also caution that you should have no more than 2 storage devices per FW 800 bus, one ideally. FW800 has bus arbitration issues and can get lots slower and unreliable with a number of units on bus. If you are going to use FW800 or are stuck with it, DO NOT use anything but disk drives on the bus....no cameras, card readers, etc. Many of these devices are not designed well and make the FW800 bus problems even worse.

Anyway, off my soapbox, lol, hope this helps...


Sincerely,

K.J. Doyle

Feb 21, 2010 4:51 PM in response to Kevin J. Doyle

thanks again for the info Kevin.

According to your solution however, I'd have to purchase 1-2 more drives (in addition to the 2 i'm currently using, 1 for my referenced masters, another for backing up my my cpu)...not that it's impossible, but obviously not the most economically feasible solution..but the tradeoff like you mentioned is safety and performance, so i'll have to consider it.

Also when I think about the fact that of my 3 libraries, i've had trouble upgrading two of them, and the one that has given me the most trouble aside from being my largest, is also my oldest (it dates back to 1.5)..i've rebuilt it several times in AP2, but never defragged it..during my most recent recovery of this library (there have been 3 in the past week) after a rebuilding failure, it took over 40 hours to process 140,000 items...more than triple the time it took on the previous recovery a few days earlier..so i think this lends to your theory of fragmentation pretty well. I think my best option for this library is to migrate everything out of it into a new one, as this one is over 3 years old and probably not the best place to keep any projects.

Feb 21, 2010 8:56 PM in response to Abbstrack

Abbstrack wrote:
thanks again for the info Kevin.

According to your solution however, I'd have to purchase 1-2 more drives (in addition to the 2 i'm currently using, 1 for my referenced masters, another for backing up my my cpu)...not that it's impossible, but obviously not the most economically feasible solution..but the tradeoff like you mentioned is safety and performance, so i'll have to consider it.

Also when I think about the fact that of my 3 libraries, i've had trouble upgrading two of them, and the one that has given me the most trouble aside from being my largest, is also my oldest (it dates back to 1.5)..i've rebuilt it several times in AP2, but never defragged it..during my most recent recovery of this library (there have been 3 in the past week) after a rebuilding failure, it took over 40 hours to process 140,000 items...more than triple the time it took on the previous recovery a few days earlier..so i think this lends to your theory of fragmentation pretty well. I think my best option for this library is to migrate everything out of it into a new one, as this one is over 3 years old and probably not the best place to keep any projects.


Glad to help...

The best way to basically eliminate this problem is to alternate 2 Library dedicated disks. I use two 2-drive striped RAID 0 arrays, but even 2 single mechanisms would be fine.

This works because copying a single large file to an empty disk guarantees that the copy will be made into contiguous space, with no fragments.

1) Start with 2 new external disks, at least twice as large as the library they will hold.
2) Format the disks, and name them differently, like A & B. Neither is backup or source, depending on the day they both can be either.
3) Use eSATA connectivity if possible, you would need a MacPro with an eSATA host card or a MBP with an Express 34 slot. Unfortunately, other machines will be stuck with FW800, sadly it is 3-4 times slower than eSATA.
4) DO NOT USE USB Drives, as they are 10 times slower, and their lack of speed will adversely impact operation.
5) Store ONLY the Library on the drives, that way the only source of operational fragmentation is controlled.
6) Every day you work with the library, when you are done make a full copy of the Library to the other drive. Assume you work with drive A on Monday, copy A to B on Monday night.
7) The next day you go to work with Aperture, connect to the Library you copied. On Tuesday connect to Library on drive B, at the end of that session copy it over to drive A.

Add to this setup a third drive you make a copy on once a week, and store that drive offsite for safety.

Hope this helps,


Sincerely,

K.J. Doyle

Feb 21, 2010 9:08 PM in response to Kevin J. Doyle

Hi Kevin - may I simply add my thanks for this timely and masterly exposition on good database management practice and on how those of us who have been following the default Apple practice of leaving one's iPhoto and / or Aperture Libraries in the Pictures Folder in our Home Folders. Despite making regular backups (SuperDuper! - Smart Update) of the volume I've been getting increasingly uneasy and using iDefrag (terrific utility) on the whole volume is a bit of a heavyweight and time-consuming operation. I've been OK with Aperture 2 except for some performance issues - very likely a consequence of fragmentation; but Aperture 3 has really bitten the dust.

So - it's bite the bullet time and completely re-organize my internal hard drives - and re-allocate my external drives.

Again my thanks. And to think I used to work in the field of very high speed 'real time' relational databases!

Keith Dixon

Aperture 3 Library Fragmentation causes problems

Welcome to Apple Support Community
A forum where Apple customers help each other with their products. Get started with your Apple ID.