I am currently having big trouble with Aperture (2.1.4) running on my iMac 24" 3.06 GHz (running 10.6). The problem has nothing to do with the Snow Leopard/Aperture issue: I had the same problems running Leopard.
It takes just painfully long to adjust images (white balance, rotating, etc.). I see this rotating rainbow all the time, so frustrating! I only use RAW images from my 400D.
My library is currently 135GB big. Should I split it or something, so that Aperture can handle it better?
If so, is there a 'maximum size' the library should be?
Or is there an other solution for this problem?
Kevin,
Further to your posts about RAM and Ernie's follow-up, I got curious about the behaviour of my 4GB 24 inch iMac (May 2008). All the previous advice I'd seen stated that 2GB was the realistic minimum, but performance improved significantly with 4GB. This would appear to be consistent with my experience in use.
Aperture performs very well under Leopard and I've been extremely happy with it. After seeing another poster obsessing about the number of page-outs, I decided to see what my machine was doing.
Using iStat Pro and Activity Monitor, I apparently have had no pageouts at all, but over 100,000 page-ins after 3 days of use. This includes deliberately pushing the machine by browsing, using Aperture, Tiffen DFX2, OpenOffice, a photo import session etc more or less simultaneously.
Does this mean that there is something in the hardware platform that drives the need for additional memory, or is it the difference between Leopard and Snow Leopard?
The 8 core machine runs Aperture differently than a dual core laptop, as you would expect. Now Aperture is multiprocessor aware and does a ton in the background. If you look at your 8 core usage in Activity monitor, with only Aperture running, it is queueing fewer threads than a dual because it is getting through them in parallel, and therefore providing better performance. On my laptop, Aperture averages about 35 active threads according to Activity monitor, and the CPU monitor has both processors going as I am flipping through images.
The extra processing of the Mac Pro can make operation liveable and mask memory issues that if you switched the setup to a laptop would be a problem. PLUS...if you are running a discrete channel SATA array the sheer speed of the pageout is going to be 3-4 times as fast as the internal single drive on a laptop.
The pageout rule still applies, however...even with all your processing power, you will have demonstrably superior performance by having enough RAM to eliminate pageouts during your whole operation.
My recommendation for 6GB was based on the 6 laptop systems we are running here, especially my own laptop. I was unable to operate without generating pageouts if I dropped below 6GB, or ran other apps simultaneously. The number itself is not magic, just testing to see when pageouts occur and how to eliminate them is key. Due to our traveling, this is reality for us, a Mac Pro would work better, but we have to have images to meet press deadlines from the road, so these MBPs with SATA cards and small arrays are as good as it gets for us.
Another issue is the size of the libraries and the sizes of the files Aperture is being asked to work on. We have a mix of files low end about 16MB, and high end about 150MB from our Hasselblad, not counting the TIFF or PSDs in there from editing. Our average project is going to be 3000 or so shots, and we will be flipping through them fast as the machine can go in the initial culling stage.
We have 6 active operating libraries, averaging 800GB and at the office a single archive of all combined plus legacy material, scans, etc. totaling almost 9TB.
I did do a test with a 100GB library with smaller projects and smaller files and found that Aperture needed much less to work in that setup, although still would generate pageouts under heavy use, or when using less than 4GB of RAM.
Just to make sure we are on the same page...(no pun intended)...first describe your Library, average file size, average # of pics in Project and overall size. Then, I want you to specifically verify that you have no pageouts with Terminal. Simply restart your machine, which will reset the page counters, reboot, than open Terminal and type top (just three letters) and hit return. The fifth line on the display that starts with VM is what I want you to note. Given you are not running anything yet, the end of the line 5 should read zero pageouts. Quit terminal.
Now go ahead and exercise the machine, as you normally would.. run Aperture plus the usual PS and other apps, then launch Terminal and type top and hit return again. If you are reading pageouts on line five, you can gain performance by adding physical RAM until the same exercise results in a reading of zero pageouts after use. Once you have reached that state, adding faster storage systems improves your box, but little else save a newer machine.
As I said, the key to max performance is measure your actual setups in your operating environment. There is no magic amount, just what the system tells you is working without pageouts.
"Does this mean that there is something in the hardware platform that drives the need for additional memory, or is it the difference between Leopard and Snow Leopard?"
All of the above, that is why testing on your own setup is key...library characteristics have everything to do with performance. Remember the demo datafiles of Aperture will run blazing fast on any machine, we know that does not reflect professional reality.
I encourage you to do the exercise I described above, and define your library as well.
Remember, page-ins are no problem, simply the program running normally, page-outs are the performance stealers.
Kevin,
Thanks for the prompt response. My usage profile is somewhat less demanding than yours - less than 10,000 mainly raw images at 6MP, so around 6MB each, in projects ranging in size from 30 to 400 images. A few tiffs as well, from PS.
I did the exercise, and the pageout number was again zero, so call me a happy chappie! It will be interesting to see what happens when I eventually decide to install SL. I'm waiting for 10.6.3 before upgrading.
Very cogent response! I will test when time allows, as you suggest, but previous uses of "top", with lots of apps running, did not show any pageouts. This was not particularly targeted toward Aperture, as I remember, but rather at the size of real memory appearing to be used by mds in SL.
My Aperture Library is roughly 500 GB, with more than 34,000 images in 260 projects. Most images are .NEF from my D300 and some from my D200. My largest project composed only of RAW images is slightly over 1600 images from mix of three cameras (the third a friends D70), but more typically they are less than 300 images per project, the average (as you can calculate) around 130.
My Library is nearly all Managed, kept on the boot drive, which resides on a WD 1 TB Caviar Black. I have both a SL and Leopard boot on this model drive, but also have a SL clone on a WD 1 TB My Book connecting when used via FW800 -- it works also, but only recently switched that clone to be SL rather than Leopard. At various times I copy the Aperture Library I am using in Snow Leopard to the boot volume using Leopard, so I can check the same Library in either OS. Thus I am using full copy backup rather than Vault.
When I run FCP, the project files are on a separate drive from the boot, and the video assets themselves on still other drives in the MP -- never on the boot drive.
I am only using the Adobe Creative Suite 4 (Design Premium) in Snow Leopard, so I only keep Leopard around for testing and backup. My Leopard boot only has PS CS3, btw. I will often have Photoshop open while using Aperture for obvious reasons.
Doyle - great posts. In depth knowledge of large fast storage is, I agree, something that photographers need to start being aware of.
I also use striped arrays, using Port Multipliers. I have also had experience with running "many" ports in parallel too. With varying results.
My experience started in the video and music industries, where the need for speed is, frankly, "out there" compared to photography. The difference here is sustained reading, as opposed to fast access.
I have settled on Sonnet equipment, using Apple OSX soft raiding. Not the most sophisticated raid software, but at least guaranteed to work under any OSX. Your milage may vary, but so far OSX soft raid, via Disk Utilities, has not been inadequate for me as yet.
I wanted to add a few cautionary notes on setting up RAIDS.
I used to run a 4-way drive bay by Sonnet, that attached each drive to a separate eSATA bus. This was great as it went, until my drives got bigger than 500GB. I then hit major snags, disk failures, nightmares etc, and was scratching my head for some time and finally called Sonnet tech support. It turns out that the particular drive bay I was using, was only power rated to drives up to 500GB. My problem was revealed. The intermittent nature of the problem showed up as I swapped in drives of different capacities.
I then bit the bullet and upgraded my drive bays to the newer 5-way Sonnet boxes. I now own 3 drive bays, two of which are rack mounted. All rated up to 2TB drives per slot. Fantastic. At the same time, though, these newer boxes only ran in Port Multiply mode - there is only one cable per 5-drive box. The old box ran "directly"., with a eSATA port PER DRIVE. Thinking I would experience a speed hit, I again queried Sonnet. Now they said an interesting thing - their tests indicated that PM Raids actually perform FASTER than RAIDS with eSATA busses dedicated to each drive. This is contrary to what Doyle has found. I do agree with Sonnet though - my own tests (particularly on sustained large file reads) show that Port Multiplying, Sonnet style, is faster that parallel set-ups. I can't comment on other PM solutions as I have not experienced these.
But chiming in with Doyle, they also recommend running your drives at no more that 75% (or less) capacity. With port multiplication, you can run multiple streams of uncompressed HD video - we're talking hundreds of MB a second here. Speed takes a HUGE hit when your drives get 75% full or more.
So my warning is - double check the specs of the bays and equipment you are planning to configure. Think future proof too. Find out what capacity drives the bay you have your eye on is designed to host.
As a side note, I feel that a 2 way striped working disk is adequate for running Aperture, and striping up more than this is overkill, as seek times will come into the equation. This is my opinion of course, but be aware that seek times (the time it takes a disk to physically "seek" the start of a file) are actually reduced on a striped array - it's sustained read/write that would scream. This needs balancing - you want to balance seek times and read times. When it comes to HD video - no striped array can be parallel enough. For photos, where individual files never exceed 100MB really (that's a TIFF of course) it is overkill to stripe more than 2. So my own advice is, stripe yes, but I wouldn't stripe more than 2 drives. Save the bay slots for your backups.
Running the mirrors is a great way to create a backup routine, although personally I simply run a couple of vaults and swap these in and out. If you are referencing (all or part of) your library, then you really need a plan of attack for data security. This subject would deserve a separate discussion - but the rule as always is, not data is really secure unless it exists in at least three places, and is healthy and recently checked.
Let's look at the middle, cost wise here. Many professional users may find it hard to justify a 5-way (or more) disk system. For iMac users: I have found that a striped array, in Sonnet's 5-bay or 4-bay multi interface units can deliver pretty good speeds sitting on a FW 800 bus on an iMAc. These units allow hot swapping of SATA drives.
Yes it's true, you cannot attach eSATA to an iMac directly. What is not generally known however, is that the FW800 bus speed is rarely reached by most FW drives. The Sonnet box I refer to does make best use of that FW800 bus. By hosting 2 SATA drives, striped, and delivering the data via FW800, I can get acceptably snappy performance on a 1TB library on an iMac. At the same time, this little baby will allow you to hot swap out your backups, other referenced libraries etc etc. You're not f
*ing around with FW cables. Always dangerous to involve cable swaps when your data is concerned. With this Sonnet bay, you can grow your library, SATA drive by SATA drive, and eventually mature your whole thinking about your data. Now, I am not a Sonnet employee ,(and there are doubtless other solutions out there), but from my (not inconsiderable) experience with small data centres, a Sonnet setup, using soft raiding is the way to go. If you're on iMac, get the Quad Interface version. If you're on a Mac Pro, pay a little extra for a massive speed boost and get the E4P eSATA card. This is what I run on my other 2 workstations.
Anyone contemplating a new Mac - go for a tower, no question. Those committed to iMacs should, IMHO, visit Sonnet's website and check out the D400Q. The "Q" part stands for Quad Interface -meaning it supports FW800. If you're planning to use a SATA PCI card in a tower (read Mac Pro) then you don't want/need the Q, and can go for a D500P.
Hello again. Unfortunately, I have yet to set up a system based on your discussion. I've been doing some research to better familiarize myself with the topic. It's important so that I can fully understand my needs as a user.
Based on where we had left off, I'm just not sure the system is right for me. Don't get me wrong, the RAID 1 you described to me is a brilliant system. There are a few things that don't mesh well though. The first is that my computing environment is a home computing system. Therefore, taking the rotation disk with me isn't very practical. If I were computing at work, it would make a lot more sense to bring a disk home with me, but I can't see taking a disk to work with me every day, or taking one with me when I leave the house. I guess I could, but I can't envision it. The second thing is that I am not performing daily shoots, and don't edit on a daily basis.
Now, all of this may be irrelevant to your suggestions for the array that you have recommended, but I just want to see if you would recommend a different setup based on the type of user I am and my needs.
I want to clear photos (i.e. my Aperture Library) from my internal drive and work with them - edit them (when I am home, not from the road) with the fastest R/W speeds possible within reason. I have upgraded my RAM to 8 GB's, and have purchased the Sonnet eSata Pro. Now, I know that we both talked about data security and protection as being the most important, and it still is, but isn't there a way for me to work with my Aperture Library off of a 2 disk striped array, while also mirroring to a two disk RAID 1 array? I'm talking about a 2 disk array because I don't know if I need to swap out a hard drive off site. The main thing that I am concerned about from a data security/protection standpoint is my Aperture library and my iTunes music library. I was thinking that I could just burn that info to an ultra portable rugged external drive like a Lacie or WD passport. Again, this is not ideal, but would give me the peace of mind of having something off site (so would take care of the geographical concerns). I just can't see moving a regular HD around on a regular basis.
Based on what I am describing here, what would you recommend?
