Missing (most, but not all) masters from managed project

Question

Level 2

342 points

Missing (most, but not all) masters from managed project

I just found that I have a one project in my library where most of the raw master files (NEF) are missing. The project contains 160 photos, but only 14 masters are present and these are randomly distributed in the project (not sequential files). My entire library is managed (not referenced). All photos were imported at one time.

All the edited versions appear in the browser with no problem, including the "full" version and the Quick Preview ("P"). Oddly, when I press "M" to view the master image, it displays correctly with no adjustments, but the message "Master Image not Found" appears where it would normally say "Master Image." I only noticed the masters were missing when I tried to export a version and got the follwing message:

"Versions with unavailable master files cannot be exported. The selected master image is either offline or not found. Please reconnect it and try again."

I searched in the library and vaults for original the filenames, but they do not exist in either location. If all the masters were missing, I would have concluded that I imported that project as a referenced file and later deleted the masters, but 14 of the 160 are still there.

I have two questions:

First, does anyone have any idea how the master files could be deleted so I can avoid this in the future? I am beginning to think this is simply a corruption of the database (not reassuring).

Second, is there any way to recover the versions that clearly exist in the project so I can reimport these as masters? While viewing them in the browser, I can zoom in to the original resolution on both the master and the edited version, but can't perform any further edits to the image (although I can add metadata). I'd be happy to retrieve a JPEG version, even though it appears I have lost the original raw file.

By the way, I have already performed all three library repair options on Aperture launch (Repair Permissions, Repair Database and Rebuild Database).

Thanks for any help,

Dave

PowerMac, Mac OS X (10.6.7), 2009 2.66 GHz 8 Core, 16 GB RAM,

Posted on Sep 24, 2011 3:05 PM

Reply

Answer 1

Best reply

SierraDragon

Level 4

2,695 points

Sep 29, 2011 2:09 PM in response to David Strait

Sorry I cannot answer the where-are-my-pix question, except to suggest Spotlight searches on file names, and a careful review of the Trash folder.

As a database person IMO a really large DB populated with (relatively) large unchanging individual field contents should be avoided in favor of a smaller DB referencing the large unchanging individual fields. In Aperture that means I recommend using a Referenced-Masters Library.

Large files like a 300+ GB Library simply become unwieldy, and unwieldy leads to errors. Whether or not the errors are human PEBKAC or computer-generated does not matter. IMO Aperture should open with a warning message suggesting how to switch the DB to externally referenced Masters once a Managed-Masters Library reaches a certain size (say, 50 GB).

Whether using a Managed-Masters Library or a Referenced-Masters Library, backing up originals on external drives prior to import into Aperture or any other images app is the best way to provide real images security.

Good luck!

-Allen Wicks

Reply

Answer 2

Sep 29, 2011 2:56 PM in response to David Strait

Well, to get the jpeg versions that are there, you can simply drag the image from the Apeture library to the Desktop. Depending on the Preview preferences you had when you imported the images, you could have a decent sized image.

Regarding the missing, have you tried clicking on one of them and choosing, Locate Referenced File from the File menu? I know you're saying you run a managed library, but it would be interesting to see if it gives you a "There are no referenced files in this selection" message.

Reply

Answer 3

Sep 29, 2011 3:24 PM in response to David Strait

David Strait wrote:

I can zoom in to the original resolution on both the master and the edited version, but can't perform any further edits...

It sounds like your Previews may have been set to be the full size. If that is the case someone else here can probably tell you how to export the Previews, but in any event you can always recapture anything you can see on the display using OS X's Grab application and save as a TIFF file.

HTH

-Allen Wicks

Reply

Answer 4

Sep 29, 2011 3:31 PM in response to David Strait

Shutter's idea about the referenced Masters is a very good idea. If you inadvertently referenced images when importing (Save Files: In their current location) then your Library would be smaller and you would get the error message that you received.

Unfortunately if you imported direct into Aperture and have since reformatted the camera cards the originals are gone.

-Allen

Reply

Answer 5

David Strait Author

Level 2

342 points

Sep 29, 2011 4:01 PM in response to shuttersp33d

Shutters,

I will try the "Locate Referenced Files" command when I get home. I am still hopeful they wil show up!

I had figured out the trick to drag the previews to the desktop and I made a second project using those jpgs as the masters (still preserving the original project in case I locate the masters). As Allen Wicks surmised, I have set my previews at the original size, so I have a good set of replacement masters that have already been adjusted. So no big loss, unless I find many other projects with the same problem. I will look for any other missing masters as I relocate each project as move to a referenced library system.

Thanks,

Dave

Reply

Answer 6

Sep 29, 2011 4:07 PM in response to David Strait

Unless you need to do this Project by Project, go to Photos view and filter using the Rule "File Status" set to "Missing". That should group all of the Images in your Library for which Aperture cannot find a Master.

Like you, I'd be _very_ wary of your database, as it seems to be both corrupt and not repairable. (Sorry, I got no further suggestions except to move everything into a new Library.)

Message was edited by: Kirby Krieger -- reworded for clarity.

Reply

Answer 7

David Strait Author

Level 2

342 points

Sep 29, 2011 4:13 PM in response to Kirby Krieger

Wow! This is a great suggestion. I was unaware of this filter!

I will try this when I get home and find out how much damage has been done. hopefully it is only the single project.

Thanks,

Dave

Reply

Answer 8

David Strait Author

Level 2

342 points

Sep 30, 2011 11:14 AM in response to Kirby Krieger

Thanks to all the great advice I received here and a little luck, I am back up and running with all my masters. I am also migrating toward a new referenced library. The missing masters were never found in my working library, but I looked on my old laptop that I used when I took the original shots. By a stroke of luck I had not deleted the original raw files (from 2 years ago) when I first imported them from the camera. Here is a summary of what I learned in case others have similar issues:

It appears the project with the missing masters was somehow corrupted and the most of the masters from this project were no longer in the library. Spotlight could not find the masters in the vault or library. The project was a listed as a "managed" project and therefore the "Locate Referenced Files" command would yield "No Referenced Files." This also made it impossible to simply "Reconnect" the versions with the masters once I found a copy of the originals on the old laptop. I had to re-import the masters to the project, then used "Lift" and "Stamp" to copy all the adjustments and metadata from the old versions without masters to the new copy of the masters (to create identical new versions), then delete the old versions (without masters). Also note that the masters were obviously in the library for quite a while after importing them because I didn't edit these photos until about six months later (I wonder if the corruption occurred when I installed the new version 3.1.3 from the app Store? I have had the same managed library since version 1.0).

I am convinced the missing masters is a result of library corruption and not user error since (I think) it is actually impossible to delete a master from a managed library without deleting the versions (please correct me if I am wrong). I also looked to see if there was a "recovered" folder or project (there wasn't), so this type of corruption did not allow Aperture to recover the unlinked files. I also filtered my entire library using the "file status" set to "missing" and found that the only missing masters were the ~150 pics in the project I had been concerned about and the corruption did not extend to other projects (whewww!).

The bad news is that it appears that a corrupt library can actually cause the loss of the original image files. I will now move to a referenced library where I can back up my original images seperately from the library, then create smaller librarys that are subsets of my large library as a backup.

I welcome any other thoughts on this.

Dave

Reply

Answer 9

Sep 30, 2011 12:28 PM in response to David Strait

David Strait wrote:

...it appears that a corrupt library can actually cause the loss of the original image files.

Absolutely that is the case with a Managed-Masters Library.

I will now move to a referenced library where I can back up my original images seperately from the library

Note that proper backup of originals on to external drives prior to import into Aperture or any other images app has nothing to do with which Library configuration is chosen. Backup of originals should occur before any involvement with Aperture or any other images app.

...then create smaller librarys that are subsets of my large library as a backup.

I welcome any other thoughts on this.

Multiple Libraries is almost always a bad idea. Once you switch to a Referenced-Masters Library the Library will be small enough that multiple subsets will not be necessary.

-Allen

Reply

Answer 10

Sep 30, 2011 12:49 PM in response to David Strait

David Strait wrote:

The bad news is that it appears that a corrupt library can actually cause the loss of the original image files. I will now move to a referenced library where I can back up my original images seperately from the library, then create smaller librarys that are subsets of my large library as a backup.

I welcome any other thoughts on this.

Dave

Of course just as a bad HD will wipe out your masters in a referenced library setup. This shouldn't come as a surprise to anyone that has been using computers for any length of time.

Vaults (or an other backup of your library,TM, Carbon Copy Cloner) should have made restoring from your original problem a no brainier. Unfortunately from your description here and in your other thread you seem to have been hit with a 'perfect storm' of failures. Your main library and two vaults all become corrupted at the same time. Which just lends more weight to the old adage 'You can never have to many backups'.

I would strongly encourge you to postmortem this occurance to try and figure out what happened. If you could lose what, 3 disks, this time it could happen again. and it won't matter where the masters are located if the disk goes they go.

good luck

Reply

Answer 11

Sep 30, 2011 1:45 PM in response to Frank Caggiano

Frank Caggiano wrote:
If you could lose what, 3 disks, this time it could happen again. and it won't matter where the masters are located if the disk goes they go.

Not really true. Dave did not lose 3 disks concurrently.

With managed-Masters one can back up a broken Library to a dozen perfect drives and all that results is a dozen identically broken Libraries.

If the Masters are backed up prior to Aperture the problem resolves (versions still lost).

If (backed up) Referenced Masters are used and the Library becomes corrupt the problem resolves (versions still lost).

Proper use of Vaults also save Versions, but only prior to the time that the Library became corrupted - and we do not necessarily know the instant a Library becomes corrupt.

-Allen

Reply

Answer 12

Sep 30, 2011 3:06 PM in response to SierraDragon

From the OP's other thread:

I have three vaults (on three HDs) backing up one Aperture library (two onsite and one for offsite backup). Two of the vaults simultaneously became corrupted and I could no longer back up to them.

That plus the live library seems like 3 to me.

You wrote:

With managed-Masters one can back up a broken Library to a dozen perfect drives and all that results is a dozen identically broken Libraries.

Which is why backing up without verify the backup is a meaningless exercise. This goes for any backup method, How do you know your TM or CCC backup or simple copy of your masters is good?

If the Masters are backed up prior to Aperture the problem resolves (versions still lost).

And this is true also for managed libraries.

If (backed up) Referenced Masters are used and the Library becomes corrupt the problem resolves (versions still lost).

Not sure what you are trying to say here.

Proper use of Vaults also save Versions, but only prior to the time that the Library became corrupted - and we do not necessarily know the instant a Library becomes corrupt.

Proper use of vaults also saves masters if the library is managed. Ans see above for why we verify backups.

Look I agree that the masters need to be backed up in other than the vault (or TM or CCC or any one place). I use the Aperture backup dialog in the input window:

But once the masteres are backed-up referenced masters has no inherent saftey advantage over managed masters. In fact if all you do is backup before import and then place the masters on their own disk they are slightly less secure as they are only in two places.

In my operation I have my masters in three places, the import backup location the library and the vault I create from that library. In fact I have multiple vaults so in fact I have my masters in a minimum of 5 places. And its all fairly transparent.

So continue to sing the praise of referenced masters, for some and in some situations they are the way to go But from a security standpoint they have no advantage over managed masters.

Reply

Answer 13

Sep 30, 2011 4:50 PM in response to Frank Caggiano

Frank-

Frank Caggiano wrote:

From the OP's other thread:

I have three vaults (on three HDs) backing up one Aperture library (two onsite and one for offsite backup). Two of the vaults simultaneously became corrupted and I could no longer back up to them.

That plus the live library seems like 3 to me.

But he lost 3 Vaults/Libraries, not three disks as you said:

Frank Caggiano wrote:
If you could lose what, 3 disks, this time it could happen again. and it won't matter where the masters are located if the disk goes they go.

So it would matter where the Masters were located, because the disks did not fail, the Libraries/Vaults did. (Or at least that is the way I read it, I am not the OP)

Look I agree that the masters need to be backed up in other than the vault...
But once the masteres are backed-up referenced masters has no inherent saftey advantage over managed masters.

Agreed (emphasis above mine). As you know I believe that Masters should be backed up before Aperture becomes involved anyway.

And I think we both agree that either Managed or Referenced Libraries can work equally well (at least with small Libraries) and either can be safely backed up.

-Allen

Reply