1 2 Previous Next 23 Replies Latest reply: Oct 7, 2011 7:12 AM by Ernie Stamper
DLScreative Level 4 Level 4 (1,595 points)

Let me preface this by saying that I'm not one of those users that cries "Aperture deleted my images", and my first thought is always user error when I see one of those posts. However, I've had two recent experieces that make me question that position. This is one of them:

 

I wanted to copy my main library to a new drive, but it was going slowly so I decided to take a different route: I updated my vault, created a new library then restored the new library from the vault.  This is one of those rare moments when one is restoring from a vault with the original library still intact.

 

 

Everything went as planned, and as expected, until I compared the two libraries. There was a 31.16 GB discrepancy so I started digging and found that the new library had 351 fewer images than the original.

 

 

Images in the trash? no

Images offline? no

Closed/opened stacks? no

Deleted images saved out out of vault? Only accounts for about 5 GB

 

 

My Math:

Library restored from vault: 91,102 photos @ 648.21 GB

 

 

Original library 91,453 photos @  679.37 GB

 

 

351 missing 31.16 GB Difference

 

 

There are images from four different cameras ranging from 6MP to 12MP so if i average that to 9MB each I get:

351x 9MB = 31.59 GB

 

 

My conclusion: there actually are 351 missing images!

 

 

I can't think of a filtering scheme that can tell me which 351, and I can't account for this.  Can anyone else?

 

 

DLS

  • 1. Re: Unexpected Vault Behavior- missing images
    Kirby Krieger Level 6 Level 6 (11,915 points)

    First -- only because your write-up is inclusive and this wasn't there -- check to see if those are rejected Images (i.e.: rated "x" or its odd keyboard equivalent, "9").

     

    Also make sure Aperture has finished processing.  You never know. 

     

    My next suggestion is too late: always run "Repair Library" prior to moving a Library.

     

    That makes me think that maybe running "Repair" on your new, diminished Library might have some effect.

     

    Fwiw, my experience with moving very large Libraries (>100,000 Images) is that a handful of Masters are always lost.  This has happened whether I moved the Masters separately and then located them (i.e.: reconnected them) or used "File→Relocated Masters".  "Handful" = <15.  I put this down to low-level drive errors (I have not scanned the drives by sector). 

     

    I have never lost a Version, however.  (I don't use Vaults.)

     

    Comparing Masters between two Libraries is inelegant, but not difficult.  Comparing Versions is probably the same.  Just for fun you could put a tag on all the Versions in the smaller Library, then merge it back into the original Library and then filter the resulting Library for "Does not have tag".

     

    Keep us posted.  Good Luck.

  • 2. Re: Unexpected Vault Behavior- missing images
    Frank Caggiano Level 7 Level 7 (23,795 points)

    The 31.16 GB discrepancy is in the sizes of the two .aplibrary folders? And you see the difference in image counts by looking at the count in the Photos view or by looking at the toatal number of images listed in the Library Choice window? And are all the images in the library  managed?

     

    Need to give this more thought but for now is there anything in Images Removed from VaulttotalNamed folder (located in the folder where the vault is).

     

    One other thing to try as you still have the original library is to create a new vault from it and compare that vault to the one you just updated.

     

    Also the standard questions, the drive you are vaulting to and restoring to are formatted OS X Extended and are local?

  • 3. Re: Unexpected Vault Behavior- missing images
    DLScreative Level 4 Level 4 (1,595 points)

    My math was whack: 351 images couldn't possibly equal 31.16 GB!  It adds up to 3.116 GB! 

     

    I'm just not thinking straight: I had a fire in my studio a few weeks ago, and dealing with the fallout has eclipsed everything else.

     

    Back to the drawing board. 

     

    Thanks

    DLS

  • 4. Re: Unexpected Vault Behavior- missing images
    DLScreative Level 4 Level 4 (1,595 points)

    Frank Caggiano wrote:

     

    The 31.16 GB discrepancy is in the sizes of the two .aplibrary folders? And you see the difference in image counts by looking at the count in the Photos view or by looking at the toatal number of images listed in the Library Choice window? And are all the images in the library  managed?

     

    Yes the discrepancy is in the sizes of the two .aplibrary folders. I'm  looking at the count in the Photos view.  It's a mixed (about 50/50) managed/ referenced library.  I did check to see if offline images were the culprit. No dice.

     

    Need to give this more thought but for now is there anything in Images Removed from VaulttotalNamed folder (located in the folder where the vault is).

    I created a new library, and imported that folder.  There were 503 images.  However, by my thinking, those deleted  images  would not affect the count in the original library because they had been removed.

     

     

    One other thing to try as you still have the original library is to create a new vault from it and compare that vault to the one you just updated.

    No time to do that or what Kirby suggests on a library this large.

     

    Also the standard questions, the drive you are vaulting to and restoring to are formatted OS X Extended and are local?

    The vault is on an external USB dive properly formated.

     

    As disconcerting as this is, I just didn't have time to trace it down.  In the end, copied the original library over to the new drive, and it matches.

     

    Perhaps Allen Wicks is right to say that we should not be letting managed libraries get this big, and I should make it all referenced.

     

    DLS

  • 5. Re: Unexpected Vault Behavior- missing images
    Frank Caggiano Level 7 Level 7 (23,795 points)

    DLScreative wrote:

     

    Frank Caggiano wrote:

     

    The 31.16 GB discrepancy is in the sizes of the two .aplibrary folders? And you see the difference in image counts by looking at the count in the Photos view or by looking at the toatal number of images listed in the Library Choice window? And are all the images in the library  managed?

     

    Yes the discrepancy is in the sizes of the two .aplibrary folders. I'm  looking at the count in the Photos view.  It's a mixed (about 50/50) managed/ referenced library.  I did check to see if offline images were the culprit. No dice.

    Even though you have moved on from this and appear to not have the time to go any further troubleshooting the problem I do have one further question.

     

    In my original reply I asked how you were coming up with the image count in the library. You indicated  it was from the count in the Photos view. I would also be curious to see how that count compares to the count given in the Library choice window. If you find the tie could you post that.

     

    regards

     

    Ps And as far as the suggestion that larger databases are somehow less robust then small databases, that opinion has no basis in fact. There are  lot of reasons to choose a managed or a referenced workflow. Library size isn't one of them.

  • 6. Re: Unexpected Vault Behavior- missing images
    Ernie Stamper Level 8 Level 8 (37,475 points)

    I never use Vault, and thus cannot answer a question that comes to mind about new masters that are created whenever you choose to edit a photo in an externall editor.  Are those also backed up via Vault?

     

    Ernie

  • 7. Re: Unexpected Vault Behavior- missing images
    Kirby Krieger Level 6 Level 6 (11,915 points)

    Hi Ernie -- The answer is "Yes".  Afaik, they are always created as Managed Masters, and as Managed Masters are automatically included in any newly-created Vault or added to any existing Vault when that Vault is updated.

     

    DLSCreative was comparing a Library restored from a just-completed Vault to the original Library.  Since no changes were made to the Library, the two should be identical, regardless of the file status of the Image's Masters.

     

    The one thing I can think of is that creating and restoring from a Vault could give Aperture a chance to recover wasted disk space inside the Library.  I don't know any way to check this -- it would be good to know.  But 31 GB seems to be _way_ too much space to recover through database compaction.

  • 8. Re: Unexpected Vault Behavior- missing images
    DLScreative Level 4 Level 4 (1,595 points)

    Frank, I'm not sure what you mean by "Library choice window". 

     

    Allen's argument is not Aperture specific but about databases in general.  I don't know if he is correct because Aperture is the only database app that I know anything about, but I've had a couple of inexplicable events of late that shake my trust in keeping my image files in a database.

     

    Ernie,  The vault is supposed to be an exact mirror of the library. All managed files are supposed to be included,  and I have far more than 341 images that have gone to photoshop.

     

    DLS

  • 9. Re: Unexpected Vault Behavior- missing images
    Ernie Stamper Level 8 Level 8 (37,475 points)

    I think he means when you hold down the Option Key while launching Aperture, and highlight each of the available libraries.  A count of versions and Masters is presented with each.

     

    Ernie

  • 10. Re: Unexpected Vault Behavior- missing images
    Frank Caggiano Level 7 Level 7 (23,795 points)

    The window you get when you select Switch to other Library:

    Screen shot 2011-10-04 at 10-04 12.48.49.png

    At the bottom where it shows the complete path for the selected library it also shows the number of masters and versions in that library.

     

    Database's in general are one of the most reliable pieces of software out there. They have been around since allmost the beginning and have been extensivly studied and refined. One of the problems IMO is that then Aperture library isn't a database, its a mishmash of database files and flat files which actually make the code more complex then it needs to be. Hopefully at some future point the Aperture library will become a true database.

     

    Until then we have to work with what we have and adressing the problems we encounter. We all have our own particular vodoo we use to try and cope with the isues. Just bear that in mind when recommendations are made here.

     

    regards

  • 11. Re: Unexpected Vault Behavior- missing images
    DLScreative Level 4 Level 4 (1,595 points)

    I thought that's what you meant, but I guess I never noticed the image count at the bottom.  This info just makes it all stranger:

     

    The original library has 91463 versions and 92290 masters (If every master has at least one version, how can there be more masters than versions?)

     

    The restored library has 91113 versions and 91087 masters

     

    DLS

  • 12. Re: Unexpected Vault Behavior- missing images
    DLScreative Level 4 Level 4 (1,595 points)

    Since I now have two copies of the original library and need to create a new vault anyway, I'm going to try

    Frank's idea and  kill the old vault, create a new one and restore another new library from that vault to see what I get. 

     

    I really should dig a little more while I have the oportunitty. Results tomorrow.

     

    DLS

  • 13. Re: Unexpected Vault Behavior- missing images
    Ernie Stamper Level 8 Level 8 (37,475 points)

    I have one library where there 108 versions and 176 Masters.  Although this was a test of a Referenced Library, I can also confirm that the difference is due to there being a pair of RAW + JPEG.  The reason the Master count was not double was that there were JPEG alternatives created in Photoshop in the folder as well.

     

    In the Browser view of the Project there is only one version counted for the pairs.  That version can be changed with the Command in the drop down under Photo to be set either from the JPEG Master or RAW Master without changing the count.

     

    Do you have some pairs in this database?

     

    Ernie

  • 14. Re: Unexpected Vault Behavior- missing images
    DLScreative Level 4 Level 4 (1,595 points)

    I never shoot pairs, and I don't think I ever have. I may have put some in there at some point just to test the command you speak of, and I do have a vague recollection of that. (This library goes back to 2005.)  I can't look for them now because I'm creating a new vault.

     

    DLS

1 2 Previous Next