PaulCommentary
Level 1 (49 points)

Q: Any way to avoid yet another big, long, upload?

I use Photos and have iCloud Photo Library enabled.  Using the latest version of El Capitan.

 

For about the seventh time since in the last three months, Photos decided to "repair" my library.  Question 1: Why does the library break so often? My Aperture library went years without needing a repair. No other files on my machine are being corrupted. The SSD works just fine.

 

After the repair, which took several hours, Photos has decided to upload ALL 46000 of my pictures YET again. Even on a 1300 Mbps connection, the upload rate appears to be about 100-1000 pictures per hour. So I don't think it's comparing the photos on my Mac with the photo in the cloud. Because that would be a really slow compare. It's re-uploading them!  I'm looking at perhaps 46 hours. And, as before, the impact on other use of the internet is significant and affects all computers in the house. Question 2: Why is Photos designed in such an unrealistic fashion? Many of us have bandwidth caps. It seems like brute force is the wrong approach here. Question 3: Has Apple ever indicated they realize the problem exists?

 

Question 4: Given that this is occurring and will occur and was not addressed in the new version of Photos in El Capitan, are there ways to get better diagnostics other than the very limited amount of information that Photos gives: "Repairing'", "Uploading" (see the pictures).  If not, shouldn't there be?  My trust in the integrity of my photos and associated metadata is very low. And it's the most important information on my computer.

 

 

 

 

Repairing.jpgUploading.jpg

Posted on Oct 14, 2015 7:50 AM

Close
PaulCommentary

Q: Any way to avoid yet another big, long, upload?

  • All replies
  • Helpful answers

  • by PaulCommentary,

    PaulCommentary PaulCommentary Oct 14, 2015 8:14 AM in response to PaulCommentary
    Level 1 (49 points)
    Oct 14, 2015 8:14 AM in response to PaulCommentary

    I'll add some more information. After the library is rebuilt, Time Machine takes it as a whole new file (perhaps in bands) and decides to backup the 100 GB Photos Library over wireless. At the same time iCloud Photo Library is "uploading". 

     

    Additionally, Backblaze does the same. And my internet performance goes down and my move towards bandwidth cap goes up.

     

    I do realize I could turn these off.

     

    But back to the fundamental questions: how can a library need repair so often and why are the real-world consequences of that repair so severe?

  • by léonie,

    léonie léonie Oct 14, 2015 8:39 AM in response to PaulCommentary
    Level 10 (106,958 points)
    iCloud
    Oct 14, 2015 8:39 AM in response to PaulCommentary
    Even on a 1300 Mbps connection, the upload rate appears to be about 100-1000 pictures per hour. So I don't think it's comparing the photos on my Mac with the photo in the cloud. Because that would be a really slow compare.

    The next time it happens, open a Console window to check, what Photos is doing. After a repair I am mainly seeing comparisons and duplicate checks:

    Screen Shot 2015-10-13 at 17.18.56GMT.png

    It's re-uploading them!  I'm looking at perhaps 46 hours.

    I have repaired my Photos Library two times - a library with 40000 photos.  It took two days to finish the comparison, but it Photos was really only comparing, as I could see in the Console.

     

    A Repair in Photos is not the same as in Aperture. Photos has only one Repair mode, while Aperture had several different modes, from repair to rebuild. Probably a Repair in Photos can be equal to a rebuild in Aperture.

     

    But back to the fundamental questions: how can a library need repair so often and why are the real-world consequences of that repair so severe?

    Where is your Photos Library stored? Are you accessing it from more than one user account?

    Are you using any tools that might be modifying the library, for example any cleaning applications or duplicate finders ?save storage

  • by PaulCommentary,

    PaulCommentary PaulCommentary Oct 14, 2015 6:33 PM in response to PaulCommentary
    Level 1 (49 points)
    Oct 14, 2015 6:33 PM in response to PaulCommentary

    Just an update for those keeping score. Home from work after an 11 hour day. There are still 38608 photos to upload. Others say that only a comparison with the cloud is going on, maybe....  But this is a new release of the software, don't you think even that small correction in the dialog box could have been made?

    Uploading 2.jpg

  • by PaulCommentary,

    PaulCommentary PaulCommentary Oct 15, 2015 7:41 AM in response to PaulCommentary
    Level 1 (49 points)
    Oct 15, 2015 7:41 AM in response to PaulCommentary

    Down to 35682 overnight. So that's a 1000 in 10 hours.

     

    Saw these (example) entires in console log:

     

    10/14/15 9:48:58.000 PM kernel[0]: Sandbox: SpotlightNetHelp(456) deny(1) file-read-data /Applications/Photo Booth.app/Contents

    10/14/15 9:48:58.000 PM kernel[0]: Sandbox: SpotlightNetHelp(456) deny(1) file-read-data /Applications/Photo Booth.app

    10/14/15 9:48:58.000 PM kernel[0]: Sandbox: SpotlightNetHelp(456) deny(1) file-read-data /Applications/Photo Booth.app/Contents/MacOS/Photo Booth

    10/14/15 9:48:58.000 PM kernel[0]: Sandbox: SpotlightNetHelp(456) deny(1) file-read-data /Applications/Photos.app

    10/14/15 9:48:58.000 PM kernel[0]: Sandbox: SpotlightNetHelp(456) deny(1) file-read-data /Applications/Photos.app/Contents

    10/14/15 9:48:58.000 PM kernel[0]: Sandbox: SpotlightNetHelp(456) deny(1) file-read-data /Applications/Photos.app/Contents/PkgInfo

    10/14/15 9:48:58.000 PM kernel[0]: Sandbox: SpotlightNetHelp(456) deny(1) file-read-data /Applications/Photos.app/Contents/MacOS/Photos

    10/14/15 9:48:58.000 PM kernel[0]: Sandbox: SpotlightNetHelp(456) deny(1) file-read-data /Applications/Photos.app/Contents/MacOS/Photos/..namedfork/rsrc

     

    10/14/15 11:12:45.009 PM cloudphotosd[296]: Tried to cleanup temporary file file:///Users/xxx/Pictures/Photos%20Library.photoslibrary/private/com.apple.clo udphotosd/CloudSync.noindex/GeneratedImageDerivatives/CBFA66E2-B4C8-4C23-8548-BE 56E1FF160A_CS_Medium.jpg, but failed. error: Error Domain=CSLibraryManagerErrorDomain Code=12 "Not contained within temporary directory mask 2" UserInfo={NSLocalizedDescription=Not contained within temporary directory mask 2}

     

    10/15/15 6:43:53.139 AM cloudd[359]: Progress of MMCS item 99776: s:7 p:-1.00 error:Error Domain=com.apple.mmcs Code=7 "Put container(s) failed." UserInfo={NSDescription=Put container(s) failed., NSLocalizedDescription=Put container(s) failed., NSUnderlyingError=0x7fa13bc67e30 {Error Domain=com.apple.mmcs Code=5 "HTTP request failed without status!" UserInfo={NSDescription=HTTP request failed without status!, NSLocalizedDescription=HTTP request failed without status!, NSUnderlyingError=0x7fa1394951b0 {Error Domain=com.apple.mmcs Code=38 "NSURLSession error: A server with the specified hostname could not be found. ((null))" UserInfo={NSDescription=NSURLSession error: A server with the specified hostname could not be found. ((null)), NSLocalizedDescription=NSURLSession error: A server with the specified hostname could not be found. ((null)), NSUnderlyingError=0x7fa1397c82d0 {Error Domain=NSURLErrorDomain Code=-1003 "A server with the specified hostname could not be found." UserInfo=0x7fa1397a2630 (not displayed)}}}}}}

     

    10/15/15 7:33:48.620 AM cloudphotosd[296]: Skipping operation <CSLibraryManagerFetchMissingResourcesOperation: 0x7f81aa488610>, library manager operation duplicate to another operation

     

    10/15/15 7:33:48.645 AM cloudd[359]: Progress of MMCS item 99926: s:7 p:-1.00 error:Error Domain=com.apple.mmcs Code=4 "Request was cancelled." UserInfo={NSDescription=Request was cancelled., NSLocalizedDescription=Request was cancelled.}

     

    Have rebooted and all that. Several times.

  • by PaulCommentary,

    PaulCommentary PaulCommentary Oct 15, 2015 6:28 PM in response to PaulCommentary
    Level 1 (49 points)
    Oct 15, 2015 6:28 PM in response to PaulCommentary

    An update. Home again after 11 hours.  .

     

    19311 left to upload. So the rate has improved. But still....


    And websites would not come up until I paused the iCloud Photo Library upload. It impacts access that much.

     

     

    Screen Shot 2015-10-15 at 6.22.50 PM.jpg

  • by bhyahoo,

    bhyahoo bhyahoo Mar 14, 2016 7:48 PM in response to PaulCommentary
    Level 1 (4 points)
    Mar 14, 2016 7:48 PM in response to PaulCommentary

    I'm now having a similar issue but the upload never finishes and instead it turns off iCloud Photo Library with no dialog.

     

    DId your issue ever get resolved?

  • by léonie,

    léonie léonie Mar 15, 2016 12:38 AM in response to bhyahoo
    Level 10 (106,958 points)
    iCloud
    Mar 15, 2016 12:38 AM in response to bhyahoo

    Apple confirmed meanwhile in a support document ( Fix missing or incorrect thumbnails in Photos - Apple Support  )  that repairing a Photos Library will result in a new upload of the library.   Photos is doing the comparison of the library to the files in iCloud by uploading it again!

     

    Note: If you use iCloud Photo Library with Photos, this action will cause Photos to re-upload all your pictures.

    The only way to avoid this constant upload is to avoid the necessity to repair the library.

     

    • Avoid crashes.
    • Avoid moving the library to a different drive.
    • Avoid modifying the library using the Finder or accessing it by other applications.
    • Avoid permission issues - don't open the library from a different user account.
    • Avoid storing the library on a network volume or in a synced folder or on a drive with an incompatible file system.
    • Avoid restoring the library from a backup.

     

    If Photos crashes, there is no help for it. The library needs to be repaired, and if the drive dies, will have to restore it.

     

    Signing out of iCloud Photo Library and signing back in will also cause the new upload.

  • by LarryHN,

    LarryHN LarryHN Mar 15, 2016 9:11 AM in response to léonie
    Level 10 (84,646 points)
    Photos for Mac
    Mar 15, 2016 9:11 AM in response to léonie

    However the comparison is much quicker - my original upload took ten days for 7000 photos - the last comparison when I have to repair my library took less than 1 day for 17,000 photos

     

    LN

  • by léonie,

    léonie léonie Mar 15, 2016 11:19 AM in response to LarryHN
    Level 10 (106,958 points)
    iCloud
    Mar 15, 2016 11:19 AM in response to LarryHN
    However the comparison is much quicker - my original upload took ten days for 7000 photos - the last comparison when I have to repair my library took less than 1 day for 17,000 photos

    I'm no longer sure what to expect.  After the first repair of my library I thought the same. 39000 photos uploaded in less than a day. The  next time I repaired the library because faces thumbnails were blank, the upload took six days.

    And last week, after I signed out of iCloud Photo Library, to test deleting from My Photo Stream, the upload upload took again six days.

     

    I will no longer recommend repairing the library, unless I am sure the library in question is not an iCloud Photo Library or all other options have been exhausted.

  • by PaulCommentary,

    PaulCommentary PaulCommentary Mar 20, 2016 10:26 AM in response to léonie
    Level 1 (49 points)
    Mar 20, 2016 10:26 AM in response to léonie

    Thanks for confirming the observations I've been making since Photos was launched.

     

    I went to the link you posted as below.

     

     

    But in reading it, I don't see the Apple confirmations that you mentioned. Can you help clarify?

     

    This would be very important to know for all users of Photos.

  • by PaulCommentary,

    PaulCommentary PaulCommentary Mar 20, 2016 10:33 AM in response to bhyahoo
    Level 1 (49 points)
    Mar 20, 2016 10:33 AM in response to bhyahoo

    No. I don't believe the issue has been resolved for any users. It's just a matter of time until even the most savvy users encounter the massive re-upload problems.

     

    It's unfortunate that Apple has even informed users of the behavior nor improved Photos in this regard in almost a year. Users need to have a conceptual model of software behavior that they can trust so that they can best plan how to preserve and manage their data in a predictable way. Unfortunately the current state is not that.

  • by léonie,

    léonie léonie Mar 22, 2016 2:48 PM in response to PaulCommentary
    Level 10 (106,958 points)
    iCloud
    Mar 22, 2016 2:48 PM in response to PaulCommentary

    But in reading it, I don't see the Apple confirmations that you mentioned. Can you help clarify?

     

     

    I meant the last paragraph in ( Fix missing or incorrect thumbnails in Photos - Apple Support  ) . It is a warning, that the Photos Library Repair tool will cause Photos to re-upload all the photos.

    .........., you can regenerate them with the Photos library repair tool:

    1. Quit Photos.
    2. Open Photos while pressing the Command and Option keys.
    3. Click Repair in the dialog that appears.

    ......

    Note: If you use iCloud Photo Library with Photos, this action will cause Photos to re-upload all your pictures.

  • by bhyahoo,

    bhyahoo bhyahoo Mar 29, 2016 8:15 AM in response to léonie
    Level 1 (4 points)
    Mar 29, 2016 8:15 AM in response to léonie

    I'm assuming it weeds out duplicates?

     

    My problem isn't so much the entire upload again, but that the upload keeps resetting and then after so many resets, icloud photo library turns off with no warning or dialog. I only find out the reason by digging in the logs.

     

    Since this last OSX update a few days ago, it's stopped turning itself off, but keeps doing a reset sync and now after a week I wonder if my library will ever get reuploaded and finished.

  • by léonie,

    léonie léonie Mar 29, 2016 4:52 PM in response to bhyahoo
    Level 10 (106,958 points)
    iCloud
    Mar 29, 2016 4:52 PM in response to bhyahoo

    I'm assuming it weeds out duplicates?

    That is what the Console messages suggest, and what I observed. When I have two identical photos with different filenames on two macs, the merging will replace the original on one Mac by the original from the other Mac with the different name.  This way the filenames of originals can change between Macs , and also file modification dates.