Duplicate photos in Photos app

Are you perhaps taking the photos on the iPhone in the High efficiency format HEIF, but syncing them to your Mac in a more compatible format?

The conversion to JPEG when downloading from the iPhone is disabling the duplicate detection. The JPEG on the Mac will not be a duplicate of HEIC file on the iPhone and always appear as a new photo. If you are shooting on the iPhone as HEIC, do not use the conversion on Import - see this document for the Settings: Using HEIF or HEVC media on Apple devices - Apple Support

Another common reason for duplicates is using My Photo Stream and iCloud Photos or manual import together. To avoid duplicates use only one method of importing - either My Photo Stream, or manual import, or iCloud Photos. Do not mix these methods. and avoid format conversions.

Photos on Big Sur is spending a lot of time to look for duplicates, but unfortunately is not showing them to us for removal. It is just omitting the duplicates from the "Photos > Days". view. You could use the Days as a starting point which photos to keep.

There are third-party apps to search for duplicates, but some are dangerous. It will depend on the kind of duplicates you are having on your Mac to pick an automatic duplicate Finder. 

I can recommend Power Photos and Photo Sweeper as safe to use, but both are not free.

• Power Photos can detect duplicates based on the file attributes like capture date or filename. As long as your duplicates are having correct dates or identical filenames, Power Photos will detect them.
• Photo Sweeper can compare the photos if they are look-alikes. It will detect edited versions of the same photo or similar photos, even if the dates or filenames or the pixel size differ.

There are many unsuitable duplicate detectors, that will damage the Photos Library and can cause you to lose your photos. By careful to use only a program that does not modify the library directly but only programs, that will use the program interface of Photos to remove the duplicates. Apple is warning against duplicate removal software: Using third-party apps to remove duplicate photos might damage your Photos for macOS library - Apple Support

I've spoken to Apple's Media Support Specialist group a few times. Apparently this is a known issue. I've been plagued with this for a while and unfortunately have been unable to find any pattern to the duplication pattern. Duplication started several months ago with no trigger event that I can determine. Photos that haven't been duplicated suddenly appear as new photos to be transferred.

You can see the rows of duplicates in this small extract.

Regardless of method being g used, how does one remove duplicate pictures from ones library? Other than manually going through them.. Your help is much appreciated. If not in the app itself, is there a way to do so outside of the app? I'm using Big Sur. Your help would be much appreciated.

Mac Keeper has not been designed to work with Photos Libraries. I would not risk to use it on a Photos Library, also not anything with "clean" in the name. On a Mac you do not need nay apps as a keeper. MacOS has many built-in safety features. The cleaning and housekeeping apps are just likely to interfere with the built-in tools and create more damage than they will do good.

We have almost 100,000 photos on our iCloud photos and probably 50 disks with photos. . .not to mention hundreds of videos of family events.

How are others organizing all these family photos and videos? Where do I start?

You should start your own thread.

And you begin a task like this like you eat an elephant, one bite at a time.

I can't help with iCloud, but in Photos I mainly use keywords and locations to organize.

Just go through each "group" [ party, day at Disneyland...] of photos and set keywords {Family, nature, museum, Disney...} and make sure the locations are good.

Actually Photos built in organization tools, like moments, dates, people and places, are really not that bad.

I have MacKeeper, any experience with that?

And duplicate zapper..

Thanks!

Now I'm repeating... "pattern"... oops.

I had this happen with my Olympus TG-5, and found they way to fix it was to reformat the card.