Find and remove duplicated files and folders

What I've done: copy all files from all folders into a temporary folder letting replacements happen. Then sort by name and highlight delete duplicates if any. Then I use that cleansed set of files to replace all files in the various folders

I have had good results in the past with DupeGuru

https://dupeguru.voltaicideas.net/

And the price is right. But - and this is important - do not use it on a Music, AppleTV or Photos Library. These are specialised items and can contain apparent duplicates. Using an app like this on them could cause dataloss.

From where exactly? If used on the wrong libraries these kinds of app can cause a lot of damage.

Attempted to download and received this response: “dupeguru.app” can’t be opened because Apple cannot check it for malicious software.

This software needs to be updated. Contact the developer for more information.

I checked the App Store. the answer was No Results.

I hope this helps.

Don_What wrote:

Attempted to download and received this response: “dupeguru.app” can’t be opened because Apple cannot check it for malicious software.

Dodged a bullet on that one. I took one look at the web site and knew it was junk. Consider the following, "On OS X, the UI layer is written in Objective-C and uses Cocoa. On Linux & Windows, it’s written in Python and uses Qt5." This is false. It's a Qt app. And like virtually all Qt apps, is not properly signed and notarized. I've seen this many times on the developer forums, but not for a long time. I figured all the Qt folks just gave up. But at least one is still trying, and failing.

But regardless, for some years now, Apple filesystems have automatically prevented the creation of duplicate files. All file storage is optimally efficient. If you take a 10 GB file and duplicate it 3 times, that will take up 10 GB. If you take a 10 GB file, and change 1 GB of data, that will take up 11 GB.

Technically, it is still possible to duplicate a file at a low level. But ironically enough, probably only some poorly-made Qt app would actually do that. Attempted to find these kinds of duplicate files would be long and time consuming. You would have to read every single byte on the hard drive. If you do this regularly, it could prematurely age the SSD.

The correct answer here is - do nothing. The operating system already does this.

Well that's educational and thanks for that. But that about the case where - say - someone has a jpeg file and imports a copy of it from somewhere else? Will the OS simply recognise it as another iteration of the same file?

Yer_Man wrote:

Well that's educational and thanks for that. But that about the case where - say - someone has a jpeg file and imports a copy of it from somewhere else? Will the OS simply recognise it as another iteration of the same file?

That's different. There is no way easily check external files for duplicates. You would have to scan the entire disk. At least that could be optimized somewhat and only search for identically sized files. But still, that's a lot for a single file.

However, image files are special. Chances are, they are already in a smaller database. Plus, there are more sophisticated tools and algorithms to check for images that are identical or very similar. An JPEG files are tiny. That's why there are lots of duplicate image finders.