How do I copy files from many devices to one device so end up with one version of all files?

"Consolidation" came from the name of the original program called consolidate17. I assume the author made 17 attempts at a solution. I got involved in revamping the program when the original user was continuing to report problems. Turns out the program was written to run on data on the startup disk. The Guy Burns, original user, was running the program on an external drive. The report was of disk space full. I optimized the program to use less disk space. Somewhere along the line it was determined that the original user was running the program on the external drive. Files were being copied from the external drive to the startup drive's trashcan. Startup drive was filling up. Since emptying the trashcan "solved" the problem it took awhile to start out. I modified the program to work on an external drive. I'm not sure if I gave the original author, JulieJulieJulie, enough credit. I did make extensive modifications over many weeks.

Here is part of the discussion about the origin of this program.

Can I copy files – but with certain restrictions?

These online services have delete the original uploads. 😢

R

So, you get a new big drive and copy all you files to the new drive. Detach all old drives from you machine. This is a safe copy of your files. You can then experiment around and see if the app does what you want.

Why doesn't Duplicate File Finder meet your needs?

Comparison is done by multiple parameters: File name, type, size, content, etc.

This isn't the why to go. You should be using a hash like md5 [?].

If you have bash coding skills, I did once write a bash script to consolidate files. Would need modification.

R

You can have name collisions. That is when you have two files with the same name in two different folders. Imagine what happens when you move them to the same folder.

I have code that does some of the consolidation. I'll look around for it.

R

I reviewed a script that I re-wrote 7 and a half years ago. Looks like it will do what you want. Here is the link:

https://pastebin.com/raw/u5UpRa8G

It does the duplicate elimination. If the duplicates have been eliminated. I'd expect it to do the combining since no duplicate were found. You never know.

Warning: please have a backup of your complete system. Please detach any external drives you do not need.

All the files to be "consolidated" need to be in the same folder, sub-folders are allowed. Place all files on the same volume. Consolidate from there.

The scripts was developed for one user. This use successful ran the script.

Here is what I think is the users comments on running the script.

CONSOLIDATE SCRIPT

Authors

Originally written in 2010 by JulieJulieJulie of the Apple forums in response to a post by Guy Burns. Rccharles made improvements to the script to bring it to its present form.

What the Script Does

Consolidate checks all files in a folder (top level and all subfolders) and sends duplicates in subfolders to the Trash. A file in a subfolder that has the same name as a file at the top level but which is not identical (i.e. it has been edited in some way) is appended with "#1", "#2" as required. Folders that have a dot (.) in their name are untouched by this script. Therefore, if you want to protect a folder put a dot somewhere in its name.

To run the script

1. Double-click on the script icon Consolidate.command. Terminal will open.

2. Drag the folder that is to be consolidated to the Terminal window and press return. The script offers the user two options.

(a) To keep the folder structure intact press "n" (do not move files). Pressing any other key will move all files to the top level at the end of the run.

(b) To keep map files intact press "n" (do not regroup). Pressing any other key will cause map files to be regrouped. Unless you have map files (GIS, QGIS and the like) that you wish to keep together, you can ignore this option and press any key.

The script searches for duplicate files and files with the same name that have been altered, and processes them as described above. At the end of the run the script reports (amongst other things) the elapsed time and the "trash diff" – the number of files trashed. It also creates a folder at the top level called saved--consolidated-data in which are four files that record what the script did. The first file, Consolidate Trashed List, tells the user how many files were trashed and their names.

Guy B.

Aug 2010

I uploaded an example run. Here:

https://pastebin.com/zVWRDA9E

[Edited by Host]