Can I copy files – but with certain restrictions?

mv "${filePath${j}}" ~/.Trash 2>/dev/null || mv "${filePath${j}}" ~/.Trash/$RANDOM

Says do the first mv and if that should fail do the second mv.

JulieJulieJulie has a good programming technique. I am learning more about bash programming by reading the code.

Any chance of making the failing dataset available on the web?

Are you running out of disk space? It isn't obvious to me why a non-empty trashcan would bother the script.

I'll look into making up some debug statements.

Robert

-----------
-----------
Maybe the trashcan is in the path of your large folder. let's say
the trash is at /user/mac/.trash
if you put your data at /mydata the trash would be in your path. Of course you would have other problems.

----------
----------


You could be running into some resource limit with the big folder size.

Would be nice if you try trimming the large folder & see if you can get the problem to persist.

You are using consolidated14, perhaps you should try 17. Never know, maybe the problem disappeared.

--------------------

Here are some debug statements.

Notice I placed the debug statements beginning with if & ending with fi after the problem message ::: Checking checksums

You will need to change the file name. Leave the "" around the name. you want to change what is in the double quotes. Notice I copied the file name as it was typed out in the terminal.

Try with a different name on a small sample to see if the code change works.

The // in the name is funnie in my opinion. Seems to work. One or more slashes seem ok.

Copy exactly.

Notice the scripts stops waiting for you to type a return. I you do not want the script to stop, get rid of the read line.

What would be good is to see several hundred / thousands lines of debug after the read.

((( You might want to get a run with debug turned on a few files before the bad file. )))

echo
echo "Checking checksum matches for: ${filePath[${i}]}"


if [ "${filePath[${i}]}" = "/Users/mac/Desktop/see//rebate status.png" ] ; then
echo
echo "------- debug 1 ----------"
echo
read -n1 -p "Press any key to begin debug... "

set -vx
fi


Here is the actual debug code without the read command.

if [ "${filePath[${i}]}" = "/Users/mac/Desktop/see//rebate status.png" ] ; then
echo
echo "------- debug 1 ----------"
echo


set -vx
fi




Checking checksum matches for: /Users/mac/Desktop/see//force quit copy 1.png

Checking checksum matches for: /Users/mac/Desktop/see//Picture 10.png

Checking checksum matches for: /Users/mac/Desktop/see//Disk Utility Partition copy 1.png

Checking checksum matches for: /Users/mac/Desktop/see//rebate status.png

------- debug 1 ----------

Press any key to begin debug...
+ dupecount=1
+ (( j=46 ))
+ (( j<69 ))
+ '[' -z '/Users/mac/Desktop/see//Carbon Copy Cloner.app' ']'
+ '[' b3aa3467487c7fd129f947a085b6f17b = ba5d59444e67dbe115e7b509af7937d7 ']'
+ break
+ (( i++ ))
+ (( i<69 ))
+ '[' -z '/Users/mac/Desktop/see//Carbon Copy Cloner.app' ']'
+ echo




--------

as for your duplicate names in .trash folder, perhaps there is a trailing blank or other non-printable characters.

Robert

Message was edited by: rccharles

For a different reason, I was told to use a "bash" file on Terminal. I did a search and found this post. I didn't read every word, but I skimmed slowly to see what was going on. From this I learned:
1. You are all smarter than me.
2. You are all more courageous than me.
3. I will hire an intern for the summer to manually do what I wanted the computer to do.
All the best to each and every one of you. You are capable of sending people to the moon. I will sell popcorn to spectators. 🙂

A few thank you's are in order -- to JJJ for his superb programming skills (I owe him a "solved" sticker);

I learned a lot from reading JJJ's script. The script is well written and in a consistent programming style.

and to Robert for his able assistance (I'll give him some "helpfuls").

Thanks. I'll be around this summer if your interesting in tracking down any of those problems. Post on a new thread. I like to solve problems & this has been a good way of learning more about Bash.

And if either of them are ever in North-West Tasmania, look me up. I'll shout you a beer or two.

Thanks. It'll be awhile at a minimum.

Robert

Well, I have a modified consolidate script.

copy the script text to the clipboard
highlight text...
command+c
Macintosh-HD -> Applications -> Utilities -> Terminal
pwd
pico
command+v
control+x
y
consolidate
chmod 755 consolidate
./consolidate

<pre style="
font-family: Monaco, 'Courier New', Courier, monospace;
font-size: 10px;
font-weight: normal;
margin: 0px;
padding: 5px;
border: 1px solid #000000;
width: 720px; height: 340px;
color: #000000;
background-color: #FFEE80;
overflow: auto;"
title="">#!/bin/bash

# Purpose of this script:
# Find folders within a specified top folder which are not bundles. Find files
# within each found folder (and not within subfolders because each folder is
# handled individually.)
# Find potential bundles and checksum their entire content as one item.
# Get MD5 checksums for each item and save an index of all checksums and
# filepaths. Sort the index by checksum. Compare each
# item's checksum and if a duplicate is found, move the duplicate to the trash.
# Compare each item's name and if a duplicate is found, append incrementing #X
# to the name prior to the last period (as in "Filename #2.jpg")
#
# Optionally move files and potential bundles to the top-level folder
# Delete .DS_Store, .FBCIndex*, and .FBCSemaphoreFile files and subsequently
# delete empty subfolders. Optionally regroup map files into folders based on
# the name of .frq file if found, or .shp file
#
# This script is a modification of JulieJulieJulie's Consolidate17 script. See:
# http://discussions.apple.com/message.jspa?messageID=11577992#11577992
#
# I have attempted to:
# reduce the size of Bash arrays,
# add more error checking, and
# improve the preformance of the script.
#
# rccharles
#
# Copyright 2010 rccharles
#
# GNU General Public License
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, version 3
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# For a copy of the GNU General Public License see
# <http://www.gnu.org/licenses/>.


# debug info
export PS4='+(${BASH_SOURCE}:${LINENO}):'

## not in the tiger version of bash ${FUNCNAME[0]:+${FUNCNAME[0]}(): }'


##########
### Functions (subroutines which can be called by name)
function Show_Time () {
echo " Elapsed time in seconds since start of processing: $(( $( date +%s) - ${Seconds_since_the_epoch_start} ))"
echo
}


function echoTrashInfo () {
echo " -${trashFileName}"
echo "${fileName} > ${trashFileName}" >>"${saveFolder}/Consolidate_Trashed_List_${runDateTime}.txt"
let trashCount+=1
}


echo
echo "$0 script revised $(GetFileInfo -m $0)"
echo

declare folder \
MainFolder \
MoveFiles="Question" \
RegroupMapFiles="Question" \
runDateTime=$(date '+%Y-%m-%d_%H.%M') \
Seconds_since_the_epoch_start


declare -x tab=$'\t' \
verbose="Yes" \
veryVerbose="No"


declare -i count=0 \
offsetMainFolder

declare -a filePath \
fileName \
fileSum \
folders

# If the utility md5 is not present then just quit, it's needed for the checksums.
[ ! -x /sbin/md5 ] && exit 2

# Check for command-line options used when calling the script
if [ $# -gt 0 ] ; then
while getopts "mMrRV" Option ; do
case "${Option}" in
m ) echo "-m argument used on command line, moving files is disabled."
echo
MoveFiles="No"
;;
M ) echo "Moving files. -M argument used on command line."
echo
MoveFiles="Yes"
;;
r ) echo "-r argument used on command line, regrouping map files is disabled."
echo
RegroupMapFiles="No"
;;
R ) echo "Regrouping map files. -R argument used on command line."
echo
RegroupMapFiles="Yes"
;;
V ) echo "Display very detailed messages. -V argument used on command line."
echo
veryVerbose="Yes"
;;
* ) echo "Unknown argument among arguments $* on command line."
exit 6
;;
esac
done
fi

# We're done with switches / options from the command line
shift $(($OPTIND - 1))

# Main folder can be declared on the command line or interactively
MainFolder="${1}"

# If the main folder doesn't exist, prompt once, then just quit if it's still not there.
if [ ! -d "${MainFolder}" ] ; then
echo "Folder not specified. Type the path to the top-level folder or drag"
echo "it into this Terminal window, then click back in this window and press return:"
read MainFolder
echo
fi

[ ! -d "${MainFolder}" ] && echo "Folder not found." && exit 4
[ "${MainFolder}" = "/" ] && echo "You must be crazy." && exit 5

echo "Holding down Control-c on the keyboard will stop this script in an emergency."
echo

if [ "${MoveFiles}" = "Question" ] ; then
echo "WARNING: allowing the script to move files could cause problems!"

echo
echo "Do you want to move all files and bundles to the top level at the end of the run?"
read -n1 -p "Press 'n' to NOT move files and keep the folder structure intact; otherwise press any key. "
if [ "${REPLY}" = "n" -o "${REPLY}" = "N" ] ; then
MoveFiles="No"
else
MoveFiles="Yes"
fi
echo


fi

if [ "${RegroupMapFiles}" = "Question" ] ; then
echo "Do you want to disable regrouping of map files into folders based on the .shp filename?"
read -n1 -p "Press 'n' to NOT regroup map files; otherwise press any key. "
if [ "${REPLY}" = "n" -o "${REPLY}" = "N" ] ; then
RegroupMapFiles="No"
Else
RegroupMapFiles="Yes"
fi
echo
fi

[ "${veryVerbose}" = "Yes" ] \
&& ulimit -a \
&& df \
&& echo

[ "${veryVerbose}" = "Yes" ] \
&& echo "MainFolder=${MainFolder};"

let offsetMainFolder="${#MainFolder}"-1

# Ensure that the path is in a consistant format. i.e. no //
# drop trailing directory separator if needed
[ "${MainFolder:${offsetMainFolder}}" = "/" ] \
&& MainFolder="${MainFolder:0:${offsetMainFolder}}"

# Place all created files in one directory.
saveFolder="${MainFolder}/saved--Consolidated--Data"

[ "${veryVerbose}" = "Yes" ] \
&& echo "MainFolder=${MainFolder};" \
&& echo "saveFolder=${saveFolder};"


[ ! -d "${saveFolder}" ] \
&& mkdir "${saveFolder}"

#Housekeeping for Index files.

# If the index archive file does not exist,
# create it so that it will be there for sorting and merging later
[ ! -e "${saveFolder}/Index_Archive.txt" ] \
&& touch "${saveFolder}/Index_Archive.txt"

# If the index file exists, archive it and be sure to delete the index.
if [ -e "${saveFolder}/Index.txt" ] ; then
echo "----------------------------" >> "${saveFolder}/Index_Archive.txt"
cat "${saveFolder}/Index.txt" >> "${saveFolder}/Index_Archive.txt"
rm "${saveFolder}/Index.txt"
fi

touch "${saveFolder}/Index.txt"

# In case of a crash, clean up from previous run.
[ -e "${saveFolder}/names.txt" ] \
&& rm "${saveFolder}/names.txt"

touch "${saveFolder}/names.txt"

# Ensure that the trashed list file will exist even if no files were trashed
[ ! -e "${saveFolder}/Consolidate_Trashed_List_${runDateTime}.txt" ] \
&& touch "${saveFolder}/Consolidate_Trashed_List_${runDateTime}.txt"

# Set the internal field separator to newline to preserve spaces in file paths
IFS=$'\n'

# Set the start time now that the user interaction is done
Seconds_since_the_epoch_start=$( date +%s )

##########
### Find items
echo "--> Finding folders and ignoring any folder whose name contains a period and it's content (potential packages)."


echo "--> then finding files & generating list of MD5 checksums for file content."


find "${MainFolder}" -type d \
\! \( -name "*.*" -o -name "iPhoto Library" -o -name "iTunes" \) \
\! \( -path "*/*.*/*" -o -path "*/iPhoto Library/*" -o -path "*/iTunes/*" \
-o -path "${saveFolder}" -o -path "${saveFolder}/*" \) \
| (

while read folder
do

[ "${veryVerbose}" = "Yes" ] \
&& echo "folder=${folder};"

# Perform a 'find' within each of the folders for files in that folder
# (and not in subdirectories) Skip hidden files.
find "${folder}" -maxdepth 1 -type f \! -name ".*" \
| (

while read theFilePath
do
# Get an MD5 checksum for each file's combined content of both
# data and resource forks
fileCheckSum=$( cat "${theFilePath}" "${theFilePath}/rsrc" | md5 )
echo " ${fileCheckSum} ${theFilePath}"
echo "${fileCheckSum}${tab}${theFilePath}" \
>> "${saveFolder}/Index.txt"
done
)
done
)

Show_Time

echo "--> Finding and processing potential packages/bundles."
echo "--> Note; we look for a period in the base filename which" \
"may not be acurate in all cases."
# Find folders within $MainFolder which are possible packages and bundles
find "${MainFolder}" -type d \
\( -iname "*.*" -o -iname "iPhoto Library" -o -iname "iTunes" \) \
\! \( -name ".*" -o -ipath "*/*.*/*" -o -ipath "*/iPhoto Library/*" \
-o -ipath "*/iTunes/*" \) \
| (
while read folder
do

fileCheckSum=$( find "${folder}" -type f \! -name ".DS_Store" \
-exec cat '{}' '{}/rsrc' \; | md5 )

echo "${fileCheckSum} ${folder}"
echo "${fileCheckSum}${tab}${folder}" >> "${saveFolder}/Index.txt"

done
)

######### ;;;;;;;;
echo
Show_Time

unset fileSum[*] filePath[*]

echo "--> Sorting list of files and potential bundles by checksum then"
echo " Compare checksums."
sort -t "${tab}" "${saveFolder}/Index.txt" |
(
declare previousCheckSum=""

declare -i trashCount=0

# Process both files and application bundles.
while read theData
do
checkSum=${theData:0:32}
fileName=${theData:33}

[ "${veryVerbose}" = "Yes" ] \
&& echo "${checkSum} ${fileName};"

if [ "${checkSum}" = "${previousCheckSum}" ] ; then
# Duplicate file. Move current file to trash.

[ "${verbose}" = "Yes" ] \
&& echo -n " ="

# Make up trash name.
# Base name on path name so we can remember
# where file came from
trashFileName="${fileName}"
if [ "${trashFileName:0:7}" = "/Users/" ] ; then
trashFileName="${trashFileName#/Users/}"
trashFileName="${trashFileName#*/}" # chop user id
elif [ "${trashFileName:0:9}" = "/Volumes/" ] ; then
trashFileName="${trashFileName#/Volumes/}"
fi

trashFileName=$(echo -n "${trashFileName}" | tr "/" "~" )
trashFileName=$HOME"/.Trash/${trashFileName}"

# don't let a directory be copyied into another directory.
# Happens with applications.
# Might as well avoid files too.
[ ! -e "${trashFileName}" ] \
&& mv -n "${fileName}" "${trashFileName}"

if [ -e "${fileName}" ] ; then
# Move didn't work.
myRandom=$RANDOM
trashFileName="${trashFileName}~$(date '+%Y-%m-%d_%H.%M')~${myRandom}"
mv -n "${fileName}" "${trashFileName}"

if [ -e "${fileName}" ] ; then
# Odd , second move didn't work.
echo
echo "big time error "
echo "<><> fileName=${fileName};" \
"myRandom=${myRandom}" \
"trashFileName=${trashFileName};"
echo
else
# Ok, moved file to trash
echoTrashInfo
fi
else
# Ok, moved file to trash
echoTrashInfo
fi

else
# Output data for next step.
echo "${fileName##*/}${tab}${fileName}" \
>>"${saveFolder}/names.txt"
fi
previousCheckSum="${checkSum}"


done

{
echo " "
echo "TOTAL TRASHED ITEMS (bundles count as 1 item):${trashCount}"
}>> "${saveFolder}/Consolidate_Trashed_List_${runDateTime}.txt"
)


Show_Time



##########
### Compare names

echo "--> Sorting list of names."


# Sort (by basename)
sort -t "${tab}" "${saveFolder}/names.txt" |
(

declare previousBaseName="" \
baseName

declare -i alterCount

let alterCount=0

while read theData
do

baseName=${theData%${tab}*}
fileName=${theData#*${tab}}

[ "${veryVerbose}" = "Yes" ] \
&& echo "${baseName}; ${fileName};"

if [ "${previousBaseName}" = "${baseName}" ] ; then

[ "${verbose}" = "Yes" ] \
&& echo -n " ="

# Rename files with duplicate names by appending #X prior to
# the filename extension (before the last period.)

directoryName="${fileName%/*}"
extension="${baseName##*.}"

# Was an extension found?
if [ "${extension}" != "${baseName}" ] ; then
# extension found, since a period was in the data
extension=".${extension}"
name="${baseName%.*}"
else
extension=""
name="${baseName}"
fi

while [ "1" = "1" ] ; do # do forever... simulate do until

let alterCount+=1
actualAlteration=${alterCount}

if [ ${alterCount} -gt 9999 ] ; then
echo
echo "================== odd: alterCount too big!!! "
echo "previousBaseName=${previousBaseName}; baseName=${baseName}; fileName=${fileName}; "
echo "================== odd: alterCount too big!!! "
echo
actualAlteration=${alterCount}$RANDOM
break
fi

# Manually implement do until.
# If file doesn't exist, break. Unused name found.
[ ! -e "${directoryName}/${name} #${actualAlteration}${extension}" ] \
&& break

done

# rename
newName="${directoryName}/${name} #${actualAlteration}${extension}"

mv "${fileName}" "${newName}"

echo " +${newName}"

else
let alterCount=0

fi

previousBaseName="${baseName}"

done

)

Show_Time

##########
### Moving items to top-level
if [ "${MoveFiles}" = "Yes" ] ; then
echo "--> Finding possible packages/bundles and moving them to the top-level."
find "${MainFolder}" -mindepth 1 -type d \
\( -iname "*.*" -o -iname "iPhoto Library" -o -iname "iTunes" \) \
\! \( -name ".*" -o -ipath "*/*.*/*" -o -ipath "*/iPhoto Library/*" \
-o -ipath "*/iTunes/*" \) \
-exec mv -n '{}' "${MainFolder}" \;

echo "--> Finding files and moving them to the top-level."
folders=( $(find "${MainFolder}" -mindepth 1 -type d \
\! \( -name "*.*" -o -name "iPhoto Library" -o -name "iTunes" \) \
\! \( -path "*/*.*/*" -o -path "*/iPhoto Library/*" -o -path "*/iTunes/*" \
-o -path "${saveFolder}" -o -path "${saveFolder}/*" \) ) )

# Perform a 'find' within each of the folders for files in that folder
# (and not in subdirectories)
for folder in ${folders[*]} ; do
find "${folder}" -maxdepth 1 -type f \! -name ".*" \! -name "Index*.txt" -exec mv -n '{}' "${MainFolder}" \;
done
fi

echo "--> Deleting .DS_Store and .FBC files from subfolders."
find -d "${MainFolder}" -type f \( -name ".DS_Store" -o -name ".FBCIndex*" -o -name ".FBCSemaphoreFile" \) -delete 2>/dev/null

echo "--> Deleting empty subfolders. (This only deletes folders which are completely empty.)"
### 2>/dev/null
find -d "${MainFolder}" -type d -empty -delete

#echo "--> Sorting index file."
#Why???
# If the index file exists, move it
#sort -u "${saveFolder}/Index_Archive.txt" "${saveFolder}/Index.txt" > "${saveFolder}/Index_Archive.tmp" 2>/dev/null
#mv "${saveFolder}/Index_Archive.tmp" "${saveFolder}/Index_Archive.txt" 2>/dev/null
#rm "${saveFolder}/Index.txt" 2>/dev/null



##########
### Regroup map files into their own folders
if [ "${RegroupMapFiles}" = "Yes" ] ; then
echo "--> Finding map file groups in top-level and regrouping them to their own folders in the top-level."

# deal with .frq first, .frq indicates group with differing filename end prior to .shp
Maps=( $( find "${MainFolder}" -maxdepth 1 -iname "*.frq" ) )
for Map in ${Maps[*]} ; do
[ ! -e "${Map}" ] && continue
[ "${Map}" = "" ] && continue
fullfilename=$( basename "${Map}" )
filename="${fullfilename%.*}"
if [ ! -e "${saveFolder}/${filename}.map" ] ; then
mkdir "${saveFolder}/${filename}.map"
find "${MainFolder}" -maxdepth 1 -iname "${filename}*.*" -type f -exec mv '{}' "${saveFolder}/${filename}.map" \;
fi
done

Maps=( $( find "${MainFolder}" -maxdepth 1 -iname "*.shp" ) )
for Map in ${Maps[*]} ; do
[ ! -e "${Map}" ] && continue
[ "${Map}" = "" ] && continue
fullfilename=$( basename "${Map}" )
filename="${fullfilename%.*}"
if [ ! -e "${saveFolder}/${filename}.map" ] ; then
mkdir "${saveFolder}/${filename}.map"
find "${MainFolder}" -maxdepth 1 -iname "${filename}.*" -type f -exec mv '{}' "${saveFolder}/${filename}.map" \;
fi
done
fi

echo
echo "DONE."
echo "NOTE: Invisible files and files with unresolved name conflicts may be left within any remaining subfolders!"
Show_Time

exit 0</pre>

Although I've heard many good things about 'bash', and haven't programmed in it, this is a variation of the 'Bourne' shell. This is the most difficult of the Unix (or Linux) shell languages. It's used by programmers writing operating system files that won't change, and need to run fast.

People who needed to use a language interactively, to control jobs running, used the C-shell (long ago, at least). The slowest, but most intelligible shell language was the Korn shell (like Pascal, I was told). (I enjoyed it.) I don't know what acronyms these now have.

My first thought upon looking at the problem was to use either a shell language or the little language 'awk'. 'Awk' is very simple in concept, and was designed for problems like this: it's for writing reports, the file names are variables, and files are strings one can find, split, compare, join, &c.

However, when these languages were written, one didn't have 8 GB files. GNU/Linux and, presumably, Darwin, are open source and legally changeable. This is really a problem best addressed in Debian GNU/Linux, for all the languages and commands are being monitored and changed; so the file sizes are likely keeping up with the times. If not, and one can read C++ programs, GNU/Linux invites people to alter the limits in the source code, and recompile the commands - or 'awk' language.

If size limits prove a problem, the exact 'bash' program may run with no changes on GNU/Linux. Just a suggestion.

Bruce

Well, I though maybe it was some problem with the arrays which I mostly revised out. I tried to put in more error checking to catch any errors. There are a few more revisions that I could do. I guess that I will try.

Most of the heavy processing is done in system facilities. The script heavily relies on piping data .. the | .

You can download the latest version of Bash & use that. I'll take a look at that again. I found one site, but the installation required a compilation which didn't seem to include all the include files.

For me this was an opportunity to learn about Bash programming. I'd have leaned toward programming in Python. Although, I'd have assume that there are very big bash programs. I do know that Ubuntu programs in Python.

Robert

Here is my latest Consolidated Bash Script.

Limitations:
1) Limitations on filenames. Doesn't work with names with such characters like quote, double quote, etc.
2) Does work with fat partitions.

Robert



<pre style="
font-family: Monaco, 'Courier New', Courier, monospace;
font-size: 10px;
font-weight: normal;
margin: 0px;
padding: 5px;
border: 1px solid #000000;
width: 720px; height: 340px;
color: #000000;
background-color: #FFEE80;
overflow: auto;"
title="">#!/bin/bash

# Purpose of this script:
# Find folders within a specified top folder which are not bundles. Find files
# within each found folder (and not within subfolders because each folder is
# handled individually.)
# Find potential bundles and checksum their entire content as one item.
# Get MD5 checksums for each item and save an index of all checksums and
# filepaths. Sort the index by checksum. Compare each
# item's checksum and if a duplicate is found, move the duplicate to the trash.
# Compare each item's name and if a duplicate is found, append incrementing #X
# to the name prior to the last period (as in "Filename #2.jpg")
#
# Optionally move files and potential bundles to the top-level folder
# Delete .DS_Store, .FBCIndex*, and .FBCSemaphoreFile files and subsequently
# delete empty subfolders. Optionally regroup map files into folders based on
# the name of .frq file if found, or .shp file
#
# The inspiration for the design of this script as well as some code
# comes from JulieJulieJulie's Consolidate17 script. See:
# http://discussions.apple.com/message.jspa?messageID=11577992#11577992
#
# Copyright 2010 rccharles
#
# GNU General Public License
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, version 3
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# For a copy of the GNU General Public License see
# <http://www.gnu.org/licenses/>.
#
copyRight="Copyright (c) 2010 rccharles. GNU General Public License."
#
#


# debug info
export PS4='+(${BASH_SOURCE}:${LINENO}):'

## not in the tiger version of bash ${FUNCNAME[0]:+${FUNCNAME[0]}(): }'


##########
### Functions (subroutines which can be called by name)

function helpHelpIsAllINeed () {
echo
echo "${savedCommandName##*/} [ options ] [ directory-to-compress ]"
echo
echo "options:"
echo "-h Display quick help."
echo
echo "-m Moving files is disabled."
echo "-M Move files to top level."
echo
echo "-r Regrouping map files is disabled."
echo "-R Regroup map files."
echo
echo "-V Display very detailed messages."
echo
echo "example:"
echo " ./${savedCommandName##*/} -mrV /Users/mac/cons/see"
echo
}

function Show_Time () {
let currentRunTime="$( date +%s ) - ${Seconds_since_the_epoch_start}"
let stepRunTime=currentRunTime-previousRunTime
echo " Elapsed time in seconds since start of processing: ${currentRunTime}" \
" Time of this step: ${stepRunTime}"
echo
let previousRunTime=currentRunTime
}

function echoTrashInfo () {
echo " -${trashFileName}"
echo "${fileName} > ${trashFileName}" \
>>"${saveFolder}/Consolidate_Trashed_List_${runDateTime}.txt"

case "${fileType}" in
f ) let trashFileCount+=1
;;
d ) let trashDirectoryCount+=1
;;
* ) Echo
Echo "Error --- Error in echoTrashInfo. fileName=${fileName};"
Echo "Error --- fileType=${fileType};"
Echo
;;
esac

}

declare folder \
MainFolder \
MoveFiles="Question" \
RegroupMapFiles="Question" \
runDateTime=$(date '+%Y-%m-%d_%H.%M') \
Seconds_since_the_epoch_start


declare -x tab=$'\t' \
trashFolder \
verbose="Yes" \
veryVerbose="Yes" \
whereIsTrash


declare -i count=0 \
offsetMainFolder \
currentRunTime \
previousRunTime=0 \
stepRunTime \
trashDirectoryCount=0 \
trashFileCount=0

declare -a filePath \
fileName \
fileSum \
folders


savedCommandName="$0"
echo
echo "${savedCommandName} script revised $(GetFileInfo -m $0)"
echo
bigVersion="Lizard 4"

echo "=============================================================="
echo
echo " ${bigVersion}"
echo " ${bigVersion}"
echo " ${bigVersion}"
echo
echo "=============================================================="
echo
echo "${copyRight}"
echo


# If the utility md5 is not present then just quit, it's needed for the checksums.
[ ! -x /sbin/md5 ] && exit 2

# Check for command-line options used when calling the script
if [ $# -gt 0 ] ; then
while getopts "hmMrRV" Option ; do
case "${Option}" in
h ) helpHelpIsAllINeed
echo "Bye for now."
exit 7
;;
m ) echo "-m argument used on command line, moving files is disabled."
echo
MoveFiles="No"
;;
M ) echo "Moving files. -M argument used on command line."
echo
MoveFiles="Yes"
;;
r ) echo "-r argument used on command line, regrouping map files is disabled."
echo
RegroupMapFiles="No"
;;
R ) echo "Regrouping map files. -R argument used on command line."
echo
RegroupMapFiles="Yes"
;;
V ) echo "Display very detailed messages. -V argument used on command line."
echo
veryVerbose="Yes"
;;
* ) echo "Unknown argument among arguments $* on command line."
helpHelpIsAllINeed
exit 6
;;
esac
done
fi

# We're done with switches / options from the command line
shift $(($OPTIND - 1))

# Main folder can be declared on the command line or interactively
MainFolder="${1}"

# If the main folder doesn't exist, prompt once, then just quit if it's still not there.
if [ ! -d "${MainFolder}" ] ; then
echo "Folder not specified. Type the path to the top-level folder or drag"
echo "it into this Terminal window, then click back in this window and press return:"
read MainFolder
echo
fi

[ ! -d "${MainFolder}" ] \
&& echo "Folder not found. Was ${MainFolder}" \
&& exit 4

[ "${MainFolder}" = "/" ] \
&& echo "You must be crazy." \
&& exit 5

echo "Holding down Control-c on the keyboard will stop this script in an emergency."
echo

if [ "${MoveFiles}" = "Question" ] ; then
echo "WARNING: allowing the script to move files could cause problems!"

echo
echo "Do you want to move all files and bundles to the top level at the end of the run?"
read -n1 -p "Press 'n' to NOT move files and keep the folder structure intact; otherwise press any key. "
if [ "${REPLY}" = "n" -o "${REPLY}" = "N" ] ; then
MoveFiles="No"
else
MoveFiles="Yes"
fi
echo


fi

if [ "${RegroupMapFiles}" = "Question" ] ; then
echo "Do you want to disable regrouping of map files into folders based on the .shp filename?"
read -n1 -p "Press 'n' to NOT regroup map files; otherwise press any key. "
if [ "${REPLY}" = "n" -o "${REPLY}" = "N" ] ; then
RegroupMapFiles="No"
Else
RegroupMapFiles="Yes"
fi
echo
fi

[ "${veryVerbose}" = "Yes" ] \
&& ulimit -a \
&& df \
&& echo

[ "${veryVerbose}" = "Yes" ] \
&& echo "MainFolder=${MainFolder};"

# get physical path
if [ -d "$MainFolder" ]; then
pushd $PWD
cd "$MainFolder"
MainFolder=$( /bin/pwd -P )
popd
else
# just to be sure.
echo "You need to supply a directory name for ${MainFolder}"
exit 8
fi

let offsetMainFolder="${#MainFolder}"-1

# Ensure that the path is in a consistant format. i.e. no //
# drop trailing directory separator if needed
[ "${MainFolder:${offsetMainFolder}}" = "/" ] \
&& MainFolder="${MainFolder:0:${offsetMainFolder}}"

# Place all created files in one directory.
saveFolder="${MainFolder}/saved--Consolidated--Data"

# Figure out the trash folder name. Each user has his or her own trash folder.
# Each volume has a different trash folder. On the startup drive,
# the trash folders are in the users home folder. On the other volumes, the
# trash folder contains a folder with the user id for each user.
#
# This isn't perfect.
if [ $( echo -n "${MainFolder:0:7}" | tr '[:upper:]' '[:lower:]' ) \
= "/users/" ] ; then
whereIsTrash="u"
trashFolder="${HOME}/.Trash"
elif [ $( echo -n "${MainFolder:0:9}" | tr '[:upper:]' '[:lower:]' ) \
= "/volumes/" ] ; then
whereIsTrash="v"
diskName="${MainFolder:9}"
diskName="${diskName%%/*}"
trashFolder="/Volumes/${diskName}/.Trashes"
if [ ! -d ${trashFolder} ] ; then
# Needs work. Should be ok, since folder is created when formated.
Echo "Creating trash folder. ${trashFolder};"
saveUmask=$( umask )
mkdir ${trashFolder}
umask ${saveUmask}
fi
currentId=$( id -ru )
trashFolder="${trashFolder}/${currentId}"
if [ ! -d ${trashFolder} ] ; then
Echo "Creating user's individual trash folder. ${trashFolder};"
saveUmask=$( umask )
umask 077
mkdir ${trashFolder}
umask ${saveUmask}
fi
else
echo "Not sure where main folder resides. defulat to ~/.Trash"
whereIsTrash="u"
trashFolder="${HOME}/.Trash"
fi


[ "${veryVerbose}" = "Yes" ] \
&& echo "MainFolder=${MainFolder};" \
&& echo "saveFolder=${saveFolder};" \
&& echo "trashFolder=${trashFolder};"



[ ! -d "${saveFolder}" ] \
&& mkdir "${saveFolder}"

#Housekeeping for Index files.

# If the index archive file does not exist,
# create it so that it will be there for sorting and merging later
[ ! -e "${saveFolder}/Index_Archive.txt" ] \
&& touch "${saveFolder}/Index_Archive.txt"

# If the index file exists, archive it and be sure to delete the index.
if [ -e "${saveFolder}/Index.txt" ] ; then
echo "----------------------------" >> "${saveFolder}/Index_Archive.txt"
cat "${saveFolder}/Index.txt" >> "${saveFolder}/Index_Archive.txt"
rm "${saveFolder}/Index.txt"
fi

touch "${saveFolder}/Index.txt"

# In case of a crash, clean up from previous run.
[ -e "${saveFolder}/names.txt" ] \
&& rm "${saveFolder}/names.txt"

touch "${saveFolder}/names.txt"

# Ensure that the trashed list file will exist even if no files were trashed
[ ! -e "${saveFolder}/Consolidate_Trashed_List_${runDateTime}.txt" ] \
&& touch "${saveFolder}/Consolidate_Trashed_List_${runDateTime}.txt"

[ "${veryVerbose}" = "Yes" ] \
&& ls -l "${saveFolder}/Index.txt" \
&& ls -l "${saveFolder}/names.txt" \
&& ls -l "${saveFolder}/Consolidate_Trashed_List_${runDateTime}.txt" \
&& ls -ld "${trashFolder}" \
&& ls -ld "${trashFolder}/.." \
&& echo

echo "--> Deleting .DS_Store and .FBC files from subfolders."
find -d "${MainFolder}" -type f \( -name ".DS_Store" -o -name ".FBCIndex*" \
-o -name ".FBCSemaphoreFile" \) -delete


if [ "${veryVerbose}" = "Yes" ] ; then
echo "Gathering debug information. May take a moment. Counting files."
fileCountBefore=$( find ${MainFolder} -type f \
\! \( -path "${saveFolder}" -o -path "${saveFolder}/*" \) \
| wc -l )
trashCountBefore=$( find ${trashFolder} -type f | wc -l )
fi

# Set the internal field separator to newline to preserve spaces in file paths
IFS=$'\n'

# Set the start time now that the user interaction is done
Seconds_since_the_epoch_start=$( date +%s )

##########
### Find items
echo "--> Finding folders and ignoring any folder whose name contains a period and it's content (potential packages)."


echo "--> then finding files & generating list of MD5 checksums for file content."


find "${MainFolder}" -type d \
\! \( -name "*.*" -o -name "iPhoto Library" -o -name "iTunes" \) \
\! \( -path "*/*.*/*" -o -path "*/iPhoto Library/*" -o -path "*/iTunes/*" \
-o -path "${saveFolder}" -o -path "${saveFolder}/*" \) \
| (

while read folder
do

[ "${veryVerbose}" = "Yes" ] \
&& echo "folder=${folder};"

# Perform a 'find' within each of the folders for files in that folder
# (and not in subdirectories) Skip hidden files.
find "${folder}" -maxdepth 1 -type f \! -name ".*" \
| (

while read theFilePath
do
# Get an MD5 checksum for each file's combined content of both
# data and resource forks
fileCheckSum=$( cat "${theFilePath}" "${theFilePath}/rsrc" | md5 )
echo " ${fileCheckSum} ${theFilePath}"
echo "${fileCheckSum}${tab}f${tab}${theFilePath}" \
>> "${saveFolder}/Index.txt"
done
)
done
)

Show_Time

echo "--> Finding and processing potential packages/bundles."
echo "--> Note; we look for a period in the base filename which" \
"may not be acurate in all cases."
# Find folders within $MainFolder which are possible packages and bundles
find "${MainFolder}" -type d \
\( -iname "*.*" -o -iname "iPhoto Library" -o -iname "iTunes" \) \
\! \( -name ".*" -o -ipath "*/*.*/*" -o -ipath "*/iPhoto Library/*" \
-o -ipath "*/iTunes/*" \) \
| (
while read folder
do

fileCheckSum=$( find "${folder}" -type f \! -name ".DS_Store" \
-exec cat '{}' '{}/rsrc' \; | md5 )

echo "${fileCheckSum} ${folder}"
echo "${fileCheckSum}${tab}d${tab}${folder}" >> "${saveFolder}/Index.txt"

done
)

######### ;;;;;;;;
echo
Show_Time

echo "--> Sorting list of files and potential bundles by checksum then"
echo " Compare checksums."
sort -t "${tab}" "${saveFolder}/Index.txt" |
(
declare previousCheckSum=""

declare -i trashCount=0

# Process both files and application bundles.
while read theData
do
checkSum="${theData:0:32}"
fileType="${theData:33:1}"
fileName="${theData:35}"

[ "${veryVerbose}" = "Yes" ] \
&& echo "${checkSum} ${fileName};"

if [ "${checkSum}" = "${previousCheckSum}" ] ; then
# Duplicate file. Move current file to trash.

[ "${verbose}" = "Yes" ] \
&& echo -n " ="

# Make up trash name.
# Base name on path name so we can remember
# where file came from
trashFileName="${fileName}"
case "${whereIsTrash}" in
u ) trashFileName="${trashFileName:7}"
trashFileName="${trashFileName#*/}" # chop user id
;;
v) trashFileName="${trashFileName:9}"
;;
* ) echo "<><><><> Serious error. whereIsTrash=${whereIsTrash};"
esac

trashFileName=$(echo -n "${trashFileName}" | tr "/" "~" )
trashFileName="${trashFolder}/${trashFileName}"

# don't let a directory be copyied into another directory.
# Happens with applications.
# Might as well avoid files too.
[ ! -e "${trashFileName}" ] \
&& mv -n "${fileName}" "${trashFileName}"

if [ -e "${fileName}" ] ; then
# Move didn't work.
myRandom=$RANDOM
trashFileName="${trashFileName}~$(date '+%Y-%m-%d_%H.%M')~${myRandom}"
mv -n "${fileName}" "${trashFileName}"

if [ -e "${fileName}" ] ; then
# Odd , second move didn't work.
echo
echo "big time error "
echo "<><> fileName=${fileName};" \
"myRandom=${myRandom}" \
"trashFileName=${trashFileName};"
echo
else
# Ok, moved file to trash
echoTrashInfo
fi
else
# Ok, moved file to trash
echoTrashInfo
fi

else
# Output data for next step.
# Mac OS X hfs+ defaults to case independent names, so
# sort on lower case.
# ie A folder may not contain the names abc & Abc.
lowerBaseName=$( echo -n "${fileName##*/}" | \
tr '[:upper:]' '[:lower:]' )
echo "${lowerBaseName}${tab}${fileName}" \
>>"${saveFolder}/names.txt"
fi
previousCheckSum="${checkSum}"


done

outLine="Total trashed files: ${trashFileCount}"

outLine="${outLine} Total trashed applications/bundles: ${trashDirectoryCount}"
echo
echo "${outLine}"
echo
{
echo
echo "${outLine}"
echo
}>> "${saveFolder}/Consolidate_Trashed_List_${runDateTime}.txt"
)


Show_Time



##########
### Compare names

echo "--> Sorting list of names."


# Sort (by basename)
sort -t "${tab}" "${saveFolder}/names.txt" |
(

declare previousBaseName="" \
baseName

declare -i alterCount

let alterCount=0

while read theData
do
echo "..${theData}"
lowerBaseName="${theData%${tab}*}"
fileName="${theData#*${tab}}"
baseName="${fileName##*/}"

[ "${veryVerbose}" = "Yes" ] \
&& echo "${lowerBaseName}; ${baseName}; ${fileName};"

if [ "${previousBaseName}" = "${lowerBaseName}" ] ; then

[ "${verbose}" = "Yes" ] \
&& echo -n " ="

# Rename files with duplicate names by appending #X prior to
# the filename extension (before the last period.)

directoryName="${fileName%/*}"
extension="${baseName##*.}"

# Was an extension found?
if [ "${extension}" != "${baseName}" ] ; then
# extension found, since a period was in the data
extension=".${extension}"
name="${baseName%.*}"
else
extension=""
name="${baseName}"
fi

while [ "1" = "1" ] ; do # do forever... simulate do until

let alterCount+=1
actualAlteration="${alterCount}"

if [ ${alterCount} -gt 9999 ] ; then
echo
echo "================== odd: alterCount too big!!! "
echo "previousBaseName=${previousBaseName}; baseName=${baseName}; fileName=${fileName}; "
echo "================== odd: alterCount too big!!! "
echo
actualAlteration="${alterCount}${RANDOM}"
break
fi

# Manually implement do until.
# If file doesn't exist, break. Unused name found.
[ ! -e "${directoryName}/${name} #${actualAlteration}${extension}" ] \
&& break

done

# rename
newName="${directoryName}/${name} #${actualAlteration}${extension}"

mv -n "${fileName}" "${newName}"
if [ -e "${fileName}" ] ; then
# Odd , rename didn't work.
echo
echo "another big time error "
echo "<><><> fileName=${fileName};" \
"newName=${newName}"
echo
else
# all went well with rename
echo " +${newName}"
fi


else
let alterCount=0

fi

previousBaseName="${lowerBaseName}"

done

)

Show_Time

##########
### Moving items to top-level
if [ "${MoveFiles}" = "Yes" ] ; then
echo "--> Finding possible packages/bundles and moving them to the top-level."
find "${MainFolder}" -mindepth 1 -type d \
\( -iname "*.*" -o -iname "iPhoto Library" -o -iname "iTunes" \) \
\! \( -name ".*" -o -ipath "*/*.*/*" -o -ipath "*/iPhoto Library/*" \
-o -ipath "*/iTunes/*" \) \
-exec mv -n '{}' "${MainFolder}" \;

echo "--> Finding files and moving them to the top-level."
folders=( $(find "${MainFolder}" -mindepth 1 -type d \
\! \( -name "*.*" -o -name "iPhoto Library" -o -name "iTunes" \) \
\! \( -path "*/*.*/*" -o -path "*/iPhoto Library/*" -o -path "*/iTunes/*" \
-o -path "${saveFolder}" -o -path "${saveFolder}/*" \) ) )

# Perform a 'find' within each of the folders for files in that folder
# (and not in subdirectories)
for folder in ${folders[*]} ; do
find "${folder}" -maxdepth 1 -type f \! -name ".*" -exec mv -n '{}' "${MainFolder}" \;
done
fi

echo "--> Deleting empty subfolders. (This only deletes folders which are completely empty.)"

find -d "${MainFolder}" -type d -empty -delete


##########
### Regroup map files into their own folders
if [ "${RegroupMapFiles}" = "Yes" ] ; then
echo "--> Finding map file groups in top-level and regrouping them to their own folders in the top-level."

# deal with .frq first, .frq indicates group with differing filename end prior to .shp
Maps=( $( find "${MainFolder}" -maxdepth 1 -iname "*.frq" ) )
for Map in ${Maps[*]} ; do
[ ! -e "${Map}" ] && continue
[ "${Map}" = "" ] && continue
fullfilename=$( basename "${Map}" )
filename="${fullfilename%.*}"
if [ ! -e "${saveFolder}/${filename}.map" ] ; then
mkdir "${saveFolder}/${filename}.map"
find "${MainFolder}" -maxdepth 1 -iname "${filename}*.*" -type f \
-exec mv -n '{}' "${saveFolder}/${filename}.map" \;
fi
done

Maps=( $( find "${MainFolder}" -maxdepth 1 -iname "*.shp" ) )
for Map in ${Maps[*]} ; do
[ ! -e "${Map}" ] && continue
[ "${Map}" = "" ] && continue
fullfilename=$( basename "${Map}" )
filename="${fullfilename%.*}"
if [ ! -e "${saveFolder}/${filename}.map" ] ; then
mkdir "${saveFolder}/${filename}.map"
find "${MainFolder}" -maxdepth 1 -iname "${filename}.*" -type f \
-exec -n mv '{}' "${saveFolder}/${filename}.map" \;
fi
done
fi

Show_Time
echo

if [ "${veryVerbose}" = "Yes" ] ; then
echo "Gathering post run debug information. May take a moment." \
"Counting files."
fileCountAfter=$( find ${MainFolder} -type f \
\! \( -path "${saveFolder}" -o -path "${saveFolder}/*" \) \
| wc -l )
trashCountAfter=$( find ${trashFolder} -type f | wc -l )
let trashDiff=trashCountAfter-trashCountBefore
let calculateFileCountAfter=fileCountAfter+trashDiff
echo
echo "Information on files and directories in folder/directory ${MainFolder}"
echo "fileCountBefore=${fileCountBefore};" \
"fileCountAfter=${fileCountAfter};"
echo
echo "trashDiff=${trashDiff}; " \
"trashCountBefore=${trashCountBefore};" \
"trashCountAfter=${trashCountAfter};"
echo
echo "==========="
echo "calculateFileCountAfter=${calculateFileCountAfter};"
echo
if [ "${fileCountBefore}" -ne "${calculateFileCountAfter}" ] ; then
echo "- - - - - - - -"
echo "- - - - - - - - Error: Some files have gone missing."
echo "- - - - - - - -" \
"fileCountBefore should be equal to calculateFileCountAfter"
echo "- - - - - - - - Where files deleted from the trash?"
echo "- - - - - - - -"
fi

fi
echo "DONE."
echo "NOTE: Invisible files and files with unresolved name conflicts may be left within any remaining subfolders!"
echo
echo "${savedCommandName} script revised $(GetFileInfo -m $0)"
echo
echo "Bye from ${bigVersion}"
echo


exit 0</pre>