Using ditto in Terminal to batch copy directory and subfolders

Hi All-


I am not well versed in script but luckily figured out how to use ditto to copy directory files to my external hard drive.


ditto [source] [destination]


ditto [/private/var/automount/Users/techportal/Desktop/archive/Incoming/CPP] [/volumes/DATA\D1/Photogs\Import\2/]


That said I am facing two issues:


One- How to run this script as a batch to copy multiple directory folders to the destination volume in one go


Two- How to identify and rename folders that will not copy due to similar naming, upon copying, i.e. one folder is [STEVE] and one folder is [steve]


Ideally following the data transfer I would like to have a script that can compare the two collections by folder, file count and size to see if and where I missed copying any information.


I know that this is a tall order and that I am outside my abilities realm but I would apreciate any help and guidance.


Thank you.

iMac, Mac OS X (10.6.7)

Posted on Jul 13, 2011 12:45 PM

Reply
13 replies

Jul 14, 2011 3:33 PM in response to kemp0131

rsync doesn't rename files. Conceivably you could write a shell script to change the case of file names, but it's not clear how that would work.


One other thing: if any of the files have HFS extended attributes such as resource forks or Finder flags, you should pass the '-E' option to rsync when doing the actual copy. It isn't perfect, but it mostly preserves the attributes.

Jul 13, 2011 2:59 PM in response to kemp0131

Linc, that would be ideal but I am not quite sure how to accomplish this task besides drag and drop copy- and that is presenting issues due to filenames and permissions.


I've read the man ditto in Terminal further and it seems that I can do:


ditto src-1 src-2 ect. for each directory. Though I thought perhaps there is a script I can write to make a short cut. (My src directory names are quite long and I have 428 of them to copy)


Also I still can't quite figure out how to identify and rename folders that will not copy due to similar naming, upon copying, i.e. one folder is [STEVE] and one folder is [steve]


I know i can use ditto -v to produce reports but from there I am stuck.


Again any feedback or help would be much appreciated.

Jul 15, 2011 9:13 AM in response to kemp0131

I have written an applescript to rename files in the destination folder to avoid name collistions. It does not rename files in an imbedded folder from the source folder. It renames only files and folders in the first level of the source folder when they are copied to the destination folder.


Please test this script before usng in prouction. It did work once before I uploaded it! Gaurenteed to have been tested once!


Let me know if you need files in subfolders renamed too! Maybe I could work on this.


It's a applescript:



(* 
    Author: rccharles

    For testing, run in the Script Editor.
    Click on the Event Log tab to see the output from the log statement
    Click on Run

debug...
rsync -av --dry-run source/ target

 *)

on run
    -- Write a message into the event log.
    log "  --- Starting on " & ((current date) as string) & " --- "



    set theFolder to choose folder with prompt "Source folder. Files will be renamed as needed"

    --set theFolder to (get "Macintosh-HD:Users:mac:config:") as alias

    log " theFolder = " & theFolder
    set posixPath to POSIX path of theFolder
    -- Names of the folders to backup.  Use Unix names path.
    tell application "System Events"

        set sourceList to POSIX path of every file in folder posixPath

    end tell



    set my text item delimiters to ", "

    -- Volume to back up to.  
    set targetVolume to choose folder with prompt "Destination folder. Files will be renamed as needed"
    log "targetVolume = " & targetVolume
    set posixTargetVolume to POSIX path of targetVolume
    log "posixTargetVolume = " & posixTargetVolume



    repeat with theFile in sourceList
        set passedFile to theFile as string
        log "passedFile = " & passedFile
        set quotedPassedFile to quoted form of passedFile
        set theBaseName to do shell script "basename " & quotedPassedFile
        log "theBaseName= " & theBaseName

        -- Check if folder or file already exits

        set combinedName to posixTargetVolume & "/" & theBaseName
        log "combinedName = " & combinedName

        set passCombinedName to combinedName

        set theEnd to 99
        repeat with i from 1 to theEnd
            set newP to POSIX file passCombinedName as string
            log "newP = " & newP
            tell application "Finder"
                set existFileorFolder to exists file newP
            end tell
            log "existFileOrFolder = " & existFileorFolder & "  passCombinedName = " & passCombinedName

            if existFileorFolder is true then
                set passCombinedName to combinedName & "-" & i
            else
                exit repeat
            end if

        end repeat
        log "i = " & i
        if i ≥ theEnd then
            -- create a save name based on time.
            delay 1 -- make sure we do not use the same time.
            set passCombinedName to combinedName & ((current date) as string)
        end if
        log " passCombinedName = " & passCombinedName

        set passCombinedName to quoted form of passCombinedName
        log "passCombinedName = " & passCombinedName

        set quotedPassedFile to quoted form of passedFile

        -- Make the actual copy.
        set see to "ditto  -X -rsrc   " & quotedPassedFile & "     " & passCombinedName
        log "see= " & see

        try
            with timeout of 40 * 60 seconds
                set results to do shell script see
            end timeout
        on error errorMsg
            display dialog "error-2 " & errorMsg & " results for …" & return & see
        end try
        (*     
        on error errorMsg
            display dialog "error-3 " & errorMsg & " results for " & theFile
        end try
        *)
    end repeat
end run

-- ------------------------------------





The first thing that you need to do is to make the text into an AppleScript program.


Start the AppleScript Editor

/Applications/AppleScript/Script Editor.app

In Snow Leopard it's at: /Applications/Utilities/AppleScript Editor


Copy the script text to the Applescript editor.


Save the text to a file as an application and do not check any of the boxes below.

User uploaded file

If you want access to the script from your Script Menu, move the script (the saved script application file) to your

~/Library/Scripts folder. You can also drag it to your Dock or make an alias for it on the Desktop.<br><br>


To debug, run the script within the Applescript Editor. Click on the event log tab at the bottom of the window. Click on the run icon. The results from the log statement will be shown at the bottom of the screen.

User uploaded file

Jul 21, 2011 8:24 PM in response to kemp0131

I've been modifying the applescript to support name clashes and be a little more robust.



It still needs a lot of testing. Including subfolders isn't supported. They ignored now in the source & not overwritten in the target now. I'm looking at copying subfolders now. I have a coding stub!


--------------------------------


(* 
    Author: rccharles

    For testing, run in the Script Editor.
    Click on the Event Log tab to see the output from the log statement
    Click on Run

debug...
rsync -av --dry-run source/ target

 *)

on run
    -- Write a message into the event log.
    log "  --- Starting on " & ((current date) as string) & " --- "



    set theFolder to choose folder with prompt "Source folder. Files will be renamed as needed"

    --set theFolder to (get "Case-Sensitive:cs:") as alias

    log " theFolder = " & theFolder
    set posixPath to POSIX path of theFolder
    -- Names of the files and folders to backup.  Use Unix names path.
    tell application "System Events"
        set sourceFileList to POSIX path of every file in folder posixPath
    end tell
    log sourceFileList

    -- Destination folder  
    set targetFolder to choose folder with prompt "Destination folder. Files will be renamed as needed"
    log "targetFolder = " & targetFolder
    set posixTargetFolder to POSIX path of targetFolder
    log "posixTargetFolder = " & posixTargetFolder


    repeat with posixSourceFile in sourceFileList

        set combinedNameTarget to findTargetBaseName(posixSourceFile, posixTargetFolder)

        set quotedPassedFile to quoted form of posixSourceFile
        log "quotedPassedFile = " & quotedPassedFile
        set quotedcombinedNameTarget to quoted form of combinedNameTarget
        log "quotedCombinedNameTarget = " & quotedcombinedNameTarget

        -- Make the actual copy.
        set see to "ditto  -X -rsrc " & quotedPassedFile & " " & quotedcombinedNameTarget
        log "see= " & see

        try
            with timeout of 40 * 60 seconds
                set results to do shell script see
            end timeout
        on error errorMsg
            display dialog "error-2 " & errorMsg & " results for …" & return & see
        end try
        (*     
        on error errorMsg
            display dialog "error-3 " & errorMsg & " results for " & theSourceFile
        end try
        *)
    end repeat


    -- Copy folders.
    tell application "System Events"
        set sourceListFolder to POSIX path of every folder in folder posixPath
    end tell
    log sourceListFolder

end run

-- ------------------------------------

on findTargetBaseName(posixSourceItem, posixTargetFolder)
    log "---> in findTargetBaseName"
    log "posixSourceItem = " & posixSourceItem & "  class of " & class of posixSourceItem
    log "posixTargetFolder = " & posixTargetFolder & "  class of " & class of posixTargetFolder

    set quotedPassedItem to quoted form of posixSourceItem
    log "quotedPassedItem = " & quotedPassedItem




    set theBaseName to do shell script "basename " & quotedPassedItem
    log "theBaseName= " & theBaseName

    -- Check if folder or file already exits

    set alteredBaseName to theBaseName
    set theEnd to 99
    set foundName to false

    repeat with i from 1 to theEnd
        set combinedNameTarget to posixTargetFolder & alteredBaseName
        log "combinedNameTarget = " & combinedNameTarget & "   class is " & class of combinedNameTarget

        set newP to POSIX file combinedNameTarget
        log "newP = " & newP & "   class is " & class of newP

        set newP to newP as string
        log "newP = " & newP & "   class is " & class of newP
        tell application "Finder"
            set existItem to exists item newP
        end tell
        log "existItem = " & existItem & "  combinedNameTarget = " & combinedNameTarget
        if existItem is true then

            -- Use Unix to attempt to chop out the extenstion.
            -- Chop out all characters * to the fartherest ## period . to the right
            -- ( Yes, we do the chopping repeated when we could do it one time. We would
            --   do the chopping even when there was no duplicate in names.This way we
            --   avoid that waste. ) 
            set theUnixInfo to "theBaseName=" & quoted form of theBaseName & ";echo \"${theBaseName##*.}\""
            log "theUnixInfo = " & theUnixInfo
            set possibleExtension to do shell script theUnixInfo
            log "possibleExtension = " & possibleExtension

            -- Did we are did we not find an extension? No change on output means not found.
            log "character 1 = " & first character of theBaseName
            if possibleExtension = theBaseName or first character of theBaseName = "." then
                -- no extension found
                set alteredBaseName to theBaseName & "-" & i
                log "alteredBaseName = " & alteredBaseName
            else
                -- file as an extension
                set simpleName to text 1 thru (((length of possibleExtension) + 1) * -1) of theBaseName
                log "simpleName = " & simpleName
                set alteredBaseName to simpleName & "-" & i & "." & possibleExtension
                log "alteredBaseName = " & alteredBaseName
            end if

        else
            -- Found an unused name
            set foundName to true
            exit repeat
        end if

    end repeat
    log "i = " & i
    if foundName = false then
        -- create a save name based on time.
        delay 1 -- make sure we do not use the same time.
        set combinedNameTarget to posixTargetFolder & theBaseName & ((current date) as string)
        set foundName to true -- seems like a good idea
    end if

    return combinedNameTarget
end findTargetBaseName

Jul 29, 2011 8:37 PM in response to rccharles

Copies files and subfolders. Limitation -- Permissions may change on folder from source folder.



(* 
    Author: rccharles

    For testing, run in the Script Editor.
    Click on the Event Log tab to see the output from the log statement
    Click on Run

debug...
rsync -av --dry-run source/ target


    Copyright 2011 rccharles
    GNU General Public License 

    This program is free software: you can redistribute it and/or modify
    it under the terms of the GNU General Public License as published by
    the Free Software Foundation,  version 3

    This program is distributed in the hope that it will be useful,
    but WITHOUT ANY WARRANTY; without even the implied warranty of
    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
    GNU General Public License for more details.

    For a copy of the GNU General Public License see <http://www.gnu.org/licenses/>.



 *)

on run
    -- Write a message into the event log.
    log "  --- Starting on " & ((current date) as string) & " --- "



    set theSourceFolder to choose folder with prompt "Source folder. Files will be renamed as needed"

    --set theSourceFolder to (get "Case-Sensitive:cs:") as alias

    log " theSourceFolder = " & theSourceFolder
    set posixSourcePath to POSIX path of theSourceFolder

    -- Destination folder  
    set targetFolder to choose folder with prompt "Destination folder. Files will be renamed as needed"
    log "targetFolder = " & targetFolder
    set posixTargetFolder to POSIX path of targetFolder
    log "posixTargetFolder = " & posixTargetFolder

    processFolderLevel(posixSourcePath, posixTargetFolder)

end run


-- ------------------------------------

on processFolderLevel(posixSourcePath, posixTargetFolder)
    log "--> processFolderLevel"
    log "posixSourcePath = " & posixSourcePath & "  class of " & class of posixSourcePath
    log "posixTargetFolder = " & posixTargetFolder & "  class of " & class of posixTargetFolder


    -- Get the list of files in the Source folder.  Folders are gotten later.
    --   Use Unix names path.
    tell application "System Events"
        set sourceFileList to POSIX path of every file in folder posixSourcePath
    end tell
    log sourceFileList

    repeat with posixSourceFile in sourceFileList

        set combinedNameTarget to findTargetBaseName(posixSourceFile, posixTargetFolder)

        set quotedPassedFile to quoted form of posixSourceFile
        log "quotedPassedFile = " & quotedPassedFile
        set quotedcombinedNameTarget to quoted form of combinedNameTarget
        log "quotedCombinedNameTarget = " & quotedcombinedNameTarget

        -- Make the actual copy.
        set see to "ditto  -X -rsrc " & quotedPassedFile & " " & quotedcombinedNameTarget
        log "see= " & see

        try
            with timeout of 40 * 60 seconds
                set results to do shell script see
            end timeout
        on error errorMsg
            display dialog "error-2 " & errorMsg & " results for …" & return & see
        end try

    end repeat

    set sourceFileList to {}

    -- Copy folders.
    tell application "System Events"
        set sourceListFolder to POSIX path of every folder in folder posixSourcePath

    end tell
    log sourceListFolder

    repeat with posixSourceFolder in sourceListFolder
        set combinedNameTarget to findTargetBaseName(posixSourceFolder, posixTargetFolder) & "/"
        set quotedcombinedNameTarget to quoted form of combinedNameTarget
        log "quotedCombinedNameTarget = " & quotedcombinedNameTarget
        -- Make new folder.
        set see to "mkdir " & quotedcombinedNameTarget
        log "see= " & see
        try
            set results to do shell script see
        on error errorMsg
            display dialog "error-4 " & errorMsg & " results for …" & return & see
            return
        end try

        processFolderLevel(posixSourceFolder, combinedNameTarget)
    end repeat

    set sourceListFolder to {}

end processFolderLevel


-- ------------------------------------

on findTargetBaseName(posixSourceItem, posixTargetFolder)
    log "---> in findTargetBaseName"
    log "posixSourceItem = " & posixSourceItem & "  class of " & class of posixSourceItem
    log "posixTargetFolder = " & posixTargetFolder & "  class of " & class of posixTargetFolder

    set quotedPassedItem to quoted form of posixSourceItem
    log "quotedPassedItem = " & quotedPassedItem

    set theBaseName to do shell script "basename " & quotedPassedItem
    log "theBaseName= " & theBaseName

    -- Check if folder or file already exits

    set alteredBaseName to theBaseName
    set theEnd to 99
    set foundName to false

    repeat with i from 1 to theEnd
        set combinedNameTarget to posixTargetFolder & alteredBaseName
        log "combinedNameTarget = " & combinedNameTarget & "   class is " & class of combinedNameTarget

        set newP to POSIX file combinedNameTarget
        log "newP = " & newP & "   class is " & class of newP

        set newP to newP as string
        log "newP = " & newP & "   class is " & class of newP
        tell application "Finder"
            set existItem to exists item newP
        end tell
        log "existItem = " & existItem & "  combinedNameTarget = " & combinedNameTarget
        if existItem is true then

            -- Use Unix to attempt to chop out the extenstion.
            -- Chop out all characters * to the fartherest ## period . to the right
            -- ( Yes, we do the chopping repeated when we could do it one time. We would
            --   do the chopping even when there was no duplicate in names.This way we
            --   avoid that waste. ) 
            set theUnixInfo to "theBaseName=" & quoted form of theBaseName & ";echo \"${theBaseName##*.}\""
            log "theUnixInfo = " & theUnixInfo
            set possibleExtension to do shell script theUnixInfo
            log "possibleExtension = " & possibleExtension

            -- Did we are did we not find an extension? No change on output means not found.
            log "character 1 = " & first character of theBaseName
            if possibleExtension = theBaseName or first character of theBaseName = "." then
                -- no extension found
                set alteredBaseName to theBaseName & "-" & i
                log "alteredBaseName = " & alteredBaseName
            else
                -- file as an extension
                -- length of the extension 
                --  + one for period 
                --  + one to get the end of previous character
                --   boundary from right
                set simpleName to text 1 thru (((length of possibleExtension) + 1 + 1) * -1) of theBaseName
                log "simpleName = " & simpleName
                set alteredBaseName to simpleName & "-" & i & "." & possibleExtension
                log "alteredBaseName = " & alteredBaseName
            end if

        else
            -- Found an unused name
            set foundName to true
            exit repeat
        end if

    end repeat
    log "i = " & i
    if foundName = false then
        -- create a save name based on time.
        delay 1 -- make sure we do not use the same time.
        set combinedNameTarget to posixTargetFolder & theBaseName & ((current date) as string)
        set foundName to true -- seems like a good idea
    end if

    return combinedNameTarget
end findTargetBaseName



fyi -- create test case folder.



#!/bin/bash
echo "in $0"
cd /Volumes/Case-Sensitive
mkdir CopyTests
cd CopyTests
# create test files & folders
mkdir A.dir
cd A.dir
mkdir STEVE
touch steve
touch Steve

cd ..
mkdir a.dir
cd a.dir
mkdir a.dir
mkdir a
mkdir A
touch out.TXt
touch out.txt
touch out.Txt
touch OUT.txt
touch OUt.txt
touch Out.txt
mkdir STEVE
touch steve
touch Steve
cd a.dir
mkdir a.dir
mkdir a
mkdir A
touch out.TXt
touch out.txt
touch out.Txt
touch OUT.txt
touch OUt.txt
touch Out.txt
mkdir STEVE
touch steve
touch Steve

cd ..

cd ..
mkdir dirA
touch out.TXt
touch out.txt
touch out.Txt
touch OUT.txt
touch OUt.txt
touch Out.txt
mkdir STEVE
touch steve
touch Steve
# the weird ones
touch "'weird one'"
touch "'Weird One'"
mkdir "'Weird one'"
touch "'single quote"
touch "'Single quote"
touch "\"double quote"
mkdir "\"Double Quote"
mkdir ":colon :"
touch "space here"
touch "Space Here"
mkdir "Space here"
touch "name.space here"
touch "name.Space Here"
mkdir "name.Space here"
touch ".hidden"
touch ".Hidden"
touch ".Hidden.txt"
touch ".hidden.txt"
touch ".hidden.txt space"


echo "--------------------------------------"
pwd
ls -ld
ls -Rl

Oct 2, 2011 2:18 PM in response to kemp0131

Hello,


I'm reading this message a bit late. You may have already obtained the solution. Anyways, for those who would like to note it down.


The standard Unix way to copy a complete volume to another volume is to use a combination of "find" and "cpio".


This process is roughly the same whatever the Unix flavor. The only difficulty is in figuring out what switches to pass to "cpio": it differs from cpio and Unix versions to another.


Then - especially for MacOSX - the output of the "find" command may contain characters that will require being escaped to get through the piping to cpio. If the Volume contains user data, better use the "sed" script to escape all what is non "base set". Be aware that this sed script covers the most current cases. If you have files with very odd characters, it may fail. The execution will fail. You will have to rename the offending file and relaunch the script.


Here it is:


A/ You prepare the destination volume

B/ You position yourself at the base of the origin volume / directory

C/ You run the command


cd (origin volume path)

find * -print | sed -e 's/\([^a-zA-Z0-9_\/\.\-]\)/\\\1/g' | cpio -pdmu (destination volume path - relative or absolute)


Note: All files meta information will be maintained.


Then, if you have (made the error of creating) Volumes of different case sensitivness, you'll have to resolve them manually, as the "-u" option will override any duplicate by the last one.




Note:

For HPUX


find * -print | cpio -pdxmU


For Linux


find * -print | cpio -pdum

This thread has been closed by the system or the community team. You may vote for any posts you find helpful, or search the Community for additional answers.

Using ditto in Terminal to batch copy directory and subfolders

Welcome to Apple Support Community
A forum where Apple customers help each other with their products. Get started with your Apple Account.