11 Replies Latest reply: Jun 7, 2014 5:40 AM by Tony T1
'tryer'fromlinux_inJapan Level 1 Level 1 (5 points)

I've had a bootable external backup HDD ever since I migrated to Mac from Linux in 2005. I used to use rsyncX, but haven't found a Lion version, so switched to the CLI 'rsync'. If it's relevant, I'm running this on Mac Mini Server of the latest model.

 

The '/usr/bin/rsync' (ver. 2.6.9) that is standard on Mac OS X 10.7.4 runs beautifully for a while - about 21 G's worth, in my case - and then hangs inexplicably at exactly the same point (3 times).

 

Compiling the latest sources and patches -

http://rsync.samba.org/ftp/rsync/src/rsync-3.0.9.tar.gz and

http://rsync.samba.org/ftp/rsync/src/rsync-patches-3.0.9.tar.gz

results in an 'rsync' installed in /usr/local/bin that actually works the way it should and completes the job. (The compiling recipe and other comments at http://www.bombich.com/rsync.html were very useful).

 

To all appearances, the clone is perfect. It boots up fine thanks to 'bless', and everything seems to be where it should be and to work just fine in the limited testing I've done.

 

I am using the 'rsync' --delete flag to remove from the target HDD any files that have been deleted from the source HDD, but the target is nevertheless larger than the source by more than 4G (173.71 vs. 169.44G), as it was on the very first run.

 

I find this puzzling. Surely it can't be the invisible Recovery Partition, can it? What accounts for this extra bulk, though?

 

I was delighted to finally get this working for me, but remain a little uneasy about that extra 4+Gbytes.


  • 1. Re: Rsync Backup Works, but why is the Target larger than the Source?
    'tryer'fromlinux_inJapan Level 1 Level 1 (5 points)

    The NITTY GRITTY

     

    I don't think it's really relevant to the limited question at hand - what explains the extra 4+Gbytes on the 'rsync' backup HDD that should be a perfect mirror of the source? - but I'll post the shell script here that I've adapted very slightly from a script at http://nicolasgallagher.com/mac-osx-bootable-backup-drive-with-rsync/

    There's also some other useful, relevant information there about using 'rsync' for backup.

     

    The single line that calls '/usr/local/bin/rsync' to do the backup is this one:


    sudo $RSYNC -vaxE -S --exclude-from=/Users/whoknows/rsync_excludes.txt "$SRC" "$DST"

     

    (See the full script below for a definition of the variables and rsync flags that are used here).

     

    What I've adapted is a basic script that backs up an entire source filesystem ($SRC), in my case the / (root) file system of my Mini, onto a target backup device ($DST), in my case an old internal HDD in the Dual G5 Tower (that I only boot up these days in Firewire target mode to use as a backup device ;-).

     

    As it I found it, the script was not bad. It has some basic error-handling - the two if...fi conditionals - which is useful if there are access permissions problems with the source or target drive (you need to know that 'logger' puts its messages in the system log, which you can view with the CLI command 'syslog'). And the script is largely self-documenting through the comment lines introduced by # . You need to know also that a construct like if [ ! -r "$SRC" ]; then, is read "if NOT readable source, then", and you can read the whole script with no other prior knowledge (I think).


    You'll need the CLI compiler tools, by the way, and a basic working acquaintance with the Terminal to roll your own 'rsync' from the sources, although you can just cut & paste following the compile recipe linked to in the first post to enter the necessary CLI command lines.

     

    You'll find the necessary compiler tools as a 150M download at https://developer.apple.com/downloads/index.action#, if you're ever looking for them - registration is necessary, I think, but free and simple. OR, your Applestore ID should work just fine, too ;-).

     

    #!/bin/sh

    PROG=$0
    RSYNC="/usr/local/bin/rsync"
    SRC="/"
    DST="/Volumes/QuadMiniBootBack/"

    # rsync options
    # -v increase verbosity
    # -a turns on archive mode (recursive copy + retain attributes)
    # -x don't cross device boundaries (ignore mounted volumes)
    # -E preserve executability
    # -S handle spare files efficiently
    # --delete delete deletes any files that have been deleted locally
    # --exclude-from reference a list of files to exclude

    if [ ! -r "$SRC" ]; then
    /usr/bin/logger -t $PROG "Source $SRC not readable - Cannot start the sync process"
    exit;
    fi

    if [ ! -w "$DST" ]; then
    /usr/bin/logger -t $PROG "Destination $DST not writeable - Cannot start the sync process"
    exit;
    fi

    /usr/bin/logger -t $PROG "Starting rsync..."

    sudo $RSYNC -vaxE -S --exclude-from=/Users/whoknows/rsync_excludes.txt "$SRC" "$DST"

    /usr/bin/logger -t $PROG "End rsync"

    # make the backup bootable
    sudo bless -folder "$DST"/System/Library/CoreServices

    exit 0

      The rsync_excludes.txt that is referenced in the main line of the script just contains a plaintext list of things you don't want to copy over, one per line. I haven't added anything yet to the standard complement of exclusions (some of which are obviously critical):

     

    .Spotlight-*/
    .Trashes
    /tmp/*
    /Network/*
    /cores/*
    /afs/*
    /automount/*
    /private/tmp/*
    /private/var/run/*
    /private/var/spool/postfix/*
    /private/var/vm/*
    /Previous Systems.localized
    /Volumes/*



  • 2. Re: Rsync Backup Works, but why is the Target larger than the Source?
    Gnarlodious Level 4 Level 4 (3,220 points)

    Here is the problem, which may interest you as a Linux user. Linux OS is free, so rsync is included without restriction. Apple is a for-profit corporation, and so rsync can't be included due to the licensing restrictions. This licensing changed a few years ago leaving Apple unable to use newer versions of rsync, so they are stuck using an obsolete version. Newer rsync versions can handle HFS metadata (extended attributes), resource forks and ACLs, however this rsync is not allowed to be included in OSX thanks to the licensing. The older rsync can't even recognize some OSX specific files, so it may copy them but not delete them. These files may not even have a modtime, and so are not able to be compared to the original. In addition, every update of OSX overwrites the newer rsync with the older version, which is maddening.  My solution is to download, build and install the newer rsync which works well and handles all the newer OSX files like ACLs and metadata:

    curl -O http://rsync.samba.org/ftp/rsync/src/rsync-3.0.8.tar.gz
    tar -xzvf rsync-3.0.8.tar.gz
    cd rsync-3.0.8
    ./prepare-source
    ./configure
    make
    sudo make install
    
    sudo rm /usr/bin/rsync
    sudo ln -s /usr/local/bin/rsync /usr/bin/rsync
    

    These last two commands delete the original rsync and symlink the newer one from the original location. Major updates will still overwrite the symlink with an older rsync executable but you can fix it quick without a recompile. There is a way to fix it permanently by changing the command search path but I'm skipping that for now.

     

    I invoke this rsync for my Firewire disk like this:

     

    rsync -aAEXW --progress --delete

     

    This handles extended attributes, shows progress and deletes as expected. Note that files with no modtime are all copied, this can get especially lengthy for music files with a resource fork. You can filter these out from the readout with a sed command, but I'm skipping the extras for now.

  • 3. Re: Rsync Backup Works, but why is the Target larger than the Source?
    Mark Jalbert Level 5 Level 5 (4,475 points)

    What accounts for this extra bulk, though?

    You are not preserving the hfs compression in your rsync command. Look at the options --hfs-compression and --protect-decmpfs.  I'm assuming you are using rsync 3.0.9.

  • 4. Re: Rsync Backup Works, but why is the Target larger than the Source?
    BobHarris Level 6 Level 6 (13,095 points)

    I like to use Carbon Copy Cloner (which is rsync based, and generally includes the lastest rsync that applies to Mac OS X).  CCC can be used to clone a bootable volume, or it can be used to copy a subset of files via a very nice GUI interface.  It can also rsync over the LAN to another Mac.

     

    If you just want the latest Mac OS X specific rsync command, you can generally download CCC, then access the rsync via:

     

    Carbon Copy Cloner.app/Contents/MacOS/ccc_helper.app/Contents/MacOS/rsync

  • 5. Re: Rsync Backup Works, but why is the Target larger than the Source?
    'tryer'fromlinux_inJapan Level 1 Level 1 (5 points)

    Thanks for clearing this up. I need to get up to speed on the HFS+ filesystem that I've been taking for granted all this time.

     

    I've had a good look twice now, and have yet to find any documentation, as such, for the options you mentioned. The manpage that comes with the rsync 3.0.9 sources doesn't appear to mention either option, and I haven't found anything else on the rsync.samba.org site or anywhere else for that matter. Any ideas about this?

  • 6. Re: Rsync Backup Works, but why is the Target larger than the Source?
    Mark Jalbert Level 5 Level 5 (4,475 points)

    From the manpage rsync 3.0.8->

    --hfs-compression
                  This  option  causes rsync to preserve HFS+ compression if the destination filesystem supports it.  If the destination does
                  not support it, rsync will exit with an error.
    
                  Filesystem compression was introduced to HFS+ in Mac OS 10.6. A file that is compressed has  no  data  in  its  data  fork.
                  Rather,  the  compressed data is stored in an extended attribute named com.apple.decmpfs and a file flag is set to indicate
                  that the file is compressed (UF_COMPRESSED). HFS+ decompresses this data "on-the-fly" and presents it to the operating sys-
                  tem  as  a  normal  file.  Normal attempts to copy compressed files (e.g. in the Finder, via cp, ditto, etc.) will copy the
                  file's decompressed contents, remove the UF_COMPRESSED file flag, and discard  the  com.apple.decmpfs  extended  attribute.
                  This  option will preserve the data in the com.apple.decmpfs extended attribute and ignore the synthesized data in the file
                  contents.
    
                  This option implies both --fileflags and (--xattrs).
    
    --protect-decmpfs
                  The com.apple.decmpfs extended attribute is hidden by default from list/get xattr calls, therefore normal attempts to  copy
                  compressed  files will functionally decompress those files. While this is desirable behavior when copying files to filesys-
                  tems that do not support HFS+ compression, it has serious performance and capacity impacts when backing up or restoring the
                  Mac OS X filesystem.
    
                  This  option  will  transfer the com.apple.decmpfs extended attribute regardless of support on the destination. If a source
                  file is compressed and an existing file on the destination is not compressed, the data fork of the destination file will be
                  truncated  and  the com.apple.decmpfs xattr will be transferred instead. Note that compressed files will not be readable to
                  the operating system of the destination if that operating system does not support HFS+ compression. Once restored (with  or
                  without  this  option)  to  an  operating system that supports HFS+ compression, however, these files will be accessible as
                  usual.
    
                  This option implies --fileflags and --xattr
    

    Did you apply any patches when you compiled rsync?

  • 7. Re: Rsync Backup Works, but why is the Target larger than the Source?
    'tryer'fromlinux_inJapan Level 1 Level 1 (5 points)

    Yes, I applied the patches indicated by Bombich at http://www.bombich.com/rsync.html. They are:

     

    http://rsync.samba.org/ftp/rsync/src/rsync-patches-3.0.9.tar.gz


    I don't know why I can't seem to find the documentation that you so kindly provide here. The options you mention are not covered, e.g., in http://rsync.samba.org/ftp/rsync/rsync.html, and I have searched several versions of the rsync manpage, including what downloads with the 3.0.9 distribution. Could this be an indication that I am failing to apply the patches that would update the documentation (or that I have lost what little mind I had left ;-)? Anyhow, I now have the documentation I need to continue experimenting.

     

      It looks like I would only need to add the --protect-decmpfs option to the relevant line of my script above, since I won't ever be reading the target HDD from a device that doesn't support HFS+ compression. Correct me, if I'm wrong.

     

      And thanks again, Mark.

  • 8. Re: Rsync Backup Works, but why is the Target larger than the Source?
    'tryer'fromlinux_inJapan Level 1 Level 1 (5 points)

    So, I've done a bit of experimenting with the HFS+ options for 'rsync' which shows that the HFS+enabling rsync script is making a more exact replica of the source.

     

    I've used a script (see earlier entries for the full script) where this line does the heavy lifting:

     

    sudo $RSYNC -vaxE -S --delete --protect-decmpfs --exclude-from=/Users/denismcm/rsync_excludes.txt "$SRC" "$DST"

     

    I've run this same script with and without the HFS+ option --protect-decmpfs. In this first testing, the script with --protect-decmpfs appears to make a more exact copy of the source.

     

    Run without  --protect-decmpfs (source on left/ target on right). Note the difference in total size of dir and no. of items reported by the Finder:

    Screen Shot 2012-08-07 at 10.57.27 AM.jpg

    Run with --protect-decmpfs (source on left/ target on right). Total size of dir and number of items reported by Finder matches exactly:

    Screen Shot 2012-08-07 at 11.06.35 AM.jpg

     

    This looks good to me, and I'm ready to go ahead and test the full bootable backup script with the HFS+ enhancements added.

  • 9. Re: Rsync Backup Works, but why is the Target larger than the Source?
    'tryer'fromlinux_inJapan Level 1 Level 1 (5 points)

    Repeating the same experiment with a different source, this time /Library, I find the run with --protect-decmpfs turned on is a great deal more faithful to the source than the run without, but it's still not quite a perfect match, and this bothers me a bit. Why is there this discrepancy? Surely, /Library is not evolving so actively that it changes that much in the space of the three-minute run of the script (with only Finder and Terminal, as the only user applications)? Obviously, I'm missing something here.

     

    In the attached graphic, we have the /Library info immediately prior to running the script on the right, the info for the target immediately after the run in the centre, and the info for /Library immediately after the run on the left.

    Screen Shot 2012-08-07 at 3.21.33 PM.jpg

    The number of items for /Library before the run, 30018, matches exactly the number of items for the target after the run, though their sizes in bytes are not precisely the same. After the run, /Library is showing 30,106 items in the Finder, though its size is scarcely more than 1kbyte larger than the other two Info entries.

     

    I'm guessing this that is not the sign of a problem, and it's OK, but I'm not absolutely certain. What do you think?



  • 10. Re: Rsync Backup Works, but why is the Target larger than the Source?
    pmagunia Level 1 Level 1 (0 points)

    Thank for your write-up. I was able to get rsync running without much trouble using your commands.

     

    The only suggestion I have is that I used the fetch option as mentioned on rsync's webpage:

    ./prepare-source fetch

    The error I had gotten was related to make or autoconf I think.

  • 11. Re: Rsync Backup Works, but why is the Target larger than the Source?
    Tony T1 Level 6 Level 6 (8,380 points)

    Just d/l and install, no need to 'fetch'

    Be sure to install the patches for OS X:

     

    Download and unarchive rsync and its patches

              Move patches directory to rsync-3.1.0

              cd rsync-3.1.0

     

    Apply patches relevant to preserving Mac OS X metadata

              patch -p1 <patches/fileflags.diff

              patch -p1 <patches/crtimes.diff

    Apply patch relevant to preserving Mac OS X hfs+compression

              patch -p1 <patches/hfs-compression.diff

     

    Configure, make, install

            ./prepare-source

            ./configure

            make

            sudo make install

     

    Verify your installation

              rsync --version

     

    By default, rsync will be installed in /usr/local/bin. If that isn't 

    in your path, you will need to call your new version of rsync by its 

    absolute path