Skip navigation

Rsync Backup Works, but why is the Target larger than the Source?

8385 Views 9 Replies Latest reply: Aug 7, 2012 3:29 PM by \'tryer\'fromlinux_inJapan RSS
\'tryer\'fromlinux_inJapan Level 1 Level 1 (5 points)
Currently Being Moderated
Aug 4, 2012 6:59 PM

I've had a bootable external backup HDD ever since I migrated to Mac from Linux in 2005. I used to use rsyncX, but haven't found a Lion version, so switched to the CLI 'rsync'. If it's relevant, I'm running this on Mac Mini Server of the latest model.

 

The '/usr/bin/rsync' (ver. 2.6.9) that is standard on Mac OS X 10.7.4 runs beautifully for a while - about 21 G's worth, in my case - and then hangs inexplicably at exactly the same point (3 times).

 

Compiling the latest sources and patches -

http://rsync.samba.org/ftp/rsync/src/rsync-3.0.9.tar.gz and

http://rsync.samba.org/ftp/rsync/src/rsync-patches-3.0.9.tar.gz

results in an 'rsync' installed in /usr/local/bin that actually works the way it should and completes the job. (The compiling recipe and other comments at http://www.bombich.com/rsync.html were very useful).

 

To all appearances, the clone is perfect. It boots up fine thanks to 'bless', and everything seems to be where it should be and to work just fine in the limited testing I've done.

 

I am using the 'rsync' --delete flag to remove from the target HDD any files that have been deleted from the source HDD, but the target is nevertheless larger than the source by more than 4G (173.71 vs. 169.44G), as it was on the very first run.

 

I find this puzzling. Surely it can't be the invisible Recovery Partition, can it? What accounts for this extra bulk, though?

 

I was delighted to finally get this working for me, but remain a little uneasy about that extra 4+Gbytes.


  • Gnarlodious Level 4 Level 4 (3,220 points)

    Here is the problem, which may interest you as a Linux user. Linux OS is free, so rsync is included without restriction. Apple is a for-profit corporation, and so rsync can't be included due to the licensing restrictions. This licensing changed a few years ago leaving Apple unable to use newer versions of rsync, so they are stuck using an obsolete version. Newer rsync versions can handle HFS metadata (extended attributes), resource forks and ACLs, however this rsync is not allowed to be included in OSX thanks to the licensing. The older rsync can't even recognize some OSX specific files, so it may copy them but not delete them. These files may not even have a modtime, and so are not able to be compared to the original. In addition, every update of OSX overwrites the newer rsync with the older version, which is maddening.  My solution is to download, build and install the newer rsync which works well and handles all the newer OSX files like ACLs and metadata:

    curl -O http://rsync.samba.org/ftp/rsync/src/rsync-3.0.8.tar.gz
    tar -xzvf rsync-3.0.8.tar.gz
    cd rsync-3.0.8
    ./prepare-source
    ./configure
    make
    sudo make install
    
    sudo rm /usr/bin/rsync
    sudo ln -s /usr/local/bin/rsync /usr/bin/rsync
    

    These last two commands delete the original rsync and symlink the newer one from the original location. Major updates will still overwrite the symlink with an older rsync executable but you can fix it quick without a recompile. There is a way to fix it permanently by changing the command search path but I'm skipping that for now.

     

    I invoke this rsync for my Firewire disk like this:

     

    rsync -aAEXW --progress --delete

     

    This handles extended attributes, shows progress and deletes as expected. Note that files with no modtime are all copied, this can get especially lengthy for music files with a resource fork. You can filter these out from the readout with a sed command, but I'm skipping the extras for now.

  • Mark Jalbert Level 5 Level 5 (4,385 points)

    What accounts for this extra bulk, though?

    You are not preserving the hfs compression in your rsync command. Look at the options --hfs-compression and --protect-decmpfs.  I'm assuming you are using rsync 3.0.9.

  • BobHarris Level 6 Level 6 (12,505 points)

    I like to use Carbon Copy Cloner (which is rsync based, and generally includes the lastest rsync that applies to Mac OS X).  CCC can be used to clone a bootable volume, or it can be used to copy a subset of files via a very nice GUI interface.  It can also rsync over the LAN to another Mac.

     

    If you just want the latest Mac OS X specific rsync command, you can generally download CCC, then access the rsync via:

     

    Carbon Copy Cloner.app/Contents/MacOS/ccc_helper.app/Contents/MacOS/rsync

  • Mark Jalbert Level 5 Level 5 (4,385 points)

    From the manpage rsync 3.0.8->

    --hfs-compression
                  This  option  causes rsync to preserve HFS+ compression if the destination filesystem supports it.  If the destination does
                  not support it, rsync will exit with an error.
    
                  Filesystem compression was introduced to HFS+ in Mac OS 10.6. A file that is compressed has  no  data  in  its  data  fork.
                  Rather,  the  compressed data is stored in an extended attribute named com.apple.decmpfs and a file flag is set to indicate
                  that the file is compressed (UF_COMPRESSED). HFS+ decompresses this data "on-the-fly" and presents it to the operating sys-
                  tem  as  a  normal  file.  Normal attempts to copy compressed files (e.g. in the Finder, via cp, ditto, etc.) will copy the
                  file's decompressed contents, remove the UF_COMPRESSED file flag, and discard  the  com.apple.decmpfs  extended  attribute.
                  This  option will preserve the data in the com.apple.decmpfs extended attribute and ignore the synthesized data in the file
                  contents.
    
                  This option implies both --fileflags and (--xattrs).
    
    --protect-decmpfs
                  The com.apple.decmpfs extended attribute is hidden by default from list/get xattr calls, therefore normal attempts to  copy
                  compressed  files will functionally decompress those files. While this is desirable behavior when copying files to filesys-
                  tems that do not support HFS+ compression, it has serious performance and capacity impacts when backing up or restoring the
                  Mac OS X filesystem.
    
                  This  option  will  transfer the com.apple.decmpfs extended attribute regardless of support on the destination. If a source
                  file is compressed and an existing file on the destination is not compressed, the data fork of the destination file will be
                  truncated  and  the com.apple.decmpfs xattr will be transferred instead. Note that compressed files will not be readable to
                  the operating system of the destination if that operating system does not support HFS+ compression. Once restored (with  or
                  without  this  option)  to  an  operating system that supports HFS+ compression, however, these files will be accessible as
                  usual.
    
                  This option implies --fileflags and --xattr
    

    Did you apply any patches when you compiled rsync?

Actions

More Like This

  • Retrieving data ...

Bookmarked By (1)

Legend

  • This solved my question - 10 points
  • This helped me - 5 points
This site contains user submitted content, comments and opinions and is for informational purposes only. Apple disclaims any and all liability for the acts, omissions and conduct of any third parties in connection with or related to your use of the site. All postings and use of the content on this site are subject to the Apple Support Communities Terms of Use.