Looks like no one’s replied in a while. To start the conversation again, simply ask a new question.

Rsync Backup Works, but why is the Target larger than the Source?

I've had a bootable external backup HDD ever since I migrated to Mac from Linux in 2005. I used to use rsyncX, but haven't found a Lion version, so switched to the CLI 'rsync'. If it's relevant, I'm running this on Mac Mini Server of the latest model.


The '/usr/bin/rsync' (ver. 2.6.9) that is standard on Mac OS X 10.7.4 runs beautifully for a while - about 21 G's worth, in my case - and then hangs inexplicably at exactly the same point (3 times).


Compiling the latest sources and patches -

http://rsync.samba.org/ftp/rsync/src/rsync-3.0.9.tar.gz and

http://rsync.samba.org/ftp/rsync/src/rsync-patches-3.0.9.tar.gz

results in an 'rsync' installed in /usr/local/bin that actually works the way it should and completes the job. (The compiling recipe and other comments at http://www.bombich.com/rsync.html were very useful).


To all appearances, the clone is perfect. It boots up fine thanks to 'bless', and everything seems to be where it should be and to work just fine in the limited testing I've done.


I am using the 'rsync' --delete flag to remove from the target HDD any files that have been deleted from the source HDD, but the target is nevertheless larger than the source by more than 4G (173.71 vs. 169.44G), as it was on the very first run.


I find this puzzling. Surely it can't be the invisible Recovery Partition, can it? What accounts for this extra bulk, though?


I was delighted to finally get this working for me, but remain a little uneasy about that extra 4+Gbytes.

Posted on Aug 4, 2012 6:59 PM

Reply
11 replies

Aug 4, 2012 8:48 PM in response to 'tryer'fromlinux_inJapan

The NITTY GRITTY


I don't think it's really relevant to the limited question at hand - what explains the extra 4+Gbytes on the 'rsync' backup HDD that should be a perfect mirror of the source? - but I'll post the shell script here that I've adapted very slightly from a script at http://nicolasgallagher.com/mac-osx-bootable-backup-drive-with-rsync/

There's also some other useful, relevant information there about using 'rsync' for backup.


The single line that calls '/usr/local/bin/rsync' to do the backup is this one:

sudo $RSYNC -vaxE -S --exclude-from=/Users/whoknows/rsync_excludes.txt "$SRC" "$DST"


(See the full script below for a definition of the variables and rsync flags that are used here).


What I've adapted is a basic script that backs up an entire source filesystem ($SRC), in my case the / (root) file system of my Mini, onto a target backup device ($DST), in my case an old internal HDD in the Dual G5 Tower (that I only boot up these days in Firewire target mode to use as a backup device ;-).


As it I found it, the script was not bad. It has some basic error-handling - the two if...fi conditionals - which is useful if there are access permissions problems with the source or target drive (you need to know that 'logger' puts its messages in the system log, which you can view with the CLI command 'syslog'). And the script is largely self-documenting through the comment lines introduced by # . You need to know also that a construct like if [ ! -r "$SRC" ]; then, is read "if NOT readable source, then", and you can read the whole script with no other prior knowledge (I think).

You'll need the CLI compiler tools, by the way, and a basic working acquaintance with the Terminal to roll your own 'rsync' from the sources, although you can just cut & paste following the compile recipe linked to in the first post to enter the necessary CLI command lines.


You'll find the necessary compiler tools as a 150M download at https://developer.apple.com/downloads/index.action#, if you're ever looking for them - registration is necessary, I think, but free and simple. OR, your Applestore ID should work just fine, too ;-).


#!/bin/sh
PROG=$0
RSYNC="/usr/local/bin/rsync"
SRC="/"
DST="/Volumes/QuadMiniBootBack/"
# rsync options
# -v increase verbosity
# -a turns on archive mode (recursive copy + retain attributes)
# -x don't cross device boundaries (ignore mounted volumes)
# -E preserve executability
# -S handle spare files efficiently
# --delete delete deletes any files that have been deleted locally
# --exclude-from reference a list of files to exclude
if [ ! -r "$SRC" ]; then
/usr/bin/logger -t $PROG "Source $SRC not readable - Cannot start the sync process"
exit;
fi
if [ ! -w "$DST" ]; then
/usr/bin/logger -t $PROG "Destination $DST not writeable - Cannot start the sync process"
exit;
fi
/usr/bin/logger -t $PROG "Starting rsync..."
sudo $RSYNC -vaxE -S --exclude-from=/Users/whoknows/rsync_excludes.txt "$SRC" "$DST"
/usr/bin/logger -t $PROG "End rsync"
# make the backup bootable
sudo bless -folder "$DST"/System/Library/CoreServices
exit 0

The rsync_excludes.txt that is referenced in the main line of the script just contains a plaintext list of things you don't want to copy over, one per line. I haven't added anything yet to the standard complement of exclusions (some of which are obviously critical):


.Spotlight-*/
.Trashes
/tmp/*
/Network/*
/cores/*
/afs/*
/automount/*
/private/tmp/*
/private/var/run/*
/private/var/spool/postfix/*
/private/var/vm/*
/Previous Systems.localized
/Volumes/*

Aug 5, 2012 6:52 AM in response to 'tryer'fromlinux_inJapan

Here is the problem, which may interest you as a Linux user. Linux OS is free, so rsync is included without restriction. Apple is a for-profit corporation, and so rsync can't be included due to the licensing restrictions. This licensing changed a few years ago leaving Apple unable to use newer versions of rsync, so they are stuck using an obsolete version. Newer rsync versions can handle HFS metadata (extended attributes), resource forks and ACLs, however this rsync is not allowed to be included in OSX thanks to the licensing. The older rsync can't even recognize some OSX specific files, so it may copy them but not delete them. These files may not even have a modtime, and so are not able to be compared to the original. In addition, every update of OSX overwrites the newer rsync with the older version, which is maddening. My solution is to download, build and install the newer rsync which works well and handles all the newer OSX files like ACLs and metadata:

curl -O http://rsync.samba.org/ftp/rsync/src/rsync-3.0.8.tar.gz tar -xzvf rsync-3.0.8.tar.gz cd rsync-3.0.8 ./prepare-source ./configure make sudo make install sudo rm /usr/bin/rsync sudo ln -s /usr/local/bin/rsync /usr/bin/rsync

These last two commands delete the original rsync and symlink the newer one from the original location. Major updates will still overwrite the symlink with an older rsync executable but you can fix it quick without a recompile. There is a way to fix it permanently by changing the command search path but I'm skipping that for now.


I invoke this rsync for my Firewire disk like this:


rsync -aAEXW --progress --delete


This handles extended attributes, shows progress and deletes as expected. Note that files with no modtime are all copied, this can get especially lengthy for music files with a resource fork. You can filter these out from the readout with a sed command, but I'm skipping the extras for now.

Aug 5, 2012 4:26 PM in response to 'tryer'fromlinux_inJapan

I like to use Carbon Copy Cloner (which is rsync based, and generally includes the lastest rsync that applies to Mac OS X). CCC can be used to clone a bootable volume, or it can be used to copy a subset of files via a very nice GUI interface. It can also rsync over the LAN to another Mac.


If you just want the latest Mac OS X specific rsync command, you can generally download CCC, then access the rsync via:


Carbon Copy Cloner.app/Contents/MacOS/ccc_helper.app/Contents/MacOS/rsync

Aug 6, 2012 1:18 PM in response to Mark Jalbert

Thanks for clearing this up. I need to get up to speed on the HFS+ filesystem that I've been taking for granted all this time.


I've had a good look twice now, and have yet to find any documentation, as such, for the options you mentioned. The manpage that comes with the rsync 3.0.9 sources doesn't appear to mention either option, and I haven't found anything else on the rsync.samba.org site or anywhere else for that matter. Any ideas about this?

Aug 6, 2012 2:52 PM in response to 'tryer'fromlinux_inJapan

From the manpage rsync 3.0.8->

--hfs-compression
              This  option  causes rsync to preserve HFS+ compression if the destination filesystem supports it.  If the destination does
              not support it, rsync will exit with an error.

              Filesystem compression was introduced to HFS+ in Mac OS 10.6. A file that is compressed has  no  data  in  its  data  fork.
              Rather,  the  compressed data is stored in an extended attribute named com.apple.decmpfs and a file flag is set to indicate
              that the file is compressed (UF_COMPRESSED). HFS+ decompresses this data "on-the-fly" and presents it to the operating sys-
              tem  as  a  normal  file.  Normal attempts to copy compressed files (e.g. in the Finder, via cp, ditto, etc.) will copy the
              file's decompressed contents, remove the UF_COMPRESSED file flag, and discard  the  com.apple.decmpfs  extended  attribute.
              This  option will preserve the data in the com.apple.decmpfs extended attribute and ignore the synthesized data in the file
              contents.

              This option implies both --fileflags and (--xattrs).

--protect-decmpfs
              The com.apple.decmpfs extended attribute is hidden by default from list/get xattr calls, therefore normal attempts to  copy
              compressed  files will functionally decompress those files. While this is desirable behavior when copying files to filesys-
              tems that do not support HFS+ compression, it has serious performance and capacity impacts when backing up or restoring the
              Mac OS X filesystem.

              This  option  will  transfer the com.apple.decmpfs extended attribute regardless of support on the destination. If a source
              file is compressed and an existing file on the destination is not compressed, the data fork of the destination file will be
              truncated  and  the com.apple.decmpfs xattr will be transferred instead. Note that compressed files will not be readable to
              the operating system of the destination if that operating system does not support HFS+ compression. Once restored (with  or
              without  this  option)  to  an  operating system that supports HFS+ compression, however, these files will be accessible as
              usual.

              This option implies --fileflags and --xattr

Did you apply any patches when you compiled rsync?

Aug 6, 2012 3:41 PM in response to Mark Jalbert

Yes, I applied the patches indicated by Bombich at http://www.bombich.com/rsync.html. They are:


http://rsync.samba.org/ftp/rsync/src/rsync-patches-3.0.9.tar.gz

I don't know why I can't seem to find the documentation that you so kindly provide here. The options you mention are not covered, e.g., in http://rsync.samba.org/ftp/rsync/rsync.html, and I have searched several versions of the rsync manpage, including what downloads with the 3.0.9 distribution. Could this be an indication that I am failing to apply the patches that would update the documentation (or that I have lost what little mind I had left ;-)? Anyhow, I now have the documentation I need to continue experimenting.


It looks like I would only need to add the

--protect-decmpfs
option to the relevant line of my script above, since I won't ever be reading the target HDD from a device that doesn't support HFS+ compression. Correct me, if I'm wrong.


And thanks again, Mark.

Aug 7, 2012 2:32 PM in response to Mark Jalbert

So, I've done a bit of experimenting with the HFS+ options for 'rsync' which shows that the HFS+enabling rsync script is making a more exact replica of the source.


I've used a script (see earlier entries for the full script) where this line does the heavy lifting:


sudo $RSYNC -vaxE -S --delete --protect-decmpfs --exclude-from=/Users/denismcm/rsync_excludes.txt "$SRC" "$DST"


I've run this same script with and without the HFS+ option --protect-decmpfs. In this first testing, the script with --protect-decmpfs appears to make a more exact copy of the source.


Run without --protect-decmpfs (source on left/ target on right). Note the difference in total size of dir and no. of items reported by the Finder:

User uploaded file

Run with --protect-decmpfs (source on left/ target on right). Total size of dir and number of items reported by Finder matches exactly:

User uploaded file


This looks good to me, and I'm ready to go ahead and test the full bootable backup script with the HFS+ enhancements added.

Aug 7, 2012 3:29 PM in response to 'tryer'fromlinux_inJapan

Repeating the same experiment with a different source, this time /Library, I find the run with --protect-decmpfs turned on is a great deal more faithful to the source than the run without, but it's still not quite a perfect match, and this bothers me a bit. Why is there this discrepancy? Surely, /Library is not evolving so actively that it changes that much in the space of the three-minute run of the script (with only Finder and Terminal, as the only user applications)? Obviously, I'm missing something here.


In the attached graphic, we have the /Library info immediately prior to running the script on the right, the info for the target immediately after the run in the centre, and the info for /Library immediately after the run on the left.

User uploaded file

The number of items for /Library before the run, 30018, matches exactly the number of items for the target after the run, though their sizes in bytes are not precisely the same. After the run, /Library is showing 30,106 items in the Finder, though its size is scarcely more than 1kbyte larger than the other two Info entries.


I'm guessing this that is not the sign of a problem, and it's OK, but I'm not absolutely certain. What do you think?

Jun 7, 2014 5:40 AM in response to pmagunia

Just d/l and install, no need to 'fetch'

Be sure to install the patches for OS X:


Download and unarchive rsync and its patches

Move patches directory to rsync-3.1.0

cd rsync-3.1.0


Apply patches relevant to preserving Mac OS X metadata

patch -p1 <patches/fileflags.diff

patch -p1 <patches/crtimes.diff

Apply patch relevant to preserving Mac OS X hfs+compression

patch -p1 <patches/hfs-compression.diff


Configure, make, install

./prepare-source

./configure

make

sudo make install


Verify your installation

rsync --version


By default, rsync will be installed in /usr/local/bin. If that isn't

in your path, you will need to call your new version of rsync by its

absolute path

Rsync Backup Works, but why is the Target larger than the Source?

Welcome to Apple Support Community
A forum where Apple customers help each other with their products. Get started with your Apple ID.