Currently Being ModeratedAug 4, 2012 8:48 PM (in response to \'tryer\'fromlinux_inJapan)
The NITTY GRITTY
I don't think it's really relevant to the limited question at hand - what explains the extra 4+Gbytes on the 'rsync' backup HDD that should be a perfect mirror of the source? - but I'll post the shell script here that I've adapted very slightly from a script at http://nicolasgallagher.com/mac-osx-bootable-backup-drive-with-rsync/
There's also some other useful, relevant information there about using 'rsync' for backup.
The single line that calls '/usr/local/bin/rsync' to do the backup is this one:
sudo $RSYNC -vaxE -S --exclude-from=/Users/whoknows/rsync_excludes.txt "$SRC" "$DST"
(See the full script below for a definition of the variables and rsync flags that are used here).
What I've adapted is a basic script that backs up an entire source filesystem ($SRC), in my case the / (root) file system of my Mini, onto a target backup device ($DST), in my case an old internal HDD in the Dual G5 Tower (that I only boot up these days in Firewire target mode to use as a backup device ;-).
As it I found it, the script was not bad. It has some basic error-handling - the two if...fi conditionals - which is useful if there are access permissions problems with the source or target drive (you need to know that 'logger' puts its messages in the system log, which you can view with the CLI command 'syslog'). And the script is largely self-documenting through the comment lines introduced by # . You need to know also that a construct like if [ ! -r "$SRC" ]; then, is read "if NOT readable source, then", and you can read the whole script with no other prior knowledge (I think).
You'll need the CLI compiler tools, by the way, and a basic working acquaintance with the Terminal to roll your own 'rsync' from the sources, although you can just cut & paste following the compile recipe linked to in the first post to enter the necessary CLI command lines.
You'll find the necessary compiler tools as a 150M download at https://developer.apple.com/downloads/index.action#, if you're ever looking for them - registration is necessary, I think, but free and simple. OR, your Applestore ID should work just fine, too ;-).
# rsync options
# -v increase verbosity
# -a turns on archive mode (recursive copy + retain attributes)
# -x don't cross device boundaries (ignore mounted volumes)
# -E preserve executability
# -S handle spare files efficiently
# --delete delete deletes any files that have been deleted locally
# --exclude-from reference a list of files to exclude
if [ ! -r "$SRC" ]; then
/usr/bin/logger -t $PROG "Source $SRC not readable - Cannot start the sync process"
if [ ! -w "$DST" ]; then
/usr/bin/logger -t $PROG "Destination $DST not writeable - Cannot start the sync process"
/usr/bin/logger -t $PROG "Starting rsync..."
sudo $RSYNC -vaxE -S --exclude-from=/Users/whoknows/rsync_excludes.txt "$SRC" "$DST"
/usr/bin/logger -t $PROG "End rsync"
# make the backup bootable
sudo bless -folder "$DST"/System/Library/CoreServices
The rsync_excludes.txt that is referenced in the main line of the script just contains a plaintext list of things you don't want to copy over, one per line. I haven't added anything yet to the standard complement of exclusions (some of which are obviously critical):
/Volumes/*Mac mini, Mac OS X (10.7.4), Server
Currently Being ModeratedAug 5, 2012 6:52 AM (in response to \'tryer\'fromlinux_inJapan)
Here is the problem, which may interest you as a Linux user. Linux OS is free, so rsync is included without restriction. Apple is a for-profit corporation, and so rsync can't be included due to the licensing restrictions. This licensing changed a few years ago leaving Apple unable to use newer versions of rsync, so they are stuck using an obsolete version. Newer rsync versions can handle HFS metadata (extended attributes), resource forks and ACLs, however this rsync is not allowed to be included in OSX thanks to the licensing. The older rsync can't even recognize some OSX specific files, so it may copy them but not delete them. These files may not even have a modtime, and so are not able to be compared to the original. In addition, every update of OSX overwrites the newer rsync with the older version, which is maddening. My solution is to download, build and install the newer rsync which works well and handles all the newer OSX files like ACLs and metadata:
These last two commands delete the original rsync and symlink the newer one from the original location. Major updates will still overwrite the symlink with an older rsync executable but you can fix it quick without a recompile. There is a way to fix it permanently by changing the command search path but I'm skipping that for now.
I invoke this rsync for my Firewire disk like this:
rsync -aAEXW --progress --delete
This handles extended attributes, shows progress and deletes as expected. Note that files with no modtime are all copied, this can get especially lengthy for music files with a resource fork. You can filter these out from the readout with a sed command, but I'm skipping the extras for now.
Currently Being ModeratedAug 5, 2012 1:16 PM (in response to \'tryer\'fromlinux_inJapan)
What accounts for this extra bulk, though?
You are not preserving the hfs compression in your rsync command. Look at the options --hfs-compression and --protect-decmpfs. I'm assuming you are using rsync 3.0.9.
Currently Being ModeratedAug 5, 2012 4:26 PM (in response to \'tryer\'fromlinux_inJapan)
I like to use Carbon Copy Cloner (which is rsync based, and generally includes the lastest rsync that applies to Mac OS X). CCC can be used to clone a bootable volume, or it can be used to copy a subset of files via a very nice GUI interface. It can also rsync over the LAN to another Mac.
If you just want the latest Mac OS X specific rsync command, you can generally download CCC, then access the rsync via:
Carbon Copy Cloner.app/Contents/MacOS/ccc_helper.app/Contents/MacOS/rsync
Currently Being ModeratedAug 6, 2012 1:18 PM (in response to Mark Jalbert)
Thanks for clearing this up. I need to get up to speed on the HFS+ filesystem that I've been taking for granted all this time.
I've had a good look twice now, and have yet to find any documentation, as such, for the options you mentioned. The manpage that comes with the rsync 3.0.9 sources doesn't appear to mention either option, and I haven't found anything else on the rsync.samba.org site or anywhere else for that matter. Any ideas about this?
Currently Being ModeratedAug 6, 2012 2:52 PM (in response to \'tryer\'fromlinux_inJapan)
From the manpage rsync 3.0.8->
--hfs-compression This option causes rsync to preserve HFS+ compression if the destination filesystem supports it. If the destination does not support it, rsync will exit with an error. Filesystem compression was introduced to HFS+ in Mac OS 10.6. A file that is compressed has no data in its data fork. Rather, the compressed data is stored in an extended attribute named com.apple.decmpfs and a file flag is set to indicate that the file is compressed (UF_COMPRESSED). HFS+ decompresses this data "on-the-fly" and presents it to the operating sys- tem as a normal file. Normal attempts to copy compressed files (e.g. in the Finder, via cp, ditto, etc.) will copy the file's decompressed contents, remove the UF_COMPRESSED file flag, and discard the com.apple.decmpfs extended attribute. This option will preserve the data in the com.apple.decmpfs extended attribute and ignore the synthesized data in the file contents. This option implies both --fileflags and (--xattrs). --protect-decmpfs The com.apple.decmpfs extended attribute is hidden by default from list/get xattr calls, therefore normal attempts to copy compressed files will functionally decompress those files. While this is desirable behavior when copying files to filesys- tems that do not support HFS+ compression, it has serious performance and capacity impacts when backing up or restoring the Mac OS X filesystem. This option will transfer the com.apple.decmpfs extended attribute regardless of support on the destination. If a source file is compressed and an existing file on the destination is not compressed, the data fork of the destination file will be truncated and the com.apple.decmpfs xattr will be transferred instead. Note that compressed files will not be readable to the operating system of the destination if that operating system does not support HFS+ compression. Once restored (with or without this option) to an operating system that supports HFS+ compression, however, these files will be accessible as usual. This option implies --fileflags and --xattr
Did you apply any patches when you compiled rsync?
Currently Being ModeratedAug 6, 2012 3:41 PM (in response to Mark Jalbert)
Yes, I applied the patches indicated by Bombich at http://www.bombich.com/rsync.html. They are:
I don't know why I can't seem to find the documentation that you so kindly provide here. The options you mention are not covered, e.g., in http://rsync.samba.org/ftp/rsync/rsync.html, and I have searched several versions of the rsync manpage, including what downloads with the 3.0.9 distribution. Could this be an indication that I am failing to apply the patches that would update the documentation (or that I have lost what little mind I had left ;-)? Anyhow, I now have the documentation I need to continue experimenting.
It looks like I would only need to add the
--protect-decmpfsoption to the relevant line of my script above, since I won't ever be reading the target HDD from a device that doesn't support HFS+ compression. Correct me, if I'm wrong.
And thanks again, Mark.
Currently Being ModeratedAug 7, 2012 2:32 PM (in response to Mark Jalbert)
So, I've done a bit of experimenting with the HFS+ options for 'rsync' which shows that the HFS+enabling rsync script is making a more exact replica of the source.
I've used a script (see earlier entries for the full script) where this line does the heavy lifting:
sudo $RSYNC -vaxE -S --delete --protect-decmpfs --exclude-from=/Users/denismcm/rsync_excludes.txt "$SRC" "$DST"
I've run this same script with and without the HFS+ option --protect-decmpfs. In this first testing, the script with --protect-decmpfs appears to make a more exact copy of the source.
Run without --protect-decmpfs (source on left/ target on right). Note the difference in total size of dir and no. of items reported by the Finder:
Run with --protect-decmpfs (source on left/ target on right). Total size of dir and number of items reported by Finder matches exactly:
This looks good to me, and I'm ready to go ahead and test the full bootable backup script with the HFS+ enhancements added.
Currently Being ModeratedAug 7, 2012 3:29 PM (in response to \'tryer\'fromlinux_inJapan)
Repeating the same experiment with a different source, this time /Library, I find the run with --protect-decmpfs turned on is a great deal more faithful to the source than the run without, but it's still not quite a perfect match, and this bothers me a bit. Why is there this discrepancy? Surely, /Library is not evolving so actively that it changes that much in the space of the three-minute run of the script (with only Finder and Terminal, as the only user applications)? Obviously, I'm missing something here.
In the attached graphic, we have the /Library info immediately prior to running the script on the right, the info for the target immediately after the run in the centre, and the info for /Library immediately after the run on the left.
The number of items for /Library before the run, 30018, matches exactly the number of items for the target after the run, though their sizes in bytes are not precisely the same. After the run, /Library is showing 30,106 items in the Finder, though its size is scarcely more than 1kbyte larger than the other two Info entries.
I'm guessing this that is not the sign of a problem, and it's OK, but I'm not absolutely certain. What do you think?