Rsync isn't behaving the way I've come to expect

Question

Level 1

80 points

Rsync isn't behaving the way I've come to expect

I've primarily been a Linux & *BSD user in the past. I've used rsync for years. But with OS X, I'm running into behavior I'm not understanding:

Like many people, I've used rsync to do backups (over a network) of my iTunes 'stuff' to a Linux server at home.

I typically use a command line similar to the following:

rsync -aP ~/Music/iTunes remotehost:/srv/

If I were to do a Linux -> Linux transfer, it's typically very fast - mainly because the data isn't changing, and only new files are transfered.

However, with OS X, it isn't that fast at all - even if the file's MD5sum and SHA1sum are identical on the different hosts (ie. the files in both locations are identical), and only the date or permissions are different, the entire file is still transferred. (ie. the number of bytes sent or received is greater than the size of the file)

Again, with a Linux->Linux transfer, if a large file is identical on both sides (excepting the date stamp), the actual amount of data transferred when using rsync is much smaller than the actual filesize. A multi-GB file could end up sending/receiving a few MB in Linux->Linux rsync, but when I do OS X -> linux, several GB's are sent.

What am I missing? I'd expect rsync to behave more or less the same with plain files and no extended attribues...

Message was edited by: pariah0

MacBook Pro 17", Mac OS X (10.5)

Posted on Nov 28, 2007 12:05 PM

Reply

Answer 1

Tim Semic

Level 1

135 points

Feb 18, 2008 10:56 AM in response to Gnarlodious

Ok,
Trying to paste the script again...

<begin-script>

#!/bin/sh
# The following script rotates out the oldest log file by renaming them.
echo ""
printf %s "Rotating log files:"
cd /var/log
for i in daily-backup.log; do
if [ -f "${i}" ]; then
printf %s " ${i}"
if [ -x /usr/bin/gzip ]; then gzext=".gz"; else gzext=""; fi
if [ -f "${i}.6${gzext}" ]; then mv -f "${i}.6${gzext}" "${i}.7${gzext}"; fi
if [ -f "${i}.5${gzext}" ]; then mv -f "${i}.5${gzext}" "${i}.6${gzext}"; fi
if [ -f "${i}.4${gzext}" ]; then mv -f "${i}.4${gzext}" "${i}.5${gzext}"; fi
if [ -f "${i}.3${gzext}" ]; then mv -f "${i}.3${gzext}" "${i}.4${gzext}"; fi
if [ -f "${i}.2${gzext}" ]; then mv -f "${i}.2${gzext}" "${i}.3${gzext}"; fi
if [ -f "${i}.1${gzext}" ]; then mv -f "${i}.1${gzext}" "${i}.2${gzext}"; fi
if [ -f "${i}.0${gzext}" ]; then mv -f "${i}.0${gzext}" "${i}.1${gzext}"; fi
if [ -f "${i}" ]; then
touch "${i}.$$" && chmod 640 "${i}.$$" && chown root:admin "${i}.$$"
mv -f "${i}" "${i}.0" && mv "${i}.$$" "${i}" && if [ -x /usr/bin/gzip ]; then
gzip -9 "${i}.0"; fi
fi
fi
done
# The following line creates a backup of the data drive onto the backup drive.
rsync -aEv /Volumes/data/ /Volumes/backup >> /var/log/daily-backup.log 2>&1
# Mails a truncated log file wo/forks to the happy admin.
grep -v '._' /var/log/daily-backup.log | mail -s "rsync-backup on MacServe1" me@domain.com

<end-script>

Reply

Answer 2

Feb 18, 2008 1:36 PM in response to Tim Semic

You actually ran the Applescript 3 times, so the clipboard is wrapped in 3 divboxes.

But you get the idea.

Reply

Answer 3

Tim Semic

Level 1

135 points

Feb 18, 2008 4:25 PM in response to Gnarlodious

Yes, thanks for the script. However what I am pasting in is not verbose. You see the html entries, my script does not have those in there. And the 1. in the script are actually comments, pound signs.

I looked at the help for text formatting but it did not show how to enter code. I'll take the liberty to try a couple of things here.

<pre> blah </pre>

and

<code> blah </code>

and

::
blah

Hopefully one of those.

Reply

Answer 4

Tim Semic

Level 1

135 points

Feb 18, 2008 4:30 PM in response to Tim Semic

<pre>#!/bin/sh
# The following script rotates out the oldest log file by renaming them.
echo ""
printf %s "Rotating log files:"
cd /var/log
for i in daily-backup.log; do
if [ -f "${i}" ]; then
printf %s " ${i}"
if [ -x /usr/bin/gzip ]; then gzext=".gz"; else gzext=""; fi
if [ -f "${i}.6${gzext}" ]; then mv -f "${i}.6${gzext}" "${i}.7${gzext}"; fi
if [ -f "${i}.5${gzext}" ]; then mv -f "${i}.5${gzext}" "${i}.6${gzext}"; fi
if [ -f "${i}.4${gzext}" ]; then mv -f "${i}.4${gzext}" "${i}.5${gzext}"; fi
if [ -f "${i}.3${gzext}" ]; then mv -f "${i}.3${gzext}" "${i}.4${gzext}"; fi
if [ -f "${i}.2${gzext}" ]; then mv -f "${i}.2${gzext}" "${i}.3${gzext}"; fi
if [ -f "${i}.1${gzext}" ]; then mv -f "${i}.1${gzext}" "${i}.2${gzext}"; fi
if [ -f "${i}.0${gzext}" ]; then mv -f "${i}.0${gzext}" "${i}.1${gzext}"; fi
if [ -f "${i}" ]; then
touch "${i}.$$" && chmod 640 "${i}.$$" && chown root:admin "${i}.$$"
mv -f "${i}" "${i}.0" && mv "${i}.$$" "${i}" && if [ -x /usr/bin/gzip ]; then
gzip -9 "${i}.0"; fi
fi
fi
done

# The following line creates a backup of the data drive onto the backup drive.
rsync -aEv /Volumes/data/ /Volumes/backup >> /var/log/daily-backup.log 2>&1

# Mails a truncated log file wo/forks to the happy admin.
grep -v '._' /var/log/daily-backup.log | mail -s "rsync-backup on MacServe1" me@domain.com</pre>

Reply

Answer 5

Feb 18, 2008 7:02 PM in response to Tim Semic

To post code, use the word "code" inside two curly braces: "{" and "}". Sort of like this:

{ code }

but without the spaces between the "code" and the braces. And put the same tag at the end.

Something like:

if [ -f "${i}" ]; then
 printf %s " $i"
 if [ -x /usr/bin/gzip ]; then gzext=".gz"; else gzext=""; fi
 if [ -f "${i}.3${gzext}" ]; then mv -f "${i}.3${gzext}" "${i}.4${gzext}"; fi
 if [ -f "${i}.2${gzext}" ]; then mv -f "${i}.2${gzext}" "${i}.3${gzext}"; fi
 if [ -f "${i}.1${gzext}" ]; then mv -f "${i}.1${gzext}" "${i}.2${gzext}"; fi
 if [ -f "${i}.0${gzext}" ]; then mv -f "${i}.0${gzext}" "${i}.1${gzext}"; fi
 if [ -f "${i}" ]; then mv -f "${i}" "${i}.0" && if [ -x /usr/bin/gzip ]; then gzip -9 "${i}.0"; fi; fi
 touch "${i}" && chmod 640 "${i}" && chown root:admin "${i}"
 fi

charlie

Reply

Answer 6

Tim Semic

Level 1

135 points

Feb 19, 2008 9:18 AM in response to Charles Minow

Ok, another attempt to get this script posted:


#!/bin/sh
# The following script rotates out the oldest log file by renaming them.
echo ""
printf %s "Rotating log files:"
cd /var/log
for i in daily-backup.log; do
 if [ -f "${i}" ]; then
 printf %s " ${i}"
 if [ -x /usr/bin/gzip ]; then gzext=".gz"; else gzext=""; fi
 if [ -f "${i}.6${gzext}" ]; then mv -f "${i}.6${gzext}" "${i}.7${gzext}"; fi
 if [ -f "${i}.5${gzext}" ]; then mv -f "${i}.5${gzext}" "${i}.6${gzext}"; fi
 if [ -f "${i}.4${gzext}" ]; then mv -f "${i}.4${gzext}" "${i}.5${gzext}"; fi
 if [ -f "${i}.3${gzext}" ]; then mv -f "${i}.3${gzext}" "${i}.4${gzext}"; fi
 if [ -f "${i}.2${gzext}" ]; then mv -f "${i}.2${gzext}" "${i}.3${gzext}"; fi
 if [ -f "${i}.1${gzext}" ]; then mv -f "${i}.1${gzext}" "${i}.2${gzext}"; fi
 if [ -f "${i}.0${gzext}" ]; then mv -f "${i}.0${gzext}" "${i}.1${gzext}"; fi
 if [ -f "${i}" ]; then
 touch "${i}.$$" && chmod 640 "${i}.$$" && chown root:admin "${i}.$$"
 mv -f "${i}" "${i}.0" && mv "${i}.$$" "${i}" && if [ -x /usr/bin/gzip ]; then
 gzip -9 "${i}.0"; fi
 fi
 fi
done
# The following line creates a backup of the data drive onto the backup drive.
rsync -aEv /Volumes/data/ /Volumes/backup >> /var/log/daily-backup.log 2>&1
# Mails truncated log file to the happy admin.
grep -v '._' /var/log/daily-backup.log | mail -s "rsync-backup on MacServe1" me@domain.com

Reply

Answer 7

Tim Semic

Level 1

135 points

Feb 19, 2008 9:21 AM in response to Tim Semic

Thanks Charles.

The cool thing about that script is that I implemented Apple's rotating log script which can be used for just about anything. Very handy for logging your script results.

Reply