rsync works great - the first time

Question

Level 3

650 points

rsync works great - the first time

I have been learning about rsync as a tool to backup my entire drive onto an external drive (and then with the bless command, making it bootable). I finally got it to work! I can boot into External and everything works just like my main drive.

Now the problem. When I do it again, it is supposed to just replace files I've changed and delete files that have been deleted on the source. Instead, the exact same command now makes a whole new copy of the source inside the external drive, inside a /Volumes folder in an External folder.

Here is my rsync command:
sudo /usr/bin/rsync -avSE --delete --stats --exclude-from /Applications/Utilities/rsync_excludes.txt / /Volumes/External/

I've tried it with and without the trailing slash after the destination (External); it doesn't seem to make a difference.

Can anyone tell me why this is happening?

Thanks,
Jim

iBook 600, Mac OS X (10.4.8)

Posted on Oct 22, 2006 7:35 PM

Reply

Answer 1

Oct 23, 2006 6:14 AM in response to Jimbo

Jim,

Two things first...

1) I like to add --dry-run any time I run a command to have it tell me what it will copy. (but not have rsync actually copy anything) - this helps with figuring out how to craft the precise rsync command to do what you want...

2) The trailing slash matters - there is a great description with examples in the rsync man page - just type man rsync and go to the USAGE section for the help. Basically, the trailing slach controls creating a new directory (the problem with the other folder in /Volumes being created)

With that being said, there's a lot to get right (and similarly wrong) with making backups. For example - I'd be dead without symlinks (I don't think your command grabs them correctly, unless the -E flag somehow gets links...) It might be easier to not re-invent the wheel and support one of the great programs that already cracked the backup problem. I like Carbon Copy Cloner by Mike Bombich and have also heard great things about SuperDuper...

Reply

Answer 2

Oct 23, 2006 6:25 AM in response to Jimbo

How is Gunnison? :=)

It sounds like your Volumes folder is renaming your external disk upon mounting. This can happen because you may have a broken alias to your external disk in there. Finder renames duplicately named disks with a -1, -2 suffix, etc. If you find a broken alias to your external disk while it is not mounted, delete it then try again.

If it still doesn't work then post here.

Reply

Answer 3

Oct 23, 2006 8:40 AM in response to Michael Bradshaw

Michael,

Can you please explain the "--dry-run" option? How and where can it be used?

Thanks,

-Adel

Reply

Answer 4

Jimbo Author

Level 3

650 points

Oct 23, 2006 9:06 AM in response to Gnarlodious

Gnarlodius -

I haven't found any disk or volume listed with a -1 or -2 after it, with or without the external disk mounted.
Gunnison is great, but getting a little too cold for my blood!

Michael Bradshaw -

I have gone through the man page extensively, but regret to say I'm not good enough with unix to understand a lot of it. Most of the examples are complex remote network situations, not a simple external drive. But it sounds like you are saying I should be doing it without the trailing slash. I will go back to square 1, reformat the drive and do both backups (the initial one, which has worked fine, and the "update" which should only make changes) that way and see what happens.

The -a option (archive) takes care of the symlinks and a lot of other stuff. It is equivalent to -rlptgoD, meaning it is recursive, copies symlinks as symlinks, and preserves permissions, times, groups, devices and owners.

For general info, the other options are:
-v verbose (lists files to be changed)
-S handles sparse files efficiently (whatever that means!)
-E copies extended attributes, including resource forks
--delete deletes files that have been deleted locally

I started with Carbon Copy Cloner and eventually got a bootable backup with it (which I also got with rsync), but to get syncing functionality you have to install the psync command, which I couldn't get done through the CCC interface as designed. Anyway, it seems like the new rsync has all the functionality of CCC+psync.

Thanks guys!

Reply

Answer 5

Oct 23, 2006 10:54 AM in response to Jimbo

Jim,

OK - here is an example that is more fun...

First I make two directories to play with - then create three empty files in "my" directory:
<pre>mac:~ me$ cd /tmp
mac:/tmp me$ mkdir mike
mac:/tmp me$ mkdir jim
mac:/tmp me$ touch /tmp/mike/a /tmp/mike/b /tmp/mike/c
</pre> Now - the first rsync command without the / after mike will copy mike to jim (i.e. inside folder jim is folder mike and the three files from /tmp/mike are now in /tmp/jim/mike)
<pre>mac:/tmp me$ rsync -avES --delete /tmp/mike /tmp/jim
building file list ... done
mike/
mike/a
mike/b
mike/c

sent 213 bytes received 80 bytes 586.00 bytes/sec
total size is 0 speedup is 0.00
</pre> Lastly - let's rsync again with the / after mike to indicate to copy the files in mike, but not the folder mike (and since we have --delete - it will clean up the dir /tmp/jim/mike and the three files there...)
<pre>mac:/tmp me$ rsync -avES --delete /tmp/mike/ /tmp/jim
building file list ... done
deleting mike/c
deleting mike/b
deleting mike/a
deleting directory mike
./
a
b
c
sent 213 bytes received 80 bytes 586.00 bytes/sec
total size is 0 speedup is 0.00
</pre>

Reply

Answer 6

Oct 23, 2006 10:57 AM in response to Adel Raheem

--dry-run says to preview the changes (and implies that --verbose is on so that you get verbose details on what rsync is considering to do)

No files get copied - just that the two directories are compared and the potential changes are listed.

Reply

Answer 7

Jimbo Author

Level 3

650 points

Oct 23, 2006 12:44 PM in response to Michael Bradshaw

Thanks Mike. I think I understand that example and that it means I need to leave out the trailing slash. I've just done rsync twice that way onto a reformatted external drive ("/Volumes/External").

The first rsync looked fine. It looks like it copied the whole drive over. The second one, however, did just as before, started making a new copy of the drive in a folder called External in /Volumes on the External drive (until I aborted it).

Maybe this is a clue. As I watched the files fly by in Monitor, first it deleted the two files I deleted as a test to give it something to do. Then it listed every single file on my internal drive. Then it started listing them again, but this time with /Volumes/External/ in front. That's when they started showing up in the /Volumes/External folder on the External drive.

??????

Jim

Reply

Answer 8

Jimbo Author

Level 3

650 points

Oct 23, 2006 12:48 PM in response to Jimbo

By the way, that External folder inside /Volumes on the External drive - it is not an alias with a disk symbol, like my internal drive is, it is just a plain folder. Another possible clue.
Jim

Reply

Answer 9

Oct 23, 2006 1:58 PM in response to Jimbo

Jim,

I'm afraid I'm not clear on the situation inside /Volumes - would you mind posting what ls -la /Volumes shows?

Mike

Reply

Answer 10

Jimbo Author

Level 3

650 points

Oct 23, 2006 3:03 PM in response to Michael Bradshaw

Here it is:
[iBook:~] jim% ls -la /Volumes
total 8
drwxrwxrwt 4 root admin 136 Oct 23 10:13 .
drwxrwxr-t 41 root admin 1496 Oct 22 16:41 ..
drwxrwxr-t 41 jim staff 1428 Oct 23 13:16 External
lrwxr-xr-x 1 root admin 1 Oct 22 16:42 iBookHD -> /

Keep in mind there is another /Volumes/External out on the hard drive (I think). I'm not sure if that is what this is reading.

Jim

Reply

Answer 11

Oct 23, 2006 6:35 PM in response to Jimbo

I guess I forgot one important thing. df -g would be needed to depermine if your directory is really a separate disk or not... (Your ls looks fine - so it rules out the multiple mount folder potential problem...)

Reply

Answer 12

Jimbo Author

Level 3

650 points

Oct 23, 2006 8:36 PM in response to Michael Bradshaw

Here's what df -g shows:

[iBook:~] jim% df -g
Filesystem 1G-blocks Used Avail Capacity Mounted on
/dev/disk0s10 27 14 12 54% /
devfs 0 0 0 100% /dev
fdesc 0 0 0 100% /dev
<volfs> 0 0 0 100% /.vol
automount -nsl [138] 0 0 0 100% /Network
automount -fstab [145] 0 0 0 100% /automount/Servers
automount -static [145] 0 0 0 100% /automount/static
/dev/disk1s10 38 14 23 39% /Volumes/External

(Sorry, don't know how to make it monospace)

I figured out how to describe where the 2nd copy of the iBook drive is being written by rsync. It is /Volumes/External/Volumes/External
Unfortunately I deleted the final External folder, so I can't run the ls command on it right now, but I'll run rsync again and report on it tomorrow.

Thanks for the help so far Mike!
Jim

Reply

Answer 13

Jimbo Author

Level 3

650 points

Oct 24, 2006 4:56 AM in response to Michael Bradshaw

I let rsync run and it recreated /Volumes/External/Volumes/External and put a copy of the whole drive in there. Here's some relevant info in case it is informative:

[iBook:~] jim% ls -la /Volumes/External
total 37448
. . . .
drwxrwxrwt 5 jim staff 170 Oct 23 10:13 Volumes
. . . .

[iBook:~] jim% ls -la /Volumes/External/Volumes
total 24
drwxrwxrwt 5 jim staff 170 Oct 23 10:13 .
drwxrwxr-t 41 jim staff 1428 Oct 22 16:41 ..
-rw-rw-rw- 1 jim staff 6148 Oct 23 17:42 .DS_Store
drwxrwxr-t 38 jim staff 1292 Oct 23 13:16 External [contains 14.6 GB]
lrwxrwxrwx 1 jim staff 1 Oct 23 11:47 iBookHD -> /

I'm sure if I ran rsync again I would end up with a 3rd copy of my system on /Volumes/External/Volumes/External/Volumes/External! I just don't understand why.

Reply

Answer 14

Oct 24, 2006 5:18 AM in response to Jimbo

Jim,

Excellent! df -g looks great in terms of the drives having room to copy each other and such...

I'll wait for the report to try to see what rsync needs. In the mean time - everything looks like you are ready for success once you figure out the rsync syntax. ls and df commands are great.

Here is one thing to keep in mind - each actual hard disk gets a /dev/diskXsY assignment by the system. When you did the ls - you showed that /Volumes/External is actually a directory entry and when the external drive is not mounted - anything written there will actually go to / (the internal drive disk0s10)

So your rsync command could in effect copy the drive upon itself (at least until it ran out of space - and hopefully your list of files to exclude from the original post of yours prevents this as well). If you don't exclude /Volumes in your rsync - you could add a parameter
--exclude /Volumes

if you didn't want that in your exclude list... Either way - let us know how your test goes 🙂

Reply

Answer 15

Oct 24, 2006 6:35 AM in response to Michael Bradshaw

Hi Michael,
You fink! I thought of the same thing at about the same time but waited to come into work before I posted. It's funny; naturally I didn't think of it before we were told the exact path but I still didn't think of it until I started thinking about how to test this. Then the usual mental checks kicked in and I immediately knew what had to be wrong.

Technically, you don't have to exclude all of /Volumes. You avoid the recursive loop if you simply exclude /Volumes/External. That way you could back up other volumes if you wanted. Of course that's probably not a good idea since it would pick up removable media. By the way Jim, /Volumes isn't the only place this infinite loop can occur. Did you also exclude /dev?
--
Gary
~~~~
You won't skid if you stay in a rut.
-- Frank Hubbard

Reply