Rsync - it simply ignores the "exclude" command...

Question

Grant Lenahan Author

Level 4

1,488 points

Rsync - it simply ignores the "exclude" command...

nomatter what format i try.

I'm using rsynch to make backups to several remote volumes, mounted to my desktop.

(in case you care, one is a NAS drive; others are big hard drives in an old G4 that is largely a aserver now)

Aside from some odd timestamp issues, it works great.

But i want to exclude several folders and filetypes that are not essential can can literally take 10+ hours (like 100,000 email files).

I have tried the exclude command, using many format variations (as found on various internet sites). I have also tried it as "exclude-from" and created am exclude from types file (excludes.txt).

here's a sample of my latest, but i tried many permutations:

rsync -av -e --exclude-from=excludes.txt /usr/bin/ssh /Users/gfl/Documents/Grants /Volumes/90G_G4_Share/exclude_test/

This syncs to a volume mounted locally, ut physically on an old G4 server.

I also tried:

rsync -av -e --exclude=Mail/ usr/bin/ssh /Users/gfl/Documents/Grants /Volumes/90G_G4_Share/exclude_test/

and

rsync -av -e --exclude 'Mail/' usr/bin/ssh /Users/gfl/Documents/Grants /Volumes/90G_G4_Share/exclude_test/

Suggestions?

Would i be better off using rcp? For the kinds of updates i typically make, i'm not sure the delta algorithm is all that important. The slow part of copying for me is the remote processor (a cheap-$h1t WD SMB NAS drive) -- not the network. Copying to the G4 machine is much, much quicker.

TIA,

Grant

various - MacBookPro 15"; PBG4 12", Mac OS X (10.6.7)

Posted on Sep 3, 2011 6:53 AM

Reply

Answer 1

Linc Davis

Level 10

209,739 points

Sep 3, 2011 7:32 AM in response to Grant Lenahan

rsync -av --exclude=Mail/ Documents/Grants remote-host:/Volumes/90G_G4_Share/exclude_test/

That will copy the folder named "Grants" in your Documents folder to a folder with the same name on the volume named "90G_G4_Share" attached to the SSH server with the DNS or Bonjour name remote-host, at the relative path "exclude_test/Grants". The contents of any folder named "Mail" won't be copied. Files that already exist at the destination, but not at the source, will be kept. You must be able to log in to remote-host with the same user name you use on the local host. HFS extended attributes, if any, won't be preserved at the destination.

Would i be better off using rcp?

The syntax of rcp is a lot simpler, but if many files already exist at the destination, it's much less efficient.

Reply

Answer 2

Grant Lenahan Author

Level 4

1,488 points

Sep 3, 2011 8:51 AM in response to Linc Davis

If i understand you, you told me what it *should* do. And what you think it *should* do is what i *want* it to do.

But, as i said, that does NOT happen. It DOES copy the Mail subdirectory. And in the test with the excludes file, it copied all the fieltypes excluded (e.g.: .docx - just as a test).

Let me re-state.

Thsi command works perfectly, except the exclusions are ignored. No error, no mention, they are simply copied part and parcel with everything else.

Grant

Reply

Answer 3

Linc Davis

Level 10

209,739 points

Sep 3, 2011 9:08 AM in response to Grant Lenahan

I tested the command I posted, and it works for me. If the Mail folder is already on the destination, it will of course still be there after the copy, as I wrote above. Delete it and try again.

Reply

Answer 4

Grant Lenahan Author

Level 4

1,488 points

Sep 3, 2011 9:24 AM in response to Linc Davis

I did that. Its not excluding anything.

Please, dont assume i cant see what's happening. Help me find out why its not....

Yes, its weird. If it was clear, i'd just read the man page, follow it, and proceed.

Grant

Reply

Answer 5

Linc Davis

Level 10

209,739 points

Sep 3, 2011 9:43 AM in response to Grant Lenahan

If you're doing everything right, then you may be dealing with an rsync version mismatch. Research that on the rsync mailing lists, or post a query there yourself. You might have to build and install a newer version on the G4.

Reply

Answer 6

Grant Lenahan Author

Level 4

1,488 points

Sep 3, 2011 11:15 AM in response to Linc Davis

Maybe, But the same problem exists regardless of whether i use the command to:

1. the G4

2. the NAS drive (as stated)

3. Another intel laptop

So its can't be a version on the G4. It could be 4 versions on 4 machines, but running as a localvolume, i dont see why the remote machine matters - its a *local* volume.

Grant

Reply

Answer 7

Linc Davis

Level 10

209,739 points

Sep 3, 2011 12:46 PM in response to Grant Lenahan

Please try the following:

mkdir src src/foo src/bar dest

rsync -a --exclude=foo/ src dest/

ls -R dest

The output of the last command should be this:

src

dest/src:

bar

dest/src/bar:

If that's what you get, then the exclude directive is working. You can then run:

rm -r src dest

Reply

Answer 8

BobHarris

Level 9

56,921 points

Sep 3, 2011 9:15 PM in response to Grant Lenahan

Try

--exclude="/**/Mail/"

The quotes are important to protect the asterisks

Reply

Answer 9

Grant Lenahan Author

Level 4

1,488 points

Sep 5, 2011 5:46 AM in response to BobHarris

Hi Bob,

Thanks. I'll try about any syntx combination now! I'll give that a try and report back.

One gneral update. After reading a few responses here, and reading elsewhere, and some experimentation, i tried the following:

rsync -av -e --exclude="Library/Mail/" --exclude="Documents/Microsoft User Data/" /usr/bin/ssh /Users/gfl/ /Volumes/80G_FW/gfl_backup

I also tried (earlier):

rsync -av -e --exclude="*.emlx" --exclude="Documents/Microsoft User Data/" /usr/bin/ssh /Users/gfl/ /Volumes/80G_FW/gfl_backup

Oddly, i'm making progress and that's making me even more confused. In both cases the "Documents/Microsoft UUser Data" was NOT backed up (hooray!), but also in both cases the mail files (all, like 50,000 of them) were.. The syntax of the wo looks identical, so i simply don't get it.

I'll also ask again, if maybe rcp would be a better choice. I dont modify that many files, and my network is fast. Wht i'm finding is that the remote compare -especially on older servers and the dog-slow NAS drive, take forver. Maybe its easier to just copy them- since 99.99% of the files dont change really ever.

TIA,

Grant

Reply

Answer 10

BobHarris

Level 9

56,921 points

Sep 5, 2011 6:19 AM in response to Grant Lenahan

If you want to try rcp, then try it and compare. As Linc has indicated, rcp will just blindly copy everything, so unless you are manually excluding the files you do not want to copy, I do not see how rcp would be faster.

rsync exclude wildcard syntax is non-trivial. I have to read and re-read "man rsync" everytime I want to setup an include or exclude, and then I test it (generally with a small test case) repeatedly until I have it working the way I want, and then I let the script do its job.

2 more things you might consider with respect to rsync. First Carbon Copy Cloner uses rsync as its file copy engine, and because of this the CCC author always includes an up-to-date fully patched Mac OS X copy of rsync as part of the CCC application.

Carbon Copy Cloner.app/Contents/MacOS/ccc_helper.app/Contents/MacOS/rsync

And/Or if you can mount the remove file system, try Carbon Copy Cloner directly to a subdirectory tree, or a disk image container file stored on the mounted volume, as its GUI will allow you to select or exclude the files and subdirectories you want very easily.

Reply

Answer 11

Grant Lenahan Author

Level 4

1,488 points

Sep 5, 2011 6:54 AM in response to BobHarris

Thanks again, Bob. FYI I mistyped your suggested so i dont know if it works or not. Can you explain the significance of:

1. the two **s (sine * is wildcard, isnt ** redundant?)

2. the leading / on a command that is suypposedly entirely relative in its addressing

Otherwise I'm jst parroting your suggestion...

I have and use CCC but i thought i might have more control with rsync.

I suppose i need to read the rcp man pages. I suspected it would have a version of an exclude command or a "differential" command just as xcopy does in dos, and as does rsync. Maybe not. I was trying to avoid processor intensive inspection of the files -a simple data delta would do fine.

I am truly frustrated. You do see that two [apparently] identical commands are performing differently, right?

Grant

Reply

Answer 12

BobHarris

Level 9

56,921 points

Sep 5, 2011 9:15 AM in response to Grant Lenahan

1. the two **s (sine * is wildcard, isnt ** redundant?)

From the rsync man page

INCLUDE/EXCLUDE PATTERN RULES

...

o use '**' to match anything, including slashes.

...

2. the leading / on a command that is suypposedly entirely relative in its addressing

No leading / is relative. Figuring out exactly what rsync is using it relative to, is the hard part :-) Like I said, I have to read and re-read the rsync man page on include/exclude rules and then experiment until I get something that works. After than I "Can" it in a script and leave it alone.

Reply