10.6.3 rsync to external drive always re-copies certain files

Question

Level 1

8 points

10.6.3 rsync to external drive always re-copies certain files

I have been using this simple rsync script to backup files for a long time in linux and also cygwin under Windows. I recently migrated it to my new mac. It should only copy files from the source which are newer than the destination or have changed:

+rsync -ruvt /Users/sj/Pictures/ /Users/sj/backup/backup/files/p+
+(other similar lines for different directories removed)+

It works as described above for most files, but there are a number of files that it insists on copying every time I run the command, despite an exact copy already existing at the destination (from the previous run of the command). The weirdest thing about it is it doesn't do this for every file in the Pictures folder. They were all downloaded from the same camera at the same time, but as rsync is re-copying most of the files it will occasionally skip 2 or 3, so for example: img99, img100, img103, img104, etc. (skipped 101 and 102). In fact it should skip all of them.

I've tried tweaking the rsync switches according to the man page but set them back to my originals as they had no effect. I've tried googling this to no avail.

I should mention that the destination folder is a truecrypt mounted external drive formatted as FAT32.

Appreciate any suggestions.

Macbook Pro, Mac OS X (10.6.3)

Posted on May 22, 2010 3:43 PM

Reply

Answer 1

BobHarris

Level 9

53,255 points

May 22, 2010 6:14 PM in response to singaporeJJ

According to Google, FAT32 uses a 16-bit time field and has a resolution of 2 seconds.

Mac OS X has a resolution of at least 1 second, and possibily even finer granularity depending on the HZ clock frequence used by the kernel for managing the clock, where HZ=100 would be 10 milliseconds resolution


sysctl kern.clockrate

So it is very possible that img99, img100, img103, img104 all have odd time values, and when they are rounded down on the FAT32 file system, they always look older than the originals on Mac OS X.

It is a theory.

Have you experimented using a say another directory on Mac OS X to see if going HFS+ to HFS+ works as expected? Have you tried using a Linux ext2/ext3, etc... file system (which should have at least 1 second precision as well).

OK, using Disk Utility, I've created a FAT32 disk image, and copied a file with a modification time containing an odd number of seconds. Then I compaired the original time against the FAT32 time


stat tmp.1
234881026 61128371 -rw-r--r-- 1 harris harris 0 43 
"May 22 21:07:47 2010" 
"May 22 21:07:27 2010" <<<--- Modified time
"May 22 21:07:27 2010" 
"Dec 26 21:57:18 2009" 
4096 8 0 tmp.1
stat /Volumes/FAT32/tmp.1
234881031 10 -rwxrwxrwx 1 harris harris 0 43
"May 22 00:00:00 2010" 
"May 22 21:07:26 2010" <<<--- Modified time
"May 22 21:07:26 2010" 
"May 22 21:07:46 2010" 
131072 4 0 /Volumes/FAT32/tmp.1

Notice the modify time on the original has 27 seconds, but the FAT32 copy of the file has 26.

Message was edited by: BobHarris

Reply

Answer 2

Tony T1

Level 6

10,232 points

May 22, 2010 6:38 PM in response to BobHarris

So it is very possible that img99, img100, img103, img104 all have odd time values, and when
they are rounded down on the FAT32 file system, they always look older than the originals on Mac
OS X.

That's it, and the solution is --modify-window=1

rsync man page:
BUGS
When transferring to FAT filesystems rsync may re-sync unmodified
files. See the comments on the --modify-window option.


--modify-window=NUM compare mod-times with reduced accuracy
 When comparing two timestamps, rsync treats the timestamps as
 being equal if they differ by no more than the modify-window
 value. This is normally 0 (for an exact match), but you may
 find it useful to set this to a larger value in some situations.
 In particular, when transferring to or from an MS Windows FAT
 filesystem (which represents times with a 2-second resolution),
 --modify-window=1 is useful (allowing times to differ by up to 1
 second).

see also: "rsync recopies the same files" http://samba.anu.edu.au/rsync/FAQ.html

Tony

Reply

Answer 3

singaporeJJ Author

Level 1

8 points

May 23, 2010 12:47 AM in response to Tony T1

Thanks to both, it was very insightful. It seems to have solved my problem. I found it was still always copying one file. Command-i showed the file was locked, so I unlocked it. I assume this is the fat read-only attribute. Is there a way to remove these from the Terminal prompt such as the msdos attrib command?

Finally, both the command-i and "ls -l" only give me the time to the minute. What do you use to see the time stamp to the second (or milisecond?)

Thanks.

Reply

Answer 4

Tony T1

Level 6

10,232 points

May 23, 2010 7:15 AM in response to singaporeJJ

ls -lT
chflags nouchg <filename>
man ls:


 -T When used with the -l (lowercase letter ``ell'') option, display
 complete time information for the file, including month, day,
 hour, minute, second, and year.

man chgflags:


 Putting the letters ``no'' before or removing the letters ``no'' from a
 keyword causes the flag to be cleared. For example:
 nouchg clear the user immutable flag (owner or super-user only)

Reply

Answer 5

BobHarris

Level 9

53,255 points

May 23, 2010 8:32 AM in response to singaporeJJ

I was using 'stat' (see my example above; also see man stat).

If you have additional attributes on a file, you can use:


ls -leO@

and it will show you all the extra goodies a Mac file can contain. Commands to modify those attributes are chflags, chmod, chown, and xattr.

Reply

Answer 6

Tony T1

Level 6

10,232 points

May 23, 2010 8:40 AM in response to BobHarris

Regarding stat, I find that stat -x gives a better display of the file info.

Reply

Answer 7

May 31, 2010 7:44 AM in response to singaporeJJ

Another thing to keep in mind is that rsync does not handle Mac resource files very well. "Resource files" are invisible accessory files attached to your original that contain metadata such as custom icons, Spotlight info, etc. Resource files have no file modtime, and so rsync cannot evaluate them as a normal file. As a workaround, the rsync in newer versions of OSX copy EVERY resource file which significantly slows down runtime and loads the network. Also, you see every irrelevant file listed in the progress report. I filter out the resource files so I can see what's really being updated. One way to limit the copying of resource files is to use the --size switch, which compares filesize before copying them.

Reply