ventura rsync functionality to msdos/usb filesystems

Question

Level 1

6 points

ventura rsync functionality to msdos/usb filesystems

I use rsync to maintain 14GB of usb media (ms-dos fat). Often the changes are subtle, so the command typically takes under 30 seconds.

However, since upgrading to Ventura the workflow has broken. Here is a typical example:

rsync -aP --modify-window=1 $HOME/usbfiles/ /Volumes/usbstick/usbfiles/

With Venture, every invocation of the command transfers ALL OF THE FILES. This breaks the functionality of rsync, and changes a 30 second refresh operation to a 30 minute (or greater) file transfer.

Invoking the -i option with rsync shows that a time mismatch triggers a transfer of each file from the source directory. Indeed, after the transfer the files do not reflect their original timestamp, which rsync normally maintains, rather their modify time is set to the transfer time.

I presume this is a bug in the ventura kernel msdos filesystem handling?

As this is very problematic. Is their any short term workaround before a system update can be applied?

MacBook Air 13″, macOS 13.0

Posted on Nov 16, 2022 12:32 PM

Reply

Answer 1

Best reply

Georgalis Author

Level 1

6 points

Nov 28, 2022 12:20 PM in response to jackieson97

Of course, the reason we use msdos filesystem is we must, for compatibility with other usb mass storage devices, so APFS is not an option.

Apple support does not acknowledge this is an issue. I have been instructed to update Ventura via my support request, although I am already at 13.0.1 (latest), and the upgrade to 13.0.0 is what caused the problem in the first place (both stated in my original support request).

Apple support (chat) insists that I receive a call back from senior support staff to resolve the issue, but they are apparently unable able to navigate my automated phone attendant "Please state your name," and they are not leaving messages.

Apple support initially indicated they would update a solution here, although a second chat support rep said this is a public discussion, and actual support would require a call back.

The problem appears to have arisen around bugs in the Ventura kernel handling of dos filesystem timestamps, notably, it is not respecting to file modification time update by rsync. I am able correct the timestamp with the -r switch on the unix touch command. So the only workaround option I have is to wrap rsync with a "manually update timestamps script" that fixes timestamps for every file...

Additional diagnostics

apple-discussion-254383328.txt

Reply

Answer 2

lexOS1

Community Specialist

Nov 18, 2022 6:44 PM in response to Georgalis

Hi Georgalis,

Thank you for participating in the Apple Support Communities. If you are having issues with your Mac, here are a few steps to try to see if they help:

1. Restart your device to see if the issue is still there: Shut down or restart your Mac - Apple Support

2. Confirm that your device is using the most recent software to rule out the older one: Update macOS on Mac - Apple Support

3. Contact Apple Support here if you need further support: Contact - Official Apple Support

Have a great day.

Reply

Answer 3

Georgalis Author

Level 1

6 points

Nov 28, 2022 2:00 PM in response to BobHarris

I use usb compact storage to transfer data to commodity IoT devices, msdos is the only filesystem option.
I did try homebrew, same issue with new rsync, I believe this is Ventura kernel issue.
The --modify-window=1 solves the 2 second granularity issue.
This issue does not generate an error, file modification time becomes file transfer time on the destination.

The consequence is all files are transferred even if the source has not changed, a large partition cannot be maintained, only re-transferred, rsync to msdos is broken.

Reply

Answer 4

Georgalis Author

Level 1

6 points

Nov 19, 2022 11:06 PM in response to lexOS1

lexOS1, my system is all updated, Ventura 13.0.1 now. The problem started with the upgrade to Ventura 13.0.0, rsync to usb/msdos storage no longer preserves timestamps on the destination, so rsync cannot function properly.

I have already created an apple bug report. Per your message I've reached out to Apple Support, and I have a call back arranged from the senior support team.

Reply

Answer 5

Nov 27, 2022 4:53 PM in response to Georgalis

Hey Georgalis! I'm having the same problem, except my USB drive was formatted as exFat. It appears reformatting my external drive to APFS has fixed the issue. Not ideal if you want to use the drive on Windows or an older version of macOS, but that's the closest to a solution I could get. Wish I had figured that out before I re-copied 700GB...

If you really need Windows support, then I think you're out of luck until Apple resolves this :(

Reply

Answer 6

BobHarris

Level 9

53,315 points

Nov 28, 2022 1:43 PM in response to Georgalis

I would try 2 things.

Change the --modify-window=2 as FAT does not have 1 second granularity (as I seem to recall).

Use Homebrew to install a newer version of rsync and try that.

As for using FAT for USB compatibility, that only applies if you are mounting this USB device on Window or a Unix/Linux platform. Otherwise, if the files are ONLY used with macOS, then use Mac OS Extended (Journaled) or APFS as the file format.

Reply

Answer 7

mu-on

Level 1

13 points

Dec 5, 2022 1:40 PM in response to Georgalis

The following is for exfat and may not be of much help to you, for whom ms-dos formatting is mandatory, but I hope it will be of some help.

When rsync to exfat disk, there were cases where the timestamp (mtime) was not touch correctly and file copying due to mismatch occurs in the next rsync, and cases where the timestamp was touched and file copying did not occur due to identification in the next rsync.

I found that the deciding factor in the behavior was the permissions on the original file.

The case permissions 644, mtime is not copied and the next-time rsync is executed. On the other hand, the case permissions 700 (the same perm. as any files on exfat(and ms-dos)), the mtime is copied correctly and subsequent rsyncs would consider them identical and no more copy.

I try perm 700 files rsync to ms-dos test disk, but, unfortunately, not touch mtime correctly(and copy files occur in next-rsync).

Reply

Answer 8

Georgalis Author

Level 1

6 points

Dec 6, 2022 11:45 AM in response to mu-on

mu-on, thanks for this data point. I tried your technique and confirm mode 700 file permissions successfully workaround rsync APFS to VFAT partitions where mode 644 APFS files fail. This technique is only applicable if the filesystem will be read by Linux or Microsoft, not sure if other operating systems can read VFAT.

I also confirmed the mode 700 workaround does not help rsync transfers to ms-dos partitions.

This issue is also reported on the rsync github page https://github.com/WayneD/rsync/issues/412 where I've uploaded cli commands to reproduce the issue https://github.com/WayneD/rsync/files/10126464/apple-discussion-254383328.txt

In those commands it looked like rsync would fail, and work, on alternate invocations. I have since observed that on regular invocations (when rsync is expected to set the timestamp), subsequent rsync invocations would work 50% of the time, per file. Ie, if 10 files were transferred, approximately five of them would be arbitrarily re-transferred, with every subsequent rsync.

Nevertheless, the procedure of "touch -r" from every src file to every dest file, after every rsync, is effective to prevent re-transfer of unchanged files on the next invocation. It is unnecessary re-touch unchanged files, but the touch operation is lightweight so I just do it every time, to workaround the issue.

A ktrace should shed some light on what's going on with the ms-dos arbitrary re-transfers, and the impact of vfat permissions on setting the destination timestamps.

Reply

Answer 9

Jan 11, 2023 9:07 AM in response to BobHarris

I have the same problem, using rsync to a USB stick for the purpose of synchronizing several computers in different locations. I recently acquired a new iMac with Ventura 13.0. Given the rsync command I have been using successfully until now (including on OS 12.5.1) it wants to copy all my files. Fortunately I did a dry run first so stopped the process before it happened.

My USB stick is formatted as Mac OS Extended (Case-sensitive, Journaled)

Reply

Answer 10

Jan 12, 2023 12:02 AM in response to Green0123

Turns out my problem was not rsync or the file system but rather (my lack of understanding of) the migration process in setting up my new computer. I did some things in the wrong order which resulted in the migration assistant changing some dates.

Sorry for the mis-direction.

Reply

Answer 11

Feb 10, 2023 8:03 AM in response to Georgalis

@Georgalis

I have the same root problem. I use rsync to backup to external harddrive formatted as exFat. (exFat so backup drive can be read by all OS)

A workaround, albeit slow, is the -c option to compare files on contents instead of timestamp. If you have a lot of files this is a big performance hit over comparing by timestamp.

$ rsync -avPhc

For 174k files, 46G backup with very little file changes between source and dest, took 1hr 15min.

I have not dived down the hole of recursively touching every file... to compare that approach.

Start 13:45:21 
174,049 files to consider
sent 32.29M bytes  received 1.04M bytes  8.96K bytes/sec
total size is 45.78G  speedup is 1373.43
Complete 14:56:52

Reply

Answer 12

Georgalis Author

Level 1

6 points

Feb 11, 2023 3:55 PM in response to bozemantofushop

This code should speedup your case substantially! Set src and des (full path) as needed..

export src={path} des={path}
rsync -av --delete --modify-window=1 "$src/" "$des/"
( cd "$src/" && find . -type f -exec touch -r \{\} "$des/"\{\} \; )

Reply

Answer 13

Feb 11, 2023 9:32 PM in response to Georgalis

Georgalis nice. Thanks for the touch kungfu and about 13.2. I'm going to save that away just in case it breaks again. I upgraded to 13.2 and first go, no change, but I on second run, it seems to have resolved it's filetime issues. Went from about 1hr with the -c switch, to 3 minutes normal -av mode. So confirming, 13.2 appears to fix the issue with rsync to fat/exfat formatted drives.

Reply