File this under Weird: In Monterey copying a file with Finder corrupts the data in the target file

OK, here is what I am seeing. I used QBitTorrent to download the file Only.Murders.in.the.Building.S02E10.1080p.WEB.H264-CAKES[TGx]. The MD5 hash for the only.murders.in.the.building.s02e10.1080p.web.h264-cakes.mkv file is f63c9b25296bdbf4ae92640c5a65456a. If I use Finder to copy it onto my USB SSD drive, the MD5 for that file is d54e47ed0557d5239a3d656dd3831241, so clearly the files are different.

If I use cp or rsync to copy the local file onto my SSD drive it copies correctly with no corruption - they have same MD5 hash as the local file.

I opened a terminal window and use the cp command to create a duplicate copy of the original .mkv file which I called test.mkv, which has - as you would expect - the same MD5 hash as the original. If I copy this file to the SSD drive it copies correctly with no corruption. Let's try something else. I opened the original .mkv file in HexFiend and used the Save As option to duplicate it without editing it. Copying that duplicate file to the SSD works correctly too.

Copying the original .mkv file over to the SSD drive once more using Finder, I get exactly the same hash as the earlier corrupted copy, so it is being repeatably corrupted in exactly the same way. I renamed the original file and then used Finder to copy it and it was corrupted in exactly the same way.

So Finder is corrupting the file during the copy, but what does it actually look like? I wrote a program to compare the two files which produced this:

Size for both files is 1421335900 bytes

Matched 135139328 bytes from 1 to 135139329

Cleared 45056 bytes from 135139329 to 135184385

Matched 851968 bytes from 135184385 to 136036353

Cleared 40960 bytes from 136036353 to 136077313

Matched 82153472 bytes from 136077313 to 218230785

Cleared 4096 bytes from 218230785 to 218234881

Matched 28672 bytes from 218234881 to 218263553

Cleared 16384 bytes from 218263553 to 218279937

Matched 43192320 bytes from 218279937 to 261472257

Cleared 94208 bytes from 261472257 to 261566465

Matched 58478592 bytes from 261566465 to 320045057

Cleared 16384 bytes from 320045057 to 320061441

Matched 42364928 bytes from 320061441 to 362426369

Cleared 40960 bytes from 362426369 to 362467329

Matched 25534464 bytes from 362467329 to 388001793

Cleared 249856 bytes from 388001793 to 388251649

Matched 10924032 bytes from 388251649 to 399175681

Cleared 57344 bytes from 399175681 to 399233025

Matched 624123904 bytes from 399233025 to 1023356929

Cleared 20480 bytes from 1023356929 to 1023377409

Matched 397958492 bytes from 1023377409 to 1421335900 <EOF>

Processed 1421335900 bytes.

The pattern of the corruption is that every so often during the copy, large parts of the data are zeroed out in the target file. The file sizes are identical. But it doesn't happen on every file - just on the original file downloaded by qBitTorrent. You programmers might have noticed some hinky numbers there.

So the symptoms are:

1. Using Finder to copy the file results in a corrupted target file (MD5 mismatch).
2. Source and target file sizes are identical.
3. Large sections of the target file have been zeroed out during the Finder copy.
4. It is repeatable. If you copy the file using Finder once more it is corrupted identically (Same MD5 as the original copied corrupted file).
5. If you duplicate the file (using cp), Finder will copy the duplicated file correctly.
6. If you copy the file in a terminal window using cp or rsync it copies correctly.

If you are seeing this problem please report it to Apple so we can raise awareness of it and get it fixed.

It seems to be a problem with how libtorrent uses memory mapped files in OSX. You can learn more about it over here:

https://github.com/qbittorrent/qBittorrent/issues/16843

I only ran rsync once from a terminal session to see whether it would copy the file from the internal flash storage onto the external SSD correctly. It did, as confirmed by checking the md5 hash on each file.

It's the normal drag and drop file copy from my internal flash drive using Finder that isn't working, either to the SSD or to my server. It doesn't corrupt every single file but seems more likely to corrupt large files.

Copying the file from the server or SSD back to the local flash drive does work properly.

I'll try anything - can you let me know how to disable rsync and why you think it might help.

Thanks - I’m pleased to know someone else has the same problem. I hope it is fixed quickly - it’s pretty lame to have to verify a file copy actually worked. Some things you just have to be able to take for granted.

Hi Stephen,

Can you try something for me to check if it is the same problem I am experiencing?

On your MacBook, choose one of the problem video files and copy it onto a USB drive, then run the cmp against that. If you find the files are different then you now have three video files, one good one on your MacBook and two corrupted video files, one on your USB disk and one on your NAS. What we want to do is to compare the two dud files to see if they have been corrupted the same way. You can compare them using cmp or by generating a hash value for each of the files.

Create an md5 hash file for each video file, say goodmd5.txt, usbmd5.txt and nasmd5.txt and compare them. In my case goodmd5.txt was different from nasmd5.txt, but nasmd5.txt and usbmd5.txt were identical, so finder corrupted the file in exactly the same way whether it was being copied to a local USB drive or across my wireless network.

Let me know what you find.

Even if you are copying the files over ethernet I would check them afterwards to make sure they copied correctly. I noticed the same problem copying flac files from one machine to another, so it isn't just a video file problem, probably a general file copy bug in finder. If it happens over ethernet as well that would be useful to know as I can't test that easily here.

Hi Ku4hx,

Thanks for trying it out. You are braver than me - I would have checked the hash value of your VM files before overwriting an original working file. It is possible for files to be corrupted in ways that don't show up immediately.

It does appear to be an intermittent problem. I have copied multiple GBs of random data and it all worked perfectly. A bit later I downloaded some new TV shows and they were corrupted when I copied them onto my USB SSD drive. As an example, I downloaded the series We Own This City. On my local drive, I created a file shasum.local.txt by running shasum *.mkv > shasum.local.txt. The file looked like this:

dac9fea2e7e307450be3c3f79f4ef773f80eefab  We Own This City - S01E01 - Part One.mkv
7bbcf6cc2fba68ceb0d125b826e74534bfd5a627  We Own This City - S01E02 - Part Two.mkv
9d1ddb2630b055168e10d4975812097592024254  We Own This City - S01E03 - Part Three.mkv
0e8d0bd2b32e95603d46672cca60684318d2ea73  We Own This City - S01E04 - Part Four.mkv
917a3d9d1eb537eeadf3588a36d332ad991fc652  We Own This City - S01E05 - Part Five.mkv
1b8bea38f770e186f87acdcbc3b19bf480a0f09c  We Own This City - S01E06 - Part Six.mkv

Then I dragged the folder from my local machine onto a USB SSD Drive. The file sizes were all the same. I ran the shasum command again which produced the following output:

dac9fea2e7e307450be3c3f79f4ef773f80eefab  We Own This City - S01E01 - Part One.mkv
479a7ffd22e199f8b330d46f89fd63c66d8c7c67  We Own This City - S01E02 - Part Two.mkv
097a7b275e72e8746b2ebb82f2dc15667b7bd58b  We Own This City - S01E03 - Part Three.mkv
fbf5ed7aa0a1e9706c13c88c8e630f0d25e56882  We Own This City - S01E04 - Part Four.mkv
44e542888838632774860cdab9571e19962f44f6  We Own This City - S01E05 - Part Five.mkv
c43a3fdc03aa01cb45ad3404ebcc03bfab7f311d  We Own This City - S01E06 - Part Six.mkv

As you can see from the hash values, the first file is identical but the following files are all different. Looking at the S01E02 files with a binary comparison tool shows that large sections of the source file have been replaced with binary zeroes during the file copy.

Up until position $3296250 the files are identical.

Starting from position $3296250 until $3299FFF all the data was replaced with $00.

The files are the same again until position $DE1B000.

From $DE1B000 until $DE1EFFF all the data was replaced with $00.

The files are the same again until position $FA9E000.

From $FA9E000 until $FAAEFFF all the data was replaced with $00.

The rest of the files matched.

The other episodes were corrupted in a similar way. This doesn't happen if I copy the folder using rsync. The corrupted files might have played back but there would have been glitches where the data was missing, which is why a hash is so useful to detect corruption.

#Blackwhale34, are you using Monterey 12.4?

My MacBook Air internal drive is formatted as APFS. I first noticed the problem after copying files across my wifi network onto my Mac Mini Server's USB attached 40TB drive array which is formatted as APFS. The USB SSD Drive I have been using for testing is formatted as Mac OS Extended (Journaled). The hash values are derived from the file contents and should be identical on any disk format.