File this under Weird: In Monterey copying a file with Finder corrupts the data in the target file

I have a really odd problem where files are being corrupted when I use Finder to copy them from my MacBook Air to my Mac Mini server. I only noticed this after seeing video glitches when I went to play the files back on my TV. When I compared the local files with their copies on the server, whole sections of data had been replaced with #00 characters.

If I generate an MD5 hash then I can check the copied file against the original, which in this case shows that the source and target files are clearly different. Initially I thought it might be a problem with my wireless network, but what is really weird is that when I copy the same file from the MacBook Air to a locally attached USB SSD drive then it is corrupted in the exact same way - the MD5 for the corrupted file on the SSD is the same value as the corrupted file on my server.

The source computer is an Intel MacBook Air running the most recent Mac OS Monterey 12.3.1. The server is an older Mac Mini running Catalina 10.15.7. I don't think the old software version on the server is a factor as the same corruption happens copying the file to a local USB SSD disk. Rsync copies the file correctly to the USB SSD Drive, so I think it's the Finder drag and drop that's causing the problem. I don't recall seeing this problem before upgrading the MacBook Air to Monterey.

Has anyone seen anything like this before?

Thanks for your help,

Steve

MacBook Air (2020 or later)

Posted on Apr 30, 2022 11:38 PM

Top-ranking reply

Steve_Agnew_NZ Author

Level 1

17 points

Posted on Aug 25, 2022 3:56 PM

OK, here is what I am seeing. I used QBitTorrent to download the file Only.Murders.in.the.Building.S02E10.1080p.WEB.H264-CAKES[TGx]. The MD5 hash for the only.murders.in.the.building.s02e10.1080p.web.h264-cakes.mkv file is f63c9b25296bdbf4ae92640c5a65456a. If I use Finder to copy it onto my USB SSD drive, the MD5 for that file is d54e47ed0557d5239a3d656dd3831241, so clearly the files are different.

If I use cp or rsync to copy the local file onto my SSD drive it copies correctly with no corruption - they have same MD5 hash as the local file.

I opened a terminal window and use the cp command to create a duplicate copy of the original .mkv file which I called test.mkv, which has - as you would expect - the same MD5 hash as the original. If I copy this file to the SSD drive it copies correctly with no corruption. Let's try something else. I opened the original .mkv file in HexFiend and used the Save As option to duplicate it without editing it. Copying that duplicate file to the SSD works correctly too.

Copying the original .mkv file over to the SSD drive once more using Finder, I get exactly the same hash as the earlier corrupted copy, so it is being repeatably corrupted in exactly the same way. I renamed the original file and then used Finder to copy it and it was corrupted in exactly the same way.

So Finder is corrupting the file during the copy, but what does it actually look like? I wrote a program to compare the two files which produced this:

Size for both files is 1421335900 bytes

Matched 135139328 bytes from 1 to 135139329

Cleared 45056 bytes from 135139329 to 135184385

Matched 851968 bytes from 135184385 to 136036353

Cleared 40960 bytes from 136036353 to 136077313

Matched 82153472 bytes from 136077313 to 218230785

Cleared 4096 bytes from 218230785 to 218234881

Matched 28672 bytes from 218234881 to 218263553

Cleared 16384 bytes from 218263553 to 218279937

Matched 43192320 bytes from 218279937 to 261472257

Cleared 94208 bytes from 261472257 to 261566465

Matched 58478592 bytes from 261566465 to 320045057

Cleared 16384 bytes from 320045057 to 320061441

Matched 42364928 bytes from 320061441 to 362426369

Cleared 40960 bytes from 362426369 to 362467329

Matched 25534464 bytes from 362467329 to 388001793

Cleared 249856 bytes from 388001793 to 388251649

Matched 10924032 bytes from 388251649 to 399175681

Cleared 57344 bytes from 399175681 to 399233025

Matched 624123904 bytes from 399233025 to 1023356929

Cleared 20480 bytes from 1023356929 to 1023377409

Matched 397958492 bytes from 1023377409 to 1421335900 <EOF>

Processed 1421335900 bytes.

The pattern of the corruption is that every so often during the copy, large parts of the data are zeroed out in the target file. The file sizes are identical. But it doesn't happen on every file - just on the original file downloaded by qBitTorrent. You programmers might have noticed some hinky numbers there.

So the symptoms are:

Using Finder to copy the file results in a corrupted target file (MD5 mismatch).
Source and target file sizes are identical.
Large sections of the target file have been zeroed out during the Finder copy.
It is repeatable. If you copy the file using Finder once more it is corrupted identically (Same MD5 as the original copied corrupted file).
If you duplicate the file (using cp), Finder will copy the duplicated file correctly.
If you copy the file in a terminal window using cp or rsync it copies correctly.

If you are seeing this problem please report it to Apple so we can raise awareness of it and get it fixed.

View in context

Similar questions

57 replies

DG001

Level 1

8 points

Aug 23, 2022 5:19 AM in response to Brianrh7

Not that I'm aware of. Still using muCommander for transferring over large video files and that's working fine.

Sep 2, 2022 2:39 PM in response to macsnotgnu

It seems to be a problem with how libtorrent uses memory mapped files in OSX. You can learn more about it over here:

https://github.com/qbittorrent/qBittorrent/issues/16843

May 1, 2022 2:42 PM in response to ku4hx

I only ran rsync once from a terminal session to see whether it would copy the file from the internal flash storage onto the external SSD correctly. It did, as confirmed by checking the md5 hash on each file.

It's the normal drag and drop file copy from my internal flash drive using Finder that isn't working, either to the SSD or to my server. It doesn't corrupt every single file but seems more likely to corrupt large files.

Copying the file from the server or SSD back to the local flash drive does work properly.

I'll try anything - can you let me know how to disable rsync and why you think it might help.

ku4hx

Level 7

23,688 points

May 1, 2022 2:52 PM in response to Steve_Agnew_NZ

I know nothing about Rsync. I'd think you'd know since you installed/used it. Contact the developer is all I know to tell you.

As to corrupting large files, I've never seen that and I routinely copied single VM files in the 50 to 100GB range. That problem needs further investigation obviously.

May 1, 2022 10:31 PM in response to AngryPandas

Thanks - I’m pleased to know someone else has the same problem. I hope it is fixed quickly - it’s pretty lame to have to verify a file copy actually worked. Some things you just have to be able to take for granted.

Barney-15E

Level 10

121,988 points

Jun 5, 2022 3:37 PM in response to Steve_Agnew_NZ

Do you have any more ideas on what might trigger it?

I've been trying to replicate the problem and can't.

I've only got one 2.2GB movie and a couple 500MB movies to test.

I've copied them from an MBP to an M1 Mini and a Catalina Mini several times without failure.

The M1 is connected to the network via Ethernet cable and Wi-Fi. The Mini is only Wi-Fi. MBP is Wi-Fi.

Jun 6, 2022 12:38 AM in response to Steve_Agnew_NZ

I've been having this problem recently on my Mac mini, is there a fix? Video files is how I noticed, corrupted or not working when I copy to external drives, plays fine from ssd but after copying won't play or it just stutters randomly

ku4hx

Level 7

23,688 points

Jun 6, 2022 8:27 AM in response to blackwhale34

I just last night copied a 90GB VMWare Fusion Windows VM file from an iMac to a PNY 250GB portable SSD. I then copied it from the PNY to four other Macs and back to the PNY.

That was a total of 9 copy cycles. Did this bingeing Yellowstone

The VM ran from the PNY on every machine that has VMWare Fusion installed on it, and when copied back to the original Mac overwriting the existing file it also ran fine.

I cannot duplicate this problem.

Jun 6, 2022 3:51 PM in response to steve626

My MacBook Air internal drive is formatted as APFS. I first noticed the problem after copying files across my wifi network onto my Mac Mini Server's USB attached 40TB drive array which is formatted as APFS. The USB SSD Drive I have been using for testing is formatted as Mac OS Extended (Journaled). The hash values are derived from the file contents and should be identical on any disk format.

Jun 6, 2022 6:22 PM in response to Steve_Agnew_NZ

This is very similar to how my problem started, was copying files across wifi network from my macbook onto mac mini with attached USB hard drive. Only noticed when videos started randomly skipping or shuttering. Took until recently finding this thread to realize what the problem was. Got a new hard drive and restored the mac mini and nothing had worked. I realized the problem only occured when the file was copied in some way. Funny thing now is it does the same thing on both macs, my macbook running Monterey 12.4 and my mini running some version of Catalina. I upgraded my macbook to monterey within the last few months and that may have been when it started, but I am not sure.

Jul 8, 2022 10:33 PM in response to Steve_Agnew_NZ

Part of what I said above is not actually correct, I am sorry to say, because I was having trouble copying uncorrupted to an external 2.5" SSD, and there should have been no problem there. And I have to say that another thing I said (I cannot now delete that comment) was also incorrect. The file I was able to successfully copy was actually not the same file that was giving me trouble yesterday. So I'll try to figure this thing out by tomorrow and update. Sorry for the confusion.

Jul 8, 2022 10:52 PM in response to macsnotgnu

Sorry to say that the stuff I said about U3 being UHS-III is rubbish too. U3 is the speed class. NOt sure what UHS-class my SD cards are, but they are SAmsung EVO and say XC I on them. Possibly UHS-I, yet I thought that I tried to buy UHS-III. I will try to double check my facts next time before posting.

Jul 24, 2022 4:14 PM in response to Steve_Agnew_NZ

I also have this problem... I am copying a file from an attached SSD (APFS format) using an ethernet connection to a Synology NAS (formatted BRFS). The file is OK on the source SSD, and is somehow corrupted by the finder copying from location to location. This is BASIC stuff, a file copy should work. I am so glad that I'm not crazy and other people are having the same issue. If it helps I am copying large MKV files. The original is playable, the destination copy is broken (using MPV) and of course the checksums are different. It would seem that something as basic as copying a file could be accomplished in ANY OS without corruption (using 12.4 was also happening with 12.3)

Jul 26, 2022 1:31 AM in response to Toby_Manila

So weirdly I was going to test this using a variety of copy techniques, but today the Finder copy worked just fine (Monterey 12.5). Has anyone seen that it's fixed... or maybe I was just lucky this time?

DG001

Level 1

8 points

Jul 26, 2022 6:35 AM in response to Matt Whiting

I'd test it more often but I find it such a pain to make sure Finder copied it perfectly. For me, it only happens with large video files.

Any recommendations for a free app that could compare two files easily? I'm terrible with Terminal commands so any easy GUI recommendations would be great.

My temporary solution above is causing me all sorts of issues. When I downloads files from my Mac directly to the network drive it gets cut off and stalls the computer all the time. Such a pain. I'll stay on the lookout for this fix. Perhaps a better temporary solutions would be to dust off my old PC and use that for downloading files for now...ugh.

This thread has been closed by the system or the community team. You may vote for any posts you find helpful, or search the Community for additional answers.

File this under Weird: In Monterey copying a file with Finder corrupts the data in the target file