My Macbook Air 2020, now running Sequoia 15.3.1, is at this moment trying to copy a folder sized about 800Mb and containing about 27.000 files. It has been busy for more than an hour now and the end is still not near. Why does it take this long?
The problem too occurs after having booted in Safe Mode. it also occurs when copying to a USB stick as well when copying to a NAS.
MacBook Air, macOS 15.3
Think of it like multiple cars at a drag racing strip. Each vehicle is a file and to get from point A to point B it needs to accelerate but it may reach the end before it gets to full speed. Copying one file at a time will start accelerate then stop and the next file will start, accelerate and then stop.
One way to speed it up is to compress all the small files into one whopping big file and then transfer the big file and decompress on the other side. it will accelerate to full speed and maintain that speed till it finishes. While small files start and stop start and stop and ultimately end up copying much slower as a result.
A nifty way to do all that at once is to use the command line tools where you can tar compress a bunch of small files then pipe it to a network port using netcat and untar it on the other side. Using netcat eliminates any sort of filesystem or file transfer protocol. It just opens a network port and listens while the other system connects and transfers data as fast as possible. I've done this with Linux as well, boot an old PC with Vista on it with the thumb drive and I was able to netcat copy all the data off the PC without Windows running and it went so fast the fans kicked in full blast. Linux has little in the way of restrictions on these tools. They work a bit better in Linux than macOS but they do indeed work very well on macOS.
The running joke what is the fastest way to move 500TB's from New York City to Los Angeles. The answer is FedEx shipping disks.
Copying thousands of small files takes a lot longer than copying one big file, that much is a given; and copying to inherently slower media like USB thumb drives compounds this. If you were copying to the same or another directly attached SSD it would not take nearly as long.
Maybe that's normal, although more than an hour and still a long way off seems too much.
Hello Luis,
Would it not have to be much faster when copying to my NAS? Also, other systems perform better. When copying between that same NAS and the same USB stick when it is attached to my PC running Linux, it takes no longer than 5 minutes and that in both directions.
Regards,
Ed
Hello James.
Indeed copying the tarbal will go faster. But it is still remarkable that the copying process of the same folder between a NAS and the USB stick on a PC running linux takes only 5 minutes. Why does Sequoia perform so bad?
Regards,
Ed
Question: is there a way to quote answers? Now I have to explicitly mention your name to make clear that I am reacting to your post and not that of Luis.
James Brickley wrote:
Computer A
tar -czf ~/Documents | nc 192.168.1.100 2999
Computer B
nc -lp 2999 | tar -xzf ~/Documents
Computer B in my case is my Synology NAS (residing on fixed IP 192.168.178.2). It means I should login to my NAS and somehow get into its shell to enable me to give that second command?
The copying of the 800Mb folder with 27.000 files to a USB stick formatted Mac OS Extended (Journaled) takes about 9 minutes. Deleting it again (using Delete Immediately, so bypassing the Trash Bin) takes longer: about 15 minutes.
Copying a directory of 800Mb with 27.000 files to a NTFS formatted USB stick took 3 hours.
I wonder how long it will take to delete them...
I will give you an answer in the morning.
One detail: I could know that it was about 3 hours because I preceded the bash shell command to do the copy by the command 'date' en following it by that same command. Like this:
date; cp -rpf Library/Application\ Support/Firefox/* /Volumes/Untitled/; date
Now I would expect that this command is still in the history of the bash shell.
But it is not...
When I tried to edit the command to the purpose of deleting files, this editing lead to this command that now is presented in the history of the shell:
date; rm -rf -rpf Library/Application\ Support/Firefox/* /Volumes/Untitled/; date
This command was never issued.
Yet the history claims that it was.
How can this be?
EdVaessen wrote:
Copying a directory of 800Mb with 27.000 files to a NTFS formatted USB stick took 3 hours.
I wonder how long it will take to delete them...
I will give you an answer in the morning.
I don't remember asking, but thank you. :)
Computer A
tar -czf ~/Documents | nc 192.168.1.100 2999
Computer B
nc -lp 2999 | tar -xzf ~/Documents
You can quote all or part of an answer by clicking the quote marks…
If you notice, when you reply the person you replied to appears in your post.
dialabrain wrote:
You can quote all or part of an answer by clicking the quote marks…
Thanks!
Welcome.
Copying a lot of small files takes very long in Sequoia
