Anyone tried using rsync to copy the Library folder to a network storage drive?

My Users/Library is about 125GB. I tried to use Chronosync to sync this over to my NAS MyBookLive. Chronosync ran for nearly 16 hours without success. The copy process was eventually aborted because of errors. Then I tried using rsync to do it. Again, rsync ran for a long time and doesn't seem to be making any progress. I don't have a Time Capsule so I didn't use Time Machine.


I would like to regularly back up the Library folder to my NAS. One of the reasons is the backups of my iOS devices. The backups for my iPhone and iPad are found in library/application support/mobilesync and I would like these backups to be copied to the NAS as a backup in case my MacBook goes bonkers.


Has anyone successfully copied this humongous library folder to a NAS? What method did you use?


Appreciate any advice and help. Thanks!


MacBook Pro with Touch Bar

Posted on Jun 1, 2020 5:39 AM

Reply
7 replies

Jun 2, 2020 7:15 PM in response to BobHarris

I understand that. I'm just wondering why it is so much faster to sync other files to the NAS than it is to sync the Library folder, given that everything else is the same: same MacBook, same NAS, same wifi. I could sync movies and TV series which are quite huge in file size quickly to the NAS. Perhaps it's because the Library folder contains folders and files with special permissions that the rsync or Chronosync does not handle well.


Anyway, I set up a Time Machine share on the NAS and started to use Time Machine to do a backup. I've included system files and applications in the Time Machine settings. It is now backing up some 800GB of data from my MacBook to the NAS and it looks promising. Although it is 18 hours more to go for the first backup to complete, at least the copying process is moving, and it is moving pretty fast compared to rsync or Chronosync. Again, it's the same MacBook, same NAS, and same wifi.


Jun 3, 2020 4:32 PM in response to Christopher Yip

Lots of smaller files would churn the disk between creating metadata and allocating storage.


You would be amazed at home much disk seeking has to happen to create a single file (my day job is working on a commercial Unix based file system for a Fortune 100 company).


I counted the files and folders in my user/Library folder and I have 1,280,980 files (1.2 million files). That is a lot of work you are making the file system do. A single 125GB file is a lot less work.


So lets see. You have to allocate an inode (on-disk structure that keeps track of your file's metadata). Update inode with time stamps, ownership, permissions, initial file allocation. You have to find the directory file. Open the directory file. read the directory file. Find free space to add the file name. If necessary allocate more storage if no existing free space in the directory. put the file name and the inode pointer in the directory. Write the updated directory file.


Now you can start writing the file to disk. Along the way, you may need to allocate more storage. Back to the inode. look on the free block list for free storage. Remove the free blocks from the list (updating free block tracking information), add the free blocks to the inode (updating inode information). OK continue writing to the file.


The good news is that often times extending a file can be efficient if the file system looks at how large the file is when allocating and allocates proportionally large new addition. That is to day if you are creating a small file, it is unwise to pre-allocate a megabyte. But if you have a file that is already several megabytes in size, and you are adding to it, it might be a good idea to pre-allocate the next megabyte.


Some copy utilities will telegraph to the file system how big the file is going to ultimately be, and the file system can preallocate the entire file. But not all copy utilities do this, or in the case of a network attached file system, the protocol does not always allow this information to be sent back along the wire (no protocol message that says file will be this big).


So I do not think you can compare copying your user/Library folder to copying a few large movies and image files.

Jun 3, 2020 7:10 PM in response to BobHarris

Thanks Bob. I figured it was as complicated as you have explained. In any case, the Time Machine backup is making good progress. It has backed up about 650GB of the 810GB of data it is backing up in full. By the end of the day, it should all be backed up nicely.


Since I cannot access the individual folders and files in the Time Machine backup remotely, and it seems quite complicated to try and restore a broken MacBook from a Time Machine backup in a NAS over wifi, what I'm going to do is to get an external SSD hard disk to store the Time Machine backup instead. It would likely be much faster than backing up over wifi to the NAS. I think it would also be easier to restore my MacBook from such an external hard disk rather than via the NAS over the network.


I didn't want to do this previously because the external hard disk would not be able to store other folders and files once it is used for Time Machine backup. I'm going to see if I could partition this SSD hard disk so that I could use one partition for the Time Machine backup, and the other partition for my other folders and files and stuff that I want to keep.


Jun 1, 2020 9:14 PM in response to BobHarris

Thanks Bob. I also used RsyncGUI and RsyncOSX to try and back up the Library folder but it took very long and looked like it hanged after some time.


I'm using Time Machine to do a backup of my Mac to MyBookLive, but what I need is to have my folders and files from my Mac stored as folders and files in the NAS so that I can access them on my iOS devices everywhere I go. My private cloud so to speak. So the Time Machine backups don't do that as the backup is an image file. Chronosync and Rsync can do that.


I'll check out CCC to see if it can back up folders and files to MyBookLive. The issue is not that Chronosync and Rsync can't back up folders and files; the issue is the speed and time to backup large folders such as the Library folder.

This thread has been closed by the system or the community team. You may vote for any posts you find helpful, or search the Community for additional answers.

Anyone tried using rsync to copy the Library folder to a network storage drive?

Welcome to Apple Support Community
A forum where Apple customers help each other with their products. Get started with your Apple Account.