Finder says FCPX libraries are larger than external hard drive, unable to backup drive!

Hi Tom,

Thanks again, whoops on screenshots, attached are example library specific screenshots and "get info" is the same value as finder list view of libraries.

Having a 12TB hard drive with 11TB used (per earlier screenshot) and having it take up 21TB+ of data on a backup is pretty crazy! Plus it is unreasonable cost wise, especially as this is one source drive of a dozen I need to back up. Perhaps the answer is in your drive format question?

The source drive is APFS and the backup volume is Mac OS Extended (HFS+) so perhaps that's the issue? Maybe a disk image clone is the only way to backup FCPX overblown library estimates without issue? Disk images feel very difficult to deal with, especially when you just need to recover one file, slow access, many reasons to avoid disk image, but grasping at straws here.

Anyhow, appreciate all your time and energy, this is a tricky one that perhaps others can benefit from a solution.

Michael

PS I found this on Carbon Copy Cloner help page, just sent them a query, not sure if there is a solution there:

Disk usage math is not straightforward

Disk usage is not a simple matter of adding the size of every file on a volume. Special filesystem devices (e.g. hard links) have always complicated this math, but more recently Apple has introduced more special filesystem devices that complicate this even further. The cloning feature in Apple's new APFS filesystem can lead to a scenario where it appears that you have more data on the disk than it can possibly contain, and the filesystem snapshots feature can lead to scenarios where disk usage is higher than the total size of the files on that volume. APFS also supports "sparse" files, which consume less space on disk than their file size would suggest. CCC can preserve sparse files between APFS volumes, but HFS+ does not support sparse files, so these files consume more space on an HFS+ formatted backup disk

Carbon Copy Cloner support says hard links are only preserved when going from an APFS volume to another APFS, not from APFS to HFS+ which is my archive drive. Maybe rsync -h can still work? Or I have to invest in more 56TB archive drives, ouch. hmmm

SO helpful should I go rsync direction and wonderful to have someone with your knowledge willing to share.

I think the issue was APFS to HFS+ archive drive, apparently doesn't respect space savings of hard links. I reformatted my archive drive and now running 50TB of archives to see if it works, should know in a couple days.

I tend to break FCPX over and over again with weird project demands, from space, to crazy database demands, to remote collaboration efficiency, all results of a 10 year plus project, with 1000s of hours of footage. Deepest gratitude to EVERYONE for chiming in!

M

Hi Tom,

Thanks again, whoops on screenshots, attached are example library specific screenshots and "get info" is the same value as finder list view of libraries.

Having a 12TB hard drive with 11TB used (per earlier screenshot) and having it take up 21TB+ of data on a backup is pretty crazy! Plus it is unreasonable cost wise, especially as this is one source drive of a dozen I need to back up. Perhaps the answer is in your drive format question?

The source drive is APFS and the backup volume is Mac OS Extended (HFS+) so perhaps that's the issue? Maybe a disk image clone is the only way to backup FCPX overblown library estimates without issue? Disk images feel very difficult to deal with, especially when you just need to recover one file, slow access, many reasons to avoid disk image, but grasping at straws here.

Anyhow, appreciate all your time and energy, this is a tricky one that perhaps others can benefit from a solution.

Michael

PS I found this on Carbon Copy Cloner help page, just sent them a query, not sure if there is a solution there:

Disk usage math is not straightforward

Disk usage is not a simple matter of adding the size of every file on a volume. Special filesystem devices (e.g. hard links) have always complicated this math, but more recently Apple has introduced more special filesystem devices that complicate this even further. The cloning feature in Apple's new APFS filesystem can lead to a scenario where it appears that you have more data on the disk than it can possibly contain, and the filesystem snapshots feature can lead to scenarios where disk usage is higher than the total size of the files on that volume. APFS also supports "sparse" files, which consume less space on disk than their file size would suggest. CCC can preserve sparse files between APFS volumes, but HFS+ does not support sparse files, so these files consume more space on an HFS+ formatted backup disk

BTW upon further research command line rsync -h might preserve hard links and thus space savings:

https://unix.stackexchange.com/questions/44247/how-to-copy-directories-with-preserving-hardlinks

Not able to write the script myself, but I ran one for a client job before, rsync works though less user friendly.

Curious what you tech genius' think, especially if it solves the "double the media issue" when backing up ;)

M

Yep APFS reformat and thanks for the support! -M

Hi Ian,

I am so grateful for all your time and energy, but this issue is not the file size of the library but the Finder's inaccurate file size estimation.

Because the finder is over estimating the FCPX library size, it's making it impossible to archive my work to another drive. I need to protect my 10+ year project from any potential drive failure!

I attached some frame grabs in case that helps ;) Again grateful for you and the community!

M

PS Here's 2x screenshots for more clarification:

Wow Luis, you are still helping everyone! I feel like the first time you helped me was 10 years ago, you're amazing ;) I responded to Ian with more clarification. I've seen this crazy finder overestimation before, I don't understand it.

If I try to copy these libraries to another drive, it's impossible because of the size over estimation by the finder. I do use Carbon Copy Cloner since Time Machine is so impossible to customize etc. Maybe there's a trick in there. I'll reach out to them too but this size overestimation has been around for a while. I'd love to understand how to overcome this strange FCPX/MacOS anomaly that take some much time, energy and resources to address.

Many thanks in advance for being such a generous human!

M

Hi Tom,

Thank you so much for your feedback!

Get info is providing the same library size, WAY beyond the actual drive size (see attached screen grab).

These are ALL proxy workflow drives, ie proxies pointing to original full resolution media on other drives. This overestimation is something that trickles in over time ie it doesn't seem to get large when the libraries were created, so perhaps not an issue with pointers to full size media. The size over estimations slowly builds over time with library use. After 10+ years we are around double the real hard drive size ;(

Perhaps the challenge is that there IS mixing of media between FCPX libraries, ie one library pulling from another to do editing. (keeping in mind this is all from proxies and libraries on the same hard drive) This is workflow is mandatory, as originally I had one large library, but it broke FCPX, it couldn't handle it. If this is the challenge, hopefully it informs a backup solution to overcome this coding issue.

Appreciate the help so I can unravel this strange FCPX/MacOS anomaly and get back to work!

M