Used space MacOS Extended vs APFS

Some apps might insert invisible data.

For example: I archive images and movies to MacOS Extended HDDs. One HDD is the master and I use Carbon Copy Cloner to clone it to two HDDs with no CCC "Safety Net" option.

A while ago I wondered why the identical clone HDD had different amount of free space and why I lost 30 GB after editing a few movie's metadata on the master HDD with GraphicConverter.

After some sleuthing I could re-gain lost disk space by making .DocumentRevisions-V100 visible in Finder via Shift-Command-. and trash it.

I guess that after a few minutes macOS revisiond should clear such old DocumentRevisions but it very often leaves all or one DocumentRevisions files behind so disk space is lost. This might very well be macOS flaw (I guess I used Sequoia at the time).

.DocumentRevisions-V100 folder is a hidden macOS system directory that stores local version histories, auto-saves, and revert data for apps using AppKit/SwiftUI versioning. Located at the root of APFS/HFS+ volumes, it enables "Browse All Versions" functionality, managed by the revisiond process.

I asked this from GC author and he promptly gave the fix: GC > Settings... > Browser > Cache > Undo: "Use temporary files". It works the same as "Use system file versions" BUT is lot faster with no extra disk activity and no lost disk space.

Those extra invisible files can also deleted via GC > Filesystem Attributes > File Versions > Delete File Versions...

...

p.s. memo from my sleuthing for the terminally interested:

Check how much old DocumentRevisions data is on mounted disk image, after editing 4 x 100 MB movies (somehow revisiond does not seem to clear mounted disk images at all):

sudo du -sh .DocumentRevisions-V100

390M .DocumentRevisions-V100

That pesky .DocumentRevisions-V100 folder can be deleted on mounted disk image's root folder with:

sudo rm -r .DocumentRevisions-V100

Somehow that Terminal command is not permitted on real external data disks while via Finder that folder can be easily deleted.

Those old DocumentRevisions files are stored in the following (numbered) folder(s):

.DocumentRevisions-V100/AllUIDs/(number)/com.apple.documentVersions/0F9EAE5D-3BAF-429E-887F-1A5F282353D1.mp4

On System volumes .DocumentRevisions-V100 is on the Data volume. But its contents or deletion do not seem to be possible even with sudo commands.

...

cd /Volumes/Untitled

sudo ls -laR

-rw-r--r--@ 1 _unknown _unknown 100330021 Aug 5 2020 2020-0805-1101-24.m4v

./.DocumentRevisions-V100/AllUIDs/6/com.apple.documentVersions:

total 0

d--x--x--x 3 _unknown _unknown 96 Feb 16 22:13 .

d--x--x--x 3 _unknown _unknown 96 Feb 16 22:13 ..

-r--r--r--@ 1 _unknown _unknown 100330025 Aug 5 2020 E1C61AE6-6BD7-4007-98AD-548A3B2B57CB.m4v

.DocumentRevisions-V100/AllUIDs/*/com.apple.documentVersions/0F9EAE5D-3BAF-429E-887F-1A5F282353D1.m4v

sudo rm -r .DocumentRevisions-V100

Cloudhr wrote:

Now I'm in the situation where the only copies of these files I have is on these two drives, and they're fairly old drives. There are 27 folders at the top level of both drives. I compared the size of each of the 27 folders using get info and of the 27 folders three differ by a total of 13,152 bytes. Most of these files are video and photos so there aren't that many small files.

If these are older drives, then I would highly recommend not doing anything to either drive. Instead purchase a new drive for your new backup copy of these files.

Currently the best case scenario you have two copies of your files here. Since there seems to be a possible discrepancy between these two drives, it is possible the one drive may have items the other one does not have.

Instead of looking just at the amount of used space on the drive (or even within each folder), you should also check how many items are in each folder as well.

If the contents of these two drives are supposed to be identical, then you will likely need to use an app to identify the differences in content between them. I'm not sure of the best way to do this, but others may have some ideas.

One reason why the APFS volume may have more used space is due to hidden APFS snapshots which are usually associated with backup software such as Time Machine, Carbon Copy Cloner, and SuperDuper. Any items which have been deleted (or moved to other media) may remain behind until a backup is successfully made.....even then it could take two days before those snapshots are automatically deleted (or whatever time frame a third party backup app may have configured).

What I want to do is make sure that I have every thing and then, reformat one of them and use Carbon Copy Cloner to back up one drive to the other. I was going to reformat the HFS+ drive to APFS. Any thoughts on that?

CCC is a good app and has been around for several decades from a well respected developer.

How do you know that "the content on both drives is the same"?

Did you back up one to the other in an automated way? Or did you update one from the other manually, relying on your memory to identify the files and folders to copy? (The latter is prone to error.)

APFS does have a clever optimization which usually gets lost in the process of making a backup. If you duplicate a file on an APFS volume, the original file and duplicate file will share space until you change one or the other in some way that makes them logically different. Then APFS will do the actual copy. So if you duplicated a 500 MB file, and didn't change anything, the two files might take only 500 MB (plus a few bytes) on your APFS source disk. Back up that disk, and your backup software won't know anything about the optimization, and now the files will take up 1 GB of space on the backup disk.

But that doesn't seem to be an explanation here, given that you have less space used on the HFS+-formatted drive (421.95 GB) than on the APFS-formatted one (746.38 GB).

I suppose it is possible that if there are a zillion small files, minimum allocation block sizes could explain some of the difference. I looked those up now and found that

• The allocation block size on a HFS+ volume can be as low as 512 bytes (Technical Note TN1150: HFS Plus Volume Format).
• On APFS, the allocation block size is a minimum of 4096 bytes (The Eclectic Light Company – APFS: Containers and volumes).

These are minimums – the actual sizes can be higher.

So let's say that you write a short letter that has exactly 800 bytes worth of content. On a HFS volume, that might translate into two disk blocks containing a total of 1024 characters. On an APFS volume, that might translate into one disk block containing a total of 4096 characters. (I just created a small Web bookmark file on my Mac's APFS disk to test – and Get Info said that the file had "Size: 767 (8 KB on disk". So there was 767 bytes of content, and 7425 bytes of padding.)

Since you mention Carbon Copy Cloner … there are apparently tools in Carbon Copy Cloner for non-destructively comparing two volumes. There is a Compare window, and there is also the ability to do a Preview of a proposed backup without actually making changes to the destination.

Bombich Support – Carbon Copy Cloner – Comparing the source and destination

Bombich Support – Carbon Copy Cloner – Preview: See what changes CCC will make to the destination

Another possibility:

You say that you identified three top-level folders where there are differences. If the contents of these folders are small enough, you could make a 28th top-level folder on one drive (e.g., "Folders Whose Sizes Differed") and then copy in the three folders from the other drive. Then you'd have both versions of those three folders on one drive - and could figure out "duplicates vs. non-duplicates" within those three folders later.

Cloudhr wrote:

HWTech wrote:

One reason why the APFS volume may have more used space is due to hidden APFS snapshots which are usually associated with backup software such as Time Machine, Carbon Copy Cloner, and SuperDuper. Any items which have been deleted (or moved to other media) may remain behind until a backup is successfully made.....even then it could take two days before those snapshots are automatically deleted (or whatever time frame a third party backup app may have configured).

That's the answer. I had used Carbon Copy Cloner to backup one drive to the other and I forgot that I had ever done such a thing. There were four snapshots.

I'm glad it was that simple. Thanks for the update.

If I used Carbon Copy Cloner to backup the contents on the APFS volume would it also copy the hidden snapshots?

No, I would not expect it to copy the snapshots.

Matti Haveri wrote:

Some apps might insert invisible data.

I'm glad you brought this up. I had toyed with mentioning it myself. Nice to have a real world example too.