Tahoe does not recognise actual free space as free

anch in nk wrote:

The interesting thing is, however, I now have the same problem with the Mac Studio. Its 4TB drive now has 2.2TB of Thunderbird stuff on it. Thunderbird seems to have a squirrel-like approach to storing stuff for a rainy day (or a cold winter).

IIRC, Thunderbird has a clean up or compact option to help manage locally stored mail. You may need to run this manually from time to time.

So I've offloaded all the Thunderbird profile to an external 6TB WD Red disk and removed it from the Mac's onboard disk.

Guess what? the Mac tells me it has 3.5TB of space taken. Although I have removed 2.2TB.

I think I'm ghoing to have to nuke the Studio as well. OmniDiskSweeper tells me I'm actually using less than one TB of space.

It would help if Apple could sort this out. I would classify this as a bug.

Storage on an APFS volume and later versions of macOS is extremely complicated.

In your case, the most likely suspect is it will take time for the Free space value to increase since the deleted items still remain in the hidden APFS backup snapshots associated with Time Machine and some third party backup software. Usually these hidden APFS snapshots will be automatically deleted in about 24-48 hours after the backups have been successfully transferred to external media. You can view APFS snapshots using the following Apple article:

View APFS snapshots in Disk Utility on Mac - Apple Support

And like others have mentioned, the only useful & accurate storage value is the Free space value which is only shown in Disk Utility and the System Profiler. On macOS, "Available" which is shown everywhere is very misleading is not synonymous with Free. If you "Get Info" on the volume, then you can calculate the Free space:

Free space = Available space - Purgeable space

There may be some other factors in play here too, since like I said, storage is very complicated on recent versions of macOS.

anch in nk wrote:

The interesting thing is, however, I now have the same problem with the Mac Studio. Its 4TB drive now has 2.2TB of Thunderbird stuff on it. Thunderbird seems to have a squirrel-like approach to storing stuff for a rainy day (or a cold winter).

You may want to reexamine the way you use Thunderbird. That sounds like a lot of emails or large attachments or both. But Thunderbird just will do what you tell it to do. If you never delete emails, it will keep them, taking up space. It's not Thunderbird storing things for a rainy day, it's your work process that never deletes things, apparently. I still can't fathom having 2 TB of email/calendar information but that is what you have. You should look into ways to archive older, unneeded stuff off to external storage.

Programs like DaisyDisk do show where ALL your storage is being used, including the snapshots. It will even offer to delete selected snapshots if you want, which you can also do yourself inside Disk Utility.

The snapshots are created as the Mac prepares for future backups of everything inside your Mac. With large virtual machines and 2 TB of Thunderbird files, you will have enormous snapshots created. Normally, snapshots that are deleted eventually get recreated, but I THINK that if you have physically removed those associated files (Thunderbird and VM), the corresponding snapshots might not return.

As etresoft points out, the Mac storage system is not optimized for systems that have a small number of enormous folders, as you do for Thunderbird and for your VMs. The APFS storage and Time Machine processes work well for most "typical" setups but not so well for ones like yours. The "overhead" of trying to prepare for backups can take over large amounts of storage in those situations. If there is a way to move ALL your Thunderbird folders and working files to external storage, I would look into that. But even then, you will face the challenge of backing all those files up somehow; perhaps a simple clone type backup of the external drive would make sense there. People who use large virtual machines usually exclude them from Time Machine backups and simply make manual copies (backups) of those files for the same reasons.

anch in nk wrote:

Bingo!

The tip with the APFS snapshots solved it. I now have 3.5TB free.

The problem with the way MacOS calculates this is that it prevents largish files being copied to the drive if it thinks there isn't enough space, even if there is. The purgeable space was negligible.

Anyway, issue solved. Many thanks!

Alan

FYI, if you delete APFS backup snapshots before they are completely transferred to external media, then you will have weakened the backups....after all what happens when you perform the mass purge & deleted a critical document....you may have lost its most recent backup(s) if they had not been transferred to external media. Plus I believe macOS always keeps one previous backup snapshot around to use it as a reference to stage the next backup which also acts like a quick easy backup access when away from the backup drive....so deleting that snapshot may affect how the next backup performs.

There are some poor design decisions & implementations here, but I can also understand how difficult it is to accurately report the storage information due to the complexity of it all. I saw the discussions on the Linux threads for their own COW file system which also had snapshot & RAID capabilities which were utilized a lot more than Apple leverages APFS, so I was able to get an understanding that showing the storage information was not a simple act.

It does not help that Apple tends to hide things from their users which can make these things even more difficult to troubleshoot & understand especially when people encounter problems.

The biggest warning I can give you is to never let the file system completely run out of Free storage space, or bad things will happen (varies by file system).....and make sure to always have frequent & regular backups if you value your data.

anch in nk wrote:

Guess what? the Mac tells me it has 3.5TB of space taken. Although I have removed 2.2TB.

I think I'm ghoing to have to nuke the Studio as well.

As I said before, all you have to do in these situations is wait. If you've deleted a bunch of files and aren't seeing the storage returned to free space, then you just have to wait a day or so.

As you discovered, there are many, many different causes of these storage problems. This is just one specific annoyance.

Ignore "available storage". It doesn't exist. It's a fantasy. Put your trust in something more concrete and tangible, like unicorns and fairies.

OmniDiskSweeper tells me I'm actually using less than one TB of space.

3rd party storage tools can't inspect local snapshots A tool that understands how APFS works should be able to tell you how much actual free and used space you have. It may not be able to tell you anything about the used space, but it should always be correct. If you're seeing something else, then I recommend a different storage tool.

It would help if Apple could sort this out. I would classify this as a bug.

I don't disagree. But this has been a problem for several years and has only gotten worse.

You are better off using Disk Utilities to measure the Used versus the Free Space

From another contributor @etresoft regarding Free Space and Available Space 

Free vs available disk space huge differe… - Apple Community

Quote >>  “ The "available" storage is the amount of used storage that the operating system could automatically delete if it felt that it was really necessary. The "free" storage is the amount that you can actually use for something.

There are system processes that run in the background and automatically delete some of the "available" storage and convert it to "free".

If you completely run out of storage, then those system processes will try a little harder. When you "delete" files you are just hinting to the operating system that you don't need those files anymore. The operating system will eventually remove them, but on its own schedule.

Certain tools will allow you to force the issue and manually clean up some of this storage and manually delete local snapshots. But that is only temporary. "  << End Quote 

Purgeable Space is controlled by the Operating System and not the user 

Get detailed information about a disk in Disk Utility on Mac

Reducing System/Volume/Data is a common question. 

System Data Volume is where Your User Account Data is located

Being as the System/Volume/Data has a MOUNT POINT of SYSTEM you can not directly reduce the contents

1 -  System data taking too much in MacOS Sono… - Apple Community

2 - Time Machine Local Snapshot won't delete - Apple Community

3 - Over 60% storage blocked by System Data - Apple Community

How to free up ‘System Data’ and other storage on your Mac from a fellow colleague  @ neuroanatomist

Use another application to see where space is being used  Storeograph  on the Apple Apps Store 

View APFS snapshots in Disk Utility on Mac - Apple Support (CA)

anch in nk wrote:

How can I get it to reassess the free space?

Wait. You can't actually delete any files. Only the operating system can do it. The system "deleted" task seems to only run once a day, so you'll have to wait a couple of days before you recover any storage.

The whole issue started when I tried to copy a 200GB VM (VMWare Fusion) from the external storage to the internal. It told me I needed to free up about 51GB first. So I spent the next few hours removing DropBox, OneDrive and all my Documents, Downloads etc. off to the external storage (also 2TB). This amounted to about 1TB but I STILL get the same response.

That means you only ever had 149 GB of free storage to begin with.

Plus, there's no guarantee that this storage you deleted actually amounts to 1 TB. The actual "free" amounts of storage are only calculated at a very low level. The sizes of individual files can sometimes be completely irrelevant. Since you're working with very large VMs, you are more likely than most to experience this problem at its worst.

Don't even bother with that "Storage" feature in System Settings. It will only infuriate you further.

You may be able to force the issue by deleting snapshots using Disk Utility.

anch in nk wrote:

Since experiencing this issue I have restarted every day for the last 4 days. So IK assume MacOLS will have cleared the stuff it wanted to clear. I clear the Recycle Bin several times a day.

Neither restarting nor clearing the recycle bin will have any effect on this problem.

The OmniDiskSweeper utility is interesting. It shows me that I have 634+ GB in my home folder, of which 632.5 GB is in my Library folder.

607 of that is Thunderbird ... aahaa!

A few years ago when people reported this kind of problem, it was always local snapshots. But the past few years, virtually every time someone finds the cause, it's something new.

Thunderbird is the main application I have the MacBook for.

Isn't that just an e-mail and newsgroup reader?

The only other GB entries are for Group Containers, Containers and Caches (16 GB, 1.5 GB and 7 GB respectively).

Be careful deleting any files in any "Library" folder or other hidden folders. Those should be considered private app data. If you're deleting the app itself, then you can delete the associated folder. But if you don't plan to delete the app, then you should try to see if you can use the app itself to clean up. Otherwise, you run a strong risk of corrupting the data in this app.

Form a very similar question

https://discussions.apple.com/thread/256217360?sortBy=oldest_first