Unaccounted for memory usage

Question

Unaccounted for memory usage

For about a week, I was getting warnings that I was running out of hard drive space. Programs wouldn't let me save anything, and it was becoming quite a nuisance. After looking into where the usage was coming from, I could only find an approximate 150gb being used out of 500gb available. It didn't make any sense.

Running the 'du' command on the entire drive returned results that matched my manual calculations. I tried deleting things, but it seemed like my memory was being eaten my something unseen.

I reformatted yesterday, and then re-synced all of my files from the cloud. I now have ~400gb free, whereas yesterday I had only 10gb free, yet absolutely nothing has changed.

Could this problem be caused by a virus? Could it be caused by a malfunctioning OS, or hardware? The problem is solved for the mean time, but I'd love to have more information on why this happened in the first place.

Thank you!

MacBook Pro (Retina, 15-inch, Mid 2014), OS X Yosemite (10.10.5)

Posted on Aug 30, 2015 6:56 PM

Reply

Answer 1

BobHarris

Level 9

53,003 points

Aug 30, 2015 8:08 PM in response to jordancooperman

OmniDiskSweeper (free download)

<http://www.omnigroup.com/more>

When using OmniDiskSweeper, or any utility that shows all your files... See the following article if you want to run it as root

<http://www.macobserver.com/tmo/article/how_to_recover_missing_hard_drive_space>

DO NOT delete files in your Home Folder -> Library tree as there are things like your iPhone backups, your email messages, your application preferences, etc… If you think you have found something in your Home Folder -> Library that can be deleted, you should ask first.

DO NOT delete files outside your home folder, as you may end up deleting something essential to Mac OS X, and turn your Mac into an expensive “Door Stop”.

I will point out that you will find some very large files in private -> var -> vm (these are the Mac OS X virtual memory paging files (swapfiles) and where Mac OS X stores the copy of RAM when your Mac is put to sleep). The swapfiles get deleted on reboot, and the sleep image is just going to be created again when you put your Mac to sleep.

If you think you have found something to delete outside your home folder, it would be best to ask first before deleting. There are many examples of people deleting files outside their home folder, or renaming files, or changing the ownership or file permissions, and then their Mac stops running. Do not be one of those people. Ask first.

Reply

Answer 2

Aug 30, 2015 8:39 PM in response to BobHarris

Well seeing as I've already reformatted, using Omni is no longer useful in understanding the problem, but I used a similar tool, one called disc diag, which didn't show me anything informative.

I also opened the info dialog for each individual directory under my hard drive, and the total file sizes were under half of what 'About this Mac' -> 'Storage' was showing. running the 'du' command in terminal returned the same results. I can't imagine Omni doing something so different that it returned different results.

I have no reason to delete files in ~/Library. The files in my home directory were taking up only ~150gb, which should have left 250gb free, but didn't. I also had no reason to delete files outside of ~/.

I am just really looking for explanations about how my system could be so inaccurate in its calculation of memory, or if there's a chance that something else is amiss.

I am an experienced developer, and have no problem playing with files and permissions, but that is not relevant. What I'm looking for here are ideas about the differential between the sizes of files on my hard drive and the amount of space that the OS thought I was using.

Reply

Answer 3

BobHarris

Level 9

53,003 points

Aug 31, 2015 6:11 AM in response to jordancooperman

I also opened the info dialog for each individual directory under my hard drive, and the total file sizes were under half of what 'About this Mac' -> 'Storage' was showing. running the 'du' command in terminal returned the same results. I can't imagine Omni doing something so different that it returned different results.

Did you run 'du' with 'sudo'?

sudo du /

Otherwise you did not see any files that your account did have permissions to view. For that matter, did you start at top of the directory tree, or just in your home folder?

The MacObserver.com link explained how to run OmniDiskSweeper as root so it would be able to see everything on your disk (the technique could be used for other GUI based file checking utilities). The use of 'sudo' with 'du' is essentially doing the same thing. Making sure you see everything.

I am an experienced developer, and have no problem playing with files and permissions, but that is not relevant. What I'm looking for here are ideas about the differential between the sizes of files on my hard drive and the amount of space that the OS thought I was using.

The Finder information is generally based on Spotlight data. Sometimes re-indexing Spotlight will get things back in alignment.

<Spotlight: How to re-index folders or volumes - Apple Support>

The nice thing about utilities such as OmniDiskSweeper is it shows files everywhere (especially if it is started as a root owned process). Runaway log files have been known to eat up space.

If you are running Time Machine and do not have your storage device attached or accessible (a Time Capsule back home while you are away), then Time Machine takes local snapshots which it will flush to the backup disk when it becomes available. The Finder does not report this information, as Time Machine snapshots will be deleted if the space is needed (or that is the theory). These snapshots can be disabled.

I have no reason to delete files in ~/Library. The files in my home directory were taking up only ~150gb, which should have left 250gb free, but didn't. I also had no reason to delete files outside of ~/.

That is "Boiler Plate" because OmniDiskSweeper shows lots of files that many users have never seen before, and deciding they do not need 'mach_kernel' results in a system that will not boot. So I include my warnings anytime I mention OmniDiskSweeper.

Reply

Answer 4

Aug 31, 2015 7:39 AM in response to BobHarris

Thank you for taking the time to look over my answer BobHarris.

I did run 'du' with sudo. As you note, I was getting permissions warnings when not using sudo.

I didn't try re-indexing spotlight before the reformat, that probably would have been a good idea, and I will keep in in mind if I have storage issues in the future. I'll also look into disabling Time Machine snapshots, and will give Omni a shot. Thanks for your suggestions.

Reply

Answer 5

jzgoda

Level 1

0 points

Sep 10, 2015 4:48 AM in response to jordancooperman

I ran into a very similar issue, where a huge amount of my disk space (~80GB) just disappeared within a relatively short period of time.

While this may not be exactly what OP ran into, this is the closest description I've found to my problem, so I'm adding this for anyone else that searches this page out.

I tracked it down to being in a temp folder (/private/var/folders/), that starts with the name PSNormalizer.

For example: /private/var/folders/md/xt_95jfn6m7cqgzffd1174880000gn/T/PSNormalizer.lnO978/

There were dozens of these PSNormalizer folders, all containing a file that was between ~500MB - 2GB in size.

At first I suspected some kind of virus, but while running lsof (specifically: lsof -r 2 | grep PSNormalizer), I noticed that it was QuickLook that was creating these files. I also noticed that it was definitely happening when using QuickLook on EPS files. Since PSNormalizer has to do with PostScript, this kinda made sense that they would be causing an issue. I've just taken to deleting these folders every so often to temporarily alleviate the situation.

I've also opened a Bug Report with Apple to try and get this fixed.

Reply