OSX Lion: Backups/Other Disk Space

CraigMac wrote:

It has to be counting this space.

If you're talking about Local Snapshots, the Finder (and Get Info) don't, but Disk Utility does.

I went to clone my Bootcamp partition today (wanting to increase that partion size and finally found a way around reinstalling Windows), and I was alerterd my hard disk was full!!!!

You'll have to clarify that one. Do you mean you were trying to use Disk Utility to "shrink" your OSX partition? If so, there are two possible explanations:

Disk Utility does count Local snapshots, so may prevent you from shrinking the partition. But if the partition is over 80% full, the snapshots will be deleted, although that may take a while.

Partitions must occupy contiguous space on the disk. So if the disk has ever been relatively full, there may be some files towards the "end" of the OSX partition: there may be enough total space to shrink the partition, but it's not contiguous at the end.

I have "other" for 2 user accounts, and both are very LARGE. The one user account has not been used in over 2 years because it's jacked up. So, Lion is doing something to store extra data from an un-accessed user account.

Not necessarily. See below.

So is Time Machine caching backups then when it is disabled?

No. As posted before, if it's making Local snapshots at all, it makes them whether the normal Time Machine volume is available. They are completely independent.

Is there anyone to flush this crap out?

If you mean, find out what's taking up the space, yes. See Where did my Disk Space go?

I have not yet done anything with a partition, this was just how I discovered this mystery "other" storage being in use. To be specific, I was making a clone of my windows partition that I was going to reinstall later. The clone was 34GB, and I should have had plenty of room on my Lion partiion to house that file temporarily. (I can always push it out to my external drive if need be.) So essentially, I was simply trying to copy 34GB from one partion to the next... it just wouldn't let me because it says my hard disk is full when it really is not.

I know the majority of Apple users are on portables these days, but these issues aren't unique to them. It's not Time Machine holding something for the next time it's connected. Lion is holding a certain amount of space hostage. It seems as if Lion is caching a lot of data on desktops too and not flushing it properly.

I called Apple Support... of course you get the run around and denial of it being a known issue. I got the usual reset pram and the like (that I'd already done, as well as every other utility known to man) and was then told to do a reinstall. This gave me back 4Gigs of the 128GB's Lion is playing terrorist with and holding hostage on my hard drive.

What I took away from the support call was this: the Genius was not surprised by this issue at all. You could tell he's heard it 100 times before... even giving me extended service calls out of the goodness of his heart.

I am willing to bet this is simply a very nasty bug in the OS, and it's something they won't acknowledge until they have a fix for it, or are close to one as that is Apple's way anytime there is a problem.

Really, I think a lot of the performance issues I'm reading about in other threads my be related with this being the root cause. Macs aren't known for getting fragmented or needing to be defraged, so we don't often as Mac users, consider this idea. If that "other storage" is filling your hard drive, it would stand to reason you would see a good performance hit. (The Mac Genius on the phone actually concurred with this thought, because I have had horrible perforamnce since upgrading and he insists even with the age of my machine, that should not be the case.) My hard drive is always active when even doing the most mundande tasks... this really takes me back to Windows using days.

So is anyone with this issues also seeing performance slow downs? Lots of spinning beach balls? Lots of hard disk running wild in the background for no reason?

If you take the time to manually add up the size of folders and such, it matches the storage bar you get in "about this mac" perfectly down to the last kilobite. No where on the hard drive is any indication of files stored elsewhere.

I'm willing to bet that as Lion saves your last user state, documents, etc. it has something to do with that. Like it's storing too much and then not dumping those temporary files later. Using something like Onyx or any other utility should dump those files, but even those apps can't flush it. Something is just stuck.

I will call support again, and if they give me anything useful worth sharing, I'll repost... but it seems as if no one has had any luck with this issues. Those who have all have seemed to have it return almost instantly.

The next OS update should actually be landing very soon... hopefully that will have a fix.

CraigMac wrote:

. . .

Lion is holding a certain amount of space hostage.

No, it's not. The "missing" space is either some hidden files, or your directory is corrupted.

Have you verified your internal HD, per #6 in Using Disk Utility?

as well as every other utility known to man)

For this purpose, the only one I know of that will show everything at once, including the hidden things most others won't, is the WhatSize app. See the green box in Where did my Disk Space go? (Actually, try the other things listed there first).

I am willing to bet this is simply a very nasty bug in the OS,

No, not a general one anyway. Most of the size problems turn out to be one of the things in the link above. Give them a try.

Actually, you're wrong on this, and I just wanted to make mention in case others find their way here.

You can add up your files manually, and it will match perfectly. The "other" cannot be accounted for by any means of user access. Whatever is in "other" is totally hidden. The only exception is on a Windows partition, where the entire alloted used space will highlight as other.

Lion even will correctly categorize an external hard drive. It uses file extensions from the Spotlight index to determine this information. If you reset spotlight and pull your hard drive inventory, it will show everything under "other" until the re-indexing completes itself.

The link you posted to "Lions Storage Display" is about useless. Even the notable things that would show a discrepancy, they still would never add up to the amount of lost space. (Add your folders yourself if you don't beleive me. Math is an exact science, yes? The numbers don't add up... and I haven't seen discrepancies like this since Enron.)

1. Spotlight updates itself every time a file is added or deleted

2. I doubt any users on here have thought to manually force it to exclude certain files.

3. You will always have some type of other... random files spotlight doesn't understand what they are... but not 80% of your hard drive's free space!!

I appreciate that you keep trying to help people, but you need to realize all the sources you quote will be useless in the event of a bug. Yes, what those sources site is "how it SHOULD work." It's obviously not how it IS working.

CraigMac wrote:

. . .

You can add up your files manually, and it will match perfectly.

Only if you include all the hidden system folders, including Local Snapshots and the Versions Database.

As posted, the easiest way to do that is with WhatSize (in "As Admin" mode). That will get the same total, but won't match the individual categories.

The "other" cannot be accounted for by any means of user access. Whatever is in "other" is totally hidden.

No. Other includes all the things not in the remaining categories, including most of OSX, your documents, and various other things.

About the only category that will match are Backups (Local snapshots on internal HDs, or Time Machine backups on externals).

Please, have you tried the things in Where did my Disk Space go? One or more of them will very likely find the "missing" space. They have for many, many users here.

This isn't my first time to the party. I have verified my disk, and done vast amounts of trouble shooting, resets, reinstalls, run utilities... and... have just had my third support call with Apple.

Apple agrees with me. Something is not right, and they don't know (or won't say more likely) what it is.

Every source you have shared is technical right... if the software is behaving the way it is supposed to.

No one has 128GB of hidden files. You can also force hidden files to be visible if you want to manually count them.

Lion doesn't function like IOS, though it's attempted to emulate many of the feature of IOS. This is the equivilent of a plumbing problem, and waste not going down the drain.

Even if the system is using this space, it SHOULD not hold it hostage. This issue is creating fragmented hard drives. As knowledgeable as you seem to be, you should be well aware of what happens on Macs when you fill a hard drive to near capactity. Those issues are occuring with peoples systems as well.

If you don't want to take my word for it, talk to the last 3 support reps who have not only just refunded my purchase for Lion. I can tell they are aware of the issue and unable to speak of said issue.

Water shouldn't be red either... so if your water turns red, something is obviously not normal. No internet link the explains the process of how water is made and falls will account for red water, as your info isn't accounting for an obvious bug.

Pondini wrote:

CraigMac wrote:

. . .

Lion is holding a certain amount of space hostage.

No, it's not. The "missing" space is either some hidden files, or your directory is corrupted.

Have you verified your internal HD, per #6 in Using Disk Utility?

Yes, I have.

as well as every other utility known to man)

For this purpose, the only one I know of that will show everything at once, including the hidden things most others won't, is the WhatSize app. See the green box in Where did my Disk Space go? (Actually, try the other things listed there first).

I am willing to bet this is simply a very nasty bug in the OS,

No, not a general one anyway. Most of the size problems turn out to be one of the things in the link above. Give them a try.

CraigMac wrote:

This isn't my first time to the party. I have verified my disk, and done vast amounts of trouble shooting, resets, reinstalls, run utilities... and... have just had my third support call with Apple.

But have you tried all the things in Where did my Disk Space go?

I understand you're frustrated, but until you do, you haven't ruled them all out.

This is not my first time, either. I've been beta testing Lion since February, and helping folks here since it was released in July. There is no general bug taking up lots of space. There are some things that occasionally take up many GBs, most listed specifically in the article above; others will be found using WhatSize per the green box.

You might want to check, if you own an iPad, iPhone, or iPod, the mobile back up folder located: Library > Application Support > MobileSync > Backup. Choose File > Get Info. This can show you how large the file is. To access the library folder, hold down the option keys and select the Go menu in Finder.

You can delete these if the file is too large for your liking by going into the iTunes preference and under devices choose which back ups you wish to delete.

Make sure you back up in case you wish to undo your deletion.

Hello CraigMac, Pondini et al.

Thought I'd chime in with more information which can help you identify your space issues. Drop all the graphical tools. Get down to brass tacks--use the UNIX tools directly.

1. Open up a terminal window. (Applications->Utilities->Terminal.app)

2. Change to the root directory. To do this, at the terminal prompt type:

cd /

As an aside, you can always verify your current location by typing "pwd" at the prompt (without the quotes). pwd stands for "print working directory".

3. To identify the, (for example) 500 files and directories which are taking up the most space on your disk type:

sudo du -ak | sort -n | tail -500 >~/Desktop/fatties.txt

(note, this took 29 minutes to run on my MacBook Pro)

This will create a text file on your Desktop named "fatties" which contains a list of the 500 files and directories which are consuming the most space on your disk. Once the command finishes you can double-click on the fatties icon to bring it up in a text editor for inspection. Note that the list "double-counts" size with respect to folders. For example, let's say you have a folder which contains six 1 gigabyte files. Then the fatties file will show each of these 1 gig files, plus it will show the directory as taking up 6 gigabytes (that is, the sum of the contents of the folder).

Here's an explanation of the command "pipeline" I've written above:

1. sudo ==> will prompt for a password. It will give you the administrative power to inspect every file and folder (that is, it overcomes "permission denied" errors when inspecting certain privileged files on the system)
2. du -ak ==> this is a UNIX utility which means "disk usage". The hypen indicates options you are providing to this command. The "a" option means, inspect the size of all files (by default du just summarizes folder sizes). The "k" option means report the size in kilobytes. Instead of the "k" you can place "m" or "g" which means report file/folder sizes in megabytes and gigabytes respectively. However, keep in mind that these numbers tend to be rounded up. So a small file which only consumes 4k will be reported as consuming 1 meg or 1 gig if the "m" or "k" option is specified.
3. The vertical bar (called a "pipe" in UNIX nomenclature) just means that you are going to feed the output of one command to the input of another command. In my "pipeline" above, I am sending the output of the "du" command to the "sort" command for further processing.
4. sort -n ==> The du command outputs files/folders and their sizes in the order they are encountered. We are concerned with the largest files/folders, so we want to sort the output from smallest to largest in size. The "-n" option means to sort numerically (versus alphabetically).
5. tail -500 == At this point we have thousands upon thousands of files/folders which have been sorted in ascending order. The vast majority of theses are small files which don't concern us. Because we are concerned with the biggest space-consuming pigs, we only want the end, or "tail" of this sorted list. In my example I have requested the last 500 sorted files/folders. You can make the list larger if you want (say, 5000).
6. >~/Desktop/fatties.txt == Normally, the output of this pipeline would spit out in the terminal window. However, we want to preserve the output in a file so that we may inspect it at our leisure. The ">" means redirect the output to a file. The rest is the "full pathname" of the file. The tilde (~) means your home folder. The "/Desktop" means the Desktop folder which is in your home folder. And finally, the name of the file it will create on your Desktop. Note that the file will be created as soon as you start running the command. However, it will be empty until the command finishes (it writes out the contents to the file right at the very end).

Here's what the end of my fatties.txt file looks like:

And here is what the beginning of it looks like:

Again, in my example, the numbers shown are kilobytes (because of the "k" option used for the "du" command). So the first entry in the file (the 500th largest file/folder) on my disk is a Caillou video episode which takes up 338340 kilobytes. Divide by 1024 to get 330 megabytes, or divide by (1024*1024) to get 0.32 gigabytes. The last entry (".") means the "current" (that is, root) directory and all of its contents. In other words, the size taken up by everything on the disk. In my case, 302922824 / (1024 * 1024) = 288.9 gigabytes.

Let me know if I can elaborate on any of the above.

boawk wrote:

. . .

Thought I'd chime in with more information which can help you identify your space issues. Drop all the graphical tools. Get down to brass tacks--use the UNIX tools directly.

That's fine for an experienced user.

But sending a non-technical, novice, or casual user to Terminal is not a good idea. Too many things can go wrong.

There are GUI tools available, that will work fine for most users.

There are GUI tools available, that will work fine

-->for most users.<---

Precisely. This thread's been going around and around. There are disputes as to what the GUI tools are reporting, etc. An alternative strategy for the remaining frustrated users who have fallen out of the "most" category is to use the terminal. All I've provided are two lines to be entered at the command-line which can be copied and pasted. Shrug. At any rate, interrogating the disk directly will remove any ambiguity as to where disk space is going.

Hi,

I am having the similar issue. I have 2 TB HDD. my home folder says 1.98 TB has been used, but when I count the used space of the documents, movies, music and photos folder, total used space is about 900 GB (If I include application folder (guessed size).

Unable to see the rest of the available disk space. in finder info also it says used space 1.98 TB.

I dont know where it has been used.

please help

See Where did my Disk Space go?

Can you see the problem here? Any suggestions?

Long time multi Mac user, but my first post. So happy with finding the answer to my problem that I thought I should make the effort to share.

I have been having the creeping cancer tumour of "Other" storage growing for a few months now in About My Mac ("Backups" was not a problem). Throwing out unnecessary files didn't help much. The spare disk space on my 128GB SSD kept dwindling.

Searched high and low around the internet and saw that others appeared to have the same problem. Spotted Pondini's advice here and there - doing a great job and no doubt solves the vast majority of queries or problems on the subject of "Other" storage.

I went through all the obvious things, then the less obvious. OmniDiskSweeper scan - check. Time Machine fiddling - check. Nothing worked. I confess that I had started to share CraigMac's frustration. Like him I am not inexperienced with the Mac and I just knew something was wrong. Turned out the solution was in amongst Pondini's suggestions after all. The clue came from running Verify Disk in Disk Utility on the boot disk. It noted problems. If yours does too - you could be a couple of minutes away from a happy ending. (If running Lion).

Boot from the Recovery HD. (Restart holding down Command and R). Select Disk Utility from the list that eventually appears. Repair your boot drive.

After repair and restart, "Other" storage in About My Mac dropped from 77GB to 33GB. Which is about where it should be in my case.

Thanks Pondini.