Lion - Memory Usage Problems

One question on OSX that have been bugging for a while is - why 'purging' inactive memory take so freaking much time?

According to apple inactive memory is just in memory disk read cache, so only resident write-cache should be written back when 'purge' is runned. All the rest should be just discarded without writting to disc, this operation should not take a loot of time.

In case inactive memory is really a write-cache of the size that writting it to disk can take up to several seconds (with HDD writting 40-50 mb/s it would mean several hundreds mbs of data not stored on HDD), this mean that loosing power would lead to possible loss of hundreds mbs of non properly saved data.

BlackNova wrote:

One question on OSX that have been bugging for a while is - why 'purging' inactive memory take so freaking much time?

Because the purge command doesn't purge inactive memory. It just purges the disk cache. Remember, this simulates a 'cold' (empty) cache. That in turn forces the OS to rebuild the caches as needed to support all running processes. Also remember that there is never just one process running, but a mix of active & inactive ones. That's why it takes time.

According to apple inactive memory is just in memory disk read cache ...

That isn't what Apple says. Inactive memory is neither just a write cache or a read one. Some segments of inactive memory may contain data that has been modified since it was fetched from disk or created by its parent process. Some of that data can be discarded, but some may need to be written back to the backing store before that can be done. This is explained in great detail in the developer documentation that has been mentioned in this topic many times. Please refer to that if you want a better explanation.

Sorry Mac_Boston 😉

There is not picture to see 😟

After reading your information. I think I am shelving my Mac mini and going back to my custom PC.

I am a bit frightened that the techs for Apple are making a big deal over Inactive memory as if it is problem.

This remind me of the people who spend up to $5,000.00 to build a system to play a 32 bit game and complain about not working and such. (This is not directed at you Mac_Boston, just a observation)

Good luck.

R C-R wrote:

SeanChristmann wrote:

RCR, if you look at the image, you'll see that at the end of the day, "active memory" is over 1.7 GB, even though the apps running in activity monitor consume roughly the same as the beginning of the day when there is only 600 MB of active memory being used. That's 1 GB that the system simply cannot account for and never releases back.

Keep in mind that Activity Monitor does not show you a breakdown of how much active memory each process is using, just a total for all processes (real memory is not the same as active memory).

Indeed. Real memory is supposed to be bigger than the active memory used by the process. It should be at least the sum of the process' wired, active, and inactive memory used. Forgetting about inactive memory, which behavior changed with every new release of Lion, the picture shows that just the active memory is higher than the sum of the real memory used by all processes. The terminal command "top" shows the same numbers, so it shouldn't be an Activity Monitor "feature" but something that comes straight from the kernel.

The least that I can say (again) is that in OS X the collection of statistics for the memory usage is broken. I have never seen anything like that in any other Unix system (and before someone repeats that XNU stands for "X is Not Unix" I'd like to remind that since Leopard OS X is Unix 03 certified). Personally I gave up trying to figure out the memory management in OS X.

BTW am I the ony one that got spammed by this thread, in the sense that Mail is not grouping all messages into a single conversation anymore?

Just wanted to point out that the warning message in the screen shot posted by Mac_Boston indicates the startup disk is low on space and really has nothing to do with Inactive Memory. A number of the previous descriptions of InActive memory are what I understand it to be; memory containing recently used items that is available for allocation to other processes if needed or ready if the recently quit application if the user starts it up again. To the best of my knowledge it is not related to the InActive Processes list or at best a loose correlation for some of the items.

Also note that no matter how much RAM you have some things are written to Virtual Memory or the swap file and that is what was probably driving the warning message in Mac_Boston's case. For example any shared libraries needed by the newly lauched process are written to VM to make them available for any other process that may also need to use them. It is not hard to imagine that Safari uses a lot of such libraries and therefore requires more free disk space for VM / swap.

If Mac_Boston is still following this thread, how much free space did you have on your disk at the time of these errors?

For years I've seen a rule of thumb that one should have 10% free space on their start up disk for Mac OS X. As disk sizes have grown it has been pointed out that rule of thumb may be too conservative (does one really need 100 GB free on a 1 TB disk). I'm not sure what the lower limit for good performance is on free space, but I wouldn't run below 20 GB or so with a machine that had 8 GB of RAM, even that might be a little low.

Michelasso wrote:

Real memory is supposed to be bigger than the active memory used by the process. It should be at least the sum of the process' wired, active, and inactive memory used. Forgetting about inactive memory, which behavior changed with every new release of Lion, the picture shows that just the active memory is higher than the sum of the real memory used by all processes.

It should not be a surprise that active memory is not the same as the sum of the real memory used by all running processes. The real memory column in Activity Monitor includes both the active & inactive memory of each process, but it does not include any wired memory that process may be using since the system allocates that & it may be shared by several different processes. Likewise, you have to consider any shared memory shown since that is by definition used by more than one process.

I don't know how to explain it any better than this. If you sum a number of things that don't include every contribution to the total & include amounts counted more than once, I don't understand why you think that should be an accurate total.

The least that I can say (again) is that in OS X the collection of statistics for the memory usage is broken.

Keep in mind that there is an inherent limitation in gathering accurate memory statistics that applies to any OS: the process of gathering those statistics itself requires working memory & CPU time to do that, so the more time & memory you devote to that, the less representative it becomes of system behavior when that process isn't running. That's why (for instance) you get different results when you set Activity Monitor's update frequency to different sample intervals.

This applies to any utility that tries to gather statistics via samples. For instance, in the UNIX top organization man page (which applies to all UNIX-like implementations of top) the Bugs section notes that, "As with ps(1), things can change while top is collecting information for an update. The picture it gives is only a close approximation to reality." Obviously I hope, the faster things change, the further from reality the approximation gets, & the reality it approximates becomes less like the normal one the more often it collects data.

Is it normal?

Message was edited by: nkko

Message was edited by: nkko

R C-R wrote:

Michelasso wrote:

Real memory is supposed to be bigger than the active memory used by the process. It should be at least the sum of the process' wired, active, and inactive memory used. Forgetting about inactive memory, which behavior changed with every new release of Lion, the picture shows that just the active memory is higher than the sum of the real memory used by all processes.

It should not be a surprise that active memory is not the same as the sum of the real memory used by all running processes. The real memory column in Activity Monitor includes both the active & inactive memory of each process, but it does not include any wired memory that process may be using since the system allocates that & it may be shared by several different processes. Likewise, you have to consider any shared memory shown since that is by definition used by more than one process.

I don't know how to explain it any better than this. If you sum a number of things that don't include every contribution to the total & include amounts counted more than once, I don't understand why you think that should be an accurate total.

That's why when we add them up the total "real memory" should be bigger than the active memory, not smaller. That's why I insist. The point is that Activity Monitor naming doesn't make any sense. That utility should be either trashed or totally rewritten because it caused already far too much confusion. What AM calls "real memory" is instead "resident memory", as "top" rightly defines it. The values in AM and top are exactly the same. I triple checked. Calling it "real memory" make people think that it is the total amount of RAM the process is using. Well, apparently it isn't so. Otherwise I don't know how to justify this line from the top header:

MemRegions: 19363 total, 1025M resident, 44M private, 348M shared. PhysMem: 362M wired, 1442M active, 873M inactive, 2676M used, 394M free.

The 1025M is (must be actually) the total RSIZE (real mem in AM terminology) memory for all processes. Even summing up all resident values they are still slightly lower than the active memory. At this point what's the correlation between the resident memory and wired/active/inactive memory isn't given to know.

When we say "Safari web process is using 1.2GB of RAM!" we are actually saying "Safari web process resident memory is 1.2GB". Which is what top rightly says. Same number, wrong meaning in AM. So guess what. The total RAM really used by SWC or any other process is/can be even [much] higher than what we usually think. That's why the free memory can go down to 0 with AM not showing the reason. That's why this thread is 42 pages long.

My guess is that this is because we are dealing with Mach numbers for the kernel/virtual memory system, and BSD numbers for the processes. They simply don't seem to match, whatever it is the reason. What I know is that I can't trust those numbers. Not the ones for the processes at least. The whole RAM usage is much larger than the one reported for the processes. That's the only thing I know for sure and that's why many people keep complaining or wondering.

Is it normal?

nkko wrote:

Is it normal?

Message was edited by: nkko

Message was edited by: nkko

What? 2 SWC processes? I am afraid you hit a bug.

Hi gbullman,

I am still here.

If I understand your question correctly the answer is - all was available. It was a brand new machine with 8 GB RAM and 500GB memory (Core i7.) The only additionally things loaded on it besides what it came with were: Keynote, Pages, Numbers, Adobe Flash update, and Office for Mac 2011. All other files were sitting on an external drive and no additional drivers or software had been added at that point.

This was a 'testing phase' with Apple so the brand new machine stayed very clean to try to prove it wasn't my data doing it (since it occurred on 3 other brand new machines in the 6 weeks before this.) This screenshot was one of four sent to Apple during this 'testing' phase.

I only commented on the Inactive Memory because they directed me there during the troubleshooting activities. At some points the machine would have 7 GB of Inactive memory and would be so frozen I'd have to shut down. Essentially what I was experiencing, regardless of all this memory talk, was non-stop beach balls, the message that the start up disc was full, and having to shut down a lot.

Thanks

Michelasso wrote:

Calling it "real memory" make people think that it is the total amount of RAM the process is using.

It is exactly the same thing that top calls RSIZE. Hover the pointer over the real memory column in Activity Monitor for a couple of seconds & you should see a popup 'tool tip' label saying "Real Memory (RSIZE)." Do the same for the other memory columns & you should see the same thing -- the top names appear in parentheses after the plain language name.

It is called "real" memory to distinguish it from virtual memory (IOW, the VM backing store on the drive). This distinction is quite common in articles about virtual memory systems. It is not unique to Apple. (Google "real memory" to see what I mean.)

And it is a measure of the total RAM (real memory) allocated to the process, in exactly the same way top's RSIZE is. Likewise, top's "PhysMem," the total physical (resident, real or RAM) memory usage of the system (broken down into wired, active, inactive, used, and free components) is exactly the same as Activity Monitor's representation of those components (less the pie chart representation, of course).

At this point what's the correlation between the resident memory and wired/active/inactive memory isn't given to know.

Active, inactive, wired, & free memory is allocted to real (or if you prefer resident) memory according to complex algorithms that are well documented but not easy to understand. If you are interested in that, study the developer document that has been mentioned here many, many times.

Mac_Boston wrote:

If I understand your question correctly the answer is - all was available. It was a brand new machine with 8 GB RAM and 500GB memory (Core i7.)

RAM is memory. The 500 GB refers to disk storage capacity. The message that the start up disk was full suggests that (for whatever reason) the capacity of your Mac's internal drive (the start up disk) was nearly used up by files & so there was not enough room on it for all the virtual memory & other temporary files required to support the applications you were running.

If the difference between memory & disk capacities were not clear to you, you may have misunderstood what the Apple people were telling you about what the problem was they were testing for. The two capacities are related (because of VM & other temp file space) but not the same thing.

Hi RC-R,

Admittedly I can't distinguish the start up disk from the RAM in this discussion - seems like the same thing.

However, I do have a clear understanding of the difference between RAM and storage. There were no (zero) files load by the user on the machine at the time. The only thing that could have been taking up storage space were the loaded programs (not their processing but just the programs themselves and their content.) And I do know that I need a certain amount of open space on my hard drive to get the machine to be able to perform well.

I do have a lot of clarity on what Apple was testing for just do to all the time we spent together. I'm not saying their techs were correct but they believed, on 4 machines, the slowness of the machine, the constant beach ball, the start up disk being full, and the basically bad performancy was coming from the the fast consumption of RAM in a matter of minutes. First they assumed it was a problem caused by the user (me) loading my PC data on to it. We disproved that in a few ways. Next they blamed Adobe Creative Suite CS4, we removed that. No change. Then they decided that machine #1 (a 13 inch) had a corrupted video software that was copied from machine #1 to #2 to #3. Once that was fixed and the problem persisted they looked more deeply into the consumption of RAM.

When they investigated the RAM they were concerned about how much Inactive Memory was showing as consumed. For two weeks they had me sending data over when the problem happened from 3 or 4 views in the Activity Manager. When it couldn't be resolved they switched me to a 4th machine. We kept this machine totally naked for two weeks and I used an external hard drive for files. Problem still happened; they monitored it in person and through screenshots. Then we added Office for Mac 2011. No change. Finally they had me stop using Safari. The problem improved.

I have since loaded the machine with all the software and files I need to use the machine for business. It still struggles a bit with graphic intensive programs but has only given the start up disk full message once.

I briefly had this same configuration in April 2010 for two weeks and it was ridiculous capable and fast. Then I had a similarly configured Lenovo and I never once had any concern about its speed or capabilities (its graphic driver failed spectacularly in the first year however so they took it back and I came back to Apple.) I keep hoping whatever this glitch is will be repaired so that I can have the full benefit of this machine.

PS. I too am getting every response to everyone on this chain ...