Lion - Memory Usage Problems

Here's a capture of Activity Monitor showing all processes (not just user) sorted by real memory. As you can see, there are only a few basic apps running (Safari, Mail, iTunes, Finder, Address Book, Activity Monitor, and Grab); yet "free" memory is only 11.5 MB out of 4 GB. Yes, there is 1.05 GB of "inactive" memory, but this appears to be released very slowly in tiny increments with lots of beachball spinning, so that doesn't help much.

Note: iTunes is not playing anything. It's just open.

Also note the swap used is 8.14 GB. And in the couple minutes since I took this capture and wrote this post, that swap has climbed to 10.56 GB (while I've only picked up about 30 MB of "free" memory).

I have a MacBook PRO with 8Gb of memory. I have never ever seen the dreaded "out of application memory" problem until the OSX update time before last. Then I was getting it just about constantly. WIth the update last Monday I have not seen it until tonight. And I was both surprised and mad to see it again. I thought the problem was a very short term glitch caused by the OSX update that then was fixed last Monday with the most recent update.

NOT.

So what is Apple doing to us anyway? Let me think... No problem like this at all until very recently. Now with nothing else added to my machine, I get this idiotic application memory problem.

HELLO APPLE... Are you listening. I have been a loyal, very loyal, Apple supporter for a very long time. Through thick and thin. But my patience with this is really wearing thin.

Please concentrate on problems like this if you want users like me to remain loyal! More pointedly, FIX THIS... and soon... I cannot tell you how frustrating this is.

I know that memory leaks can be very bad news. But you are supposed to be a really hotshot company. So fix it!

JHS

MacBook Pro

Processor 2.3 GHz Intel Core i7

Memory 8 GB 1333 MHz DDR3

Graphics Intel HD Graphics 3000 512 MB

Serial Number C0*******F91

Software Mac OS X Lion 10.7.3 (11D50)

512 GB SSD drive

<Edited by Host>

R C-R wrote:

Michelasso wrote:

Because I was talking about the total value indeed, the one that top reports as "resident".

What specifically is the name of what you see in top that reports that total, other than the per process totals (IOW, other than RSIZE)?

Again, everything is just vague.

It isn't vague. It's just so complicated that you cannot assume anything means anything other than exactly how it is defined in those internals you don't want to read.

The top header value that I showed so many times in this line:

MemRegions: 13373 total, 915M resident, 44M private, 267M shared. PhysMem: 331M wired, 1543M active, 332M inactive, 2207M used, 864M free.

Which is an approximation (again) of the total sum of all RSIZE. Supposely it should be the size of all RAM mapped in the tables. In other words the total RAM actually used. But that conflicts with the numbers for the Active memory (which being actively used should be all mapped) that is always higher or even a multiple of that value, as I showed. And we are ignoring the wired RAM.

And it is not complicated. It lacks documentation and it changes all times. We are not talking about quantum mechanics here. Nor about memory segmentation, which was more understandable anyway. We are just asking to know how the RAM in the system gets used. If RSIZE (real memory) represents the rss, i.e. all RAM mapped for a process, the total amount of used RAM can't be higher than the sum of all RSIZE (because of the shared memory). That includes the RSIZE for the kernel that is a process itself. If that value is still smaller than the Active memory, it either means that part of the Active memory is not mapped (the mapped RAM is a subset of the Active memory) or that there is memory mapped that isn't represented by any RSIZE. Is it so complicated for Apple to explain what happens to the difference in RAM between the two values?

Right now we are looking at "real memory" numbers that are an approximation the least, misinforming the worst. If there are some many people coming out with the same question there must be a reason. Is it because of a very bad approximation? Ok Apple, just say so. That would enough for an explanation.

Michelasso wrote:

R C-R wrote:

Again, everything is just vague.

It isn't vague. It's just so complicated that you cannot assume anything means anything other than exactly how it is defined in those internals you don't want to read.

The top header value that I showed so many times in this line:

MemRegions: 13373 total, 915M resident, 44M private, 267M shared. PhysMem: 331M wired, 1543M active, 332M inactive, 2207M used, 864M free.

Like R C-R commented, with these things being as "complicated" as they are, one should not assume they mean anything esle, but exactly their definition.

Maybe you will care to note, that 'resident' and 'actve' are listed in two rows. One deals with the memory regions (mapping) and the other deals with physical memory.

As we talk about RAM shortage, we should really look at the physical memory statistics, and that incldes only wired, active, inactive and free. Everything else is accounted for in the virtual memory space and is of little interest to this discussion.

By the way, one ofter overlooked fact is that 'wired' memory is actually what the kernel allocates/manages. It may do so on behalf of user level processes (this includes the 'system' processes and the processes started by the respective user). This memory is never paged out to swap. This makes it different from 'active'. If you have significant portion of your memory wired, you need to investigate the applications that you use -- it will typically not show as attached to an specific application.

Right now we are looking at "real memory" numbers that are an approximation the least, misinforming the worst. If there are some many people coming out with the same question there must be a reason. Is it because of a very bad approximation? Ok Apple, just say so. That would enough for an explanation.

Let's be clear on this. Apple invented none of this VM stuff. Most of it comes from research projects such as Mach and BSD. It is not rocket science yes, it is well understood technology, unfortunately by not that many people.

What Apple did was to tune VM for OS X, so that you get specific experience. Apparently, that tuning might not match well the typical mix of applications for every user.

Perhaps people who experience this behaviour should look at what common set of applications they use.

By the way, when one says 'Safari ate all my memory' -- perhaps they had some plugins enabled in Safari, such as Flash that actually ask Safari to allocate all that memory for them. Same for Dock etc.

While Apple provides the OS X platform and few applications, they cannot save the user from themselves, or from the bad applications they insist to run on their computer.

JonSticklen wrote:

HELLO APPLE... Are you listening.

This is a user forum hosted & moderated by Apple but you are only talking to other users here. To send comments to Apple, use the http://www.apple.com/feedback/ link.

You should not post your product's serial number or other personal information here. A request has been sent to the hosts to obscure or remove that info from your post.

By the way, when one says 'Safari ate all my memory' -- perhaps they had some plugins enabled in Safari, such as Flash that actually ask Safari to allocate all that memory for them. Same for Dock etc.

I agree with that. While Safari tends to use a lot of memory I can't say it use more than Chrome for example, though it does use more than FF on comparable number of tabs open.

I use Safari with minimum plugins set, If I remember correctly I got only adblock installed, so If Safari sudenly start to consume gbs of memory with 15-20 tabs it is almost certain some plugin issue, unless you browsing some site with uncompressed and unscaled photos 🙂

dkalchev wrote:

Maybe you will care to note, that 'resident' and 'actve' are listed in two rows. One deals with the memory regions (mapping) and the other deals with physical memory.

Unless Apple changed them, lets' look at the definitions:

- rss (resident set size): amount of RAM paged in by a process

- wired memory: RAM currently in use by the processes that can not be paged out

- active memory: RAM currently in use by the processes that can be paged out

Note as rss is per a single process while wired and active memory are global values. Because of the shared memory common in more processes, in any other Unix system I know (OS X is Unix 03 certified) is true the following:

Sum of all rss >= RAM used

In OS X it should be

RAM used = Wired memory + Active memory

Instead we get

Sum of all rss < Wired memory + Active memory

and even worse

Sum of all rss < Active memory

This is not complicated, this is confusing. Nowhere it's explained how the delta

Active memory - Sum of all rss

is allocated. Which is something many people are asking. Unless, off course, the approximation when measuring the rss is so bad to become totally useless.

I left out the wired memory because I am startng to doubt it gets actually paged in OS X (still wondering how it would manage it, but that's ok). And I obviously didn't mention at all the inactive memory because it is generally considered a sort of "cache".

BTW the wired memory doesn't belong just to the kernel. Any process can request to have some chunks of RAM wired. Just run a virtual machine with Parallels and you'll see that most of its RAM used will be wired.

Michelasso wrote:

And it is not complicated. {...} If there are some many people coming out with the same question there must be a reason.

The reason is that it is not as simple as most users & even many developers think it is; IOW, that it is complicated. Please refer to the two Darwin-dev mailing list messages already mentioned, & tothis one, which is a follow up by Michael Smith to the one you mentioned earlier.

Each of these three messages attempts to explain why these questions are not easy to answer, why top is only telling you in a very crude sense how much real memory a process uses, & why the numbers are not exactly what you might think they are. Even though the language is different, you are making some of the same incorrect assumptions Mr. Smith addresses in his comments to the other developer. Each number accounts for memory use in a different way: page mappings are reliable indicators of residency status only for specific conditions & page maps are updated only when needed because the process is computationally expensive.

As Mr. Smith emphatically points out in his follow up, there is a limit to how much this will help developers struggling with the concepts without the background reading necessary to understand it all, & it is no less true for users.

All I can suggest at this point is if you want better answers, you should consider subscribing to the mailing list & ask for them there.

By all means used memory = wired + active + inactive Example: PhysMem: 982M wired, 1187M active, 439M inactive, 2608M used, 1485M free. 982+1187+437 = 2608 2608+1485 = 4093 (This Macbook has 4GB RAM) The inactive memory is not cache, it is memory that has not seen recent activity and might be paged out, if the OS is short in RAM. On other OS, for example FreeBSD, top reports OS caches separately: Mem: 2235M Active, 592M Inact, 4499M Wired, 85M Cache, 819M Buf, 419M Free Here the caches are "Cache" and "Buf". These don't sum up to 8GB (what that box has) because unlike OS X, these are not separate counters. Wired memory is not paged. This means that, if you have an application that needs huge amounts of wired memory, such as for example System V style shared memory, it will leave very little for other non-swapped memory and you will either get excessive swapping, or 'no memory' conditions. It is the kernel that manages wired memory -- of course virtual machine emulators need their memory non-paged or they will crawl.

R C-R wrote:

As Mr. Smith emphatically points out in his follow up, there is a limit to how much this will help developers struggling with the concepts without the background reading necessary to understand it all, & it is no less true for users.

All I can suggest at this point is if you want better answers, you should consider subscribing to the mailing list & ask for them there.

Mr Smith has an attitude indeed. He's the same that wrote that everything is documented in the (open source) code. Which from a system adminstration (but not only) point of view it is not an answer. To check how much RAM is used and how much the processes are using of it is a common task. Sysadmins want a single number, they don't want to go inside the memory maps of all processes. And from his own admission the real memory numbers are an approximation, thus wrong. OS X may have a "very efficient memory management system" as Apple claims (but has it? It might be faster but I seems to me that it just eats RAM out of nowhere) but the number of complains are far too high. I just did a google search: "windows 7" eating ram. Well, the first result is this very same thread. With number 9 being the OS X Dock using too much memory.

In any case I tried to search the mailing list. Not a single page matched even using "memory" as a research key. I subscribed to the darwin-kernel mail list with RSS, I don't want to be spammed with another mailing list for just a single question. I just wonder why the search doesn't work. Maybe I am doing something wrong. Unless one must be subscribed to search it.

dkalchev wrote:

By all means used memory = wired + active + inactive Example: PhysMem: 982M wired, 1187M active, 439M inactive, 2608M used, 1485M free. 982+1187+437 = 2608 2608+1485 = 4093 (This Macbook has 4GB RAM) The inactive memory is not cache, it is memory that has not seen recent activity and might be paged out, if the OS is short in RAM.

Yeah, thanks for the refreshement. More or less we all know that this is how Apple defines the 4 chuncks of memory in OS X. Now Iit would be nice to know the relationship between them and the "Real Memory" values shown in the Activity Monitor process list.

Thank you R C-R. I know better than to include my serial number in a post here, but I was pretty upset by what I saw as a constant degredation of OSX lately. I appreciate your reminding me, and esp your asking the moderator to remove my serial number!

What I don't understand is why Apple is not more concerned. Since Lion my Apple mail client crashes about once every few days on average. I dutifully send off the crash report to Apple to their big bit bucket in the cloud. I could always change to some other mailer, and I have thought about it. The advantage to me of using Apple stuff has always and increasingly been the integrated ecosystem. Its been very good for my use until now.

Yesterday, my clock (upper left of the bar at the top of the monitor) was "stuck" on a time. When I moved my mouse to that bar, I got a spining icon - ie, I could not get into the finder. The only way I could fix that was to reboot my system. I have never ever seen an OSX system so flaky as 7.3. And I have installed nothing new from the time I had Snow Leopard on this machine.

The issues with memory are the really thorn for me now. I appreciate all the developer's on this forum trying to work out the issues and all the Super Developers who probably rightly say that some of the more new developers do not have all the background to understand what is going on.

I am not an illiterate user. But I honestly don't care about all these things. Rather I care only to the extend that someone (preferably at Apple) understand and some team is working round the clock to fix this memory issue. I do not want to have to reboot my self every day (or every 2-3 hrs) to avoid the issue.

I am writing this just to indicate that there are multiple constituencies on this forum. People like me are bringing issues. Hopefully the developers are listenting to the issues and taking steps.

And I have to respectfully disagree about Apple listenting to this forum. I understand the offical position is that Apple does not listen here. But think about it... This forum in particular is the heart of all things Apple not using iOS. If they are not listenting to issues that come up here - even though they may not respond - if they don't listen here they are making a big mistake in my view...

Michelasso wrote:

To check how much RAM is used and how much the processes are using of it is a common task. Sysadmins want a single number, they don't want to go inside the memory maps of all processes.

System administrators who want a single number can get it very easily: it is the "Used" number displayed in Activity Monitor. If they want a detailed, process by process breakdown of that they can consult the section of the often mentioned developer document devoted to the various tools & techniques Apple provides for that. Like it or not, that does require consideration of page maps, the different ways the OS allocates memory, & so on. Wishing that it was not so won't get them (or anybody else) anywhere.

And from his own admission the real memory numbers are an approximation, thus wrong.

An approximation is by definition something close to but not completely accurate or exact. How "wrong" it is depends on how, knowing that, it is used for some particular purpose. IOW, it would be wrong to use an approximation when an exact number is required but perfectly acceptable to use one when it is not.

That's basically what Mr. Smith (among others) is trying to tell the mailing list participants. The OS does not need to know the exact allocations -- in fact, trying to do that would slow the system to a crawl. That much at least should be obvious to you from studying the developer documents. If it is not, I don't know what to tell you.

Regarding the number of complaints, consider that Lion is in use literally by millions of users. The number of users complaining about memory use problems is very small compared to that. In fact, the number of users that explicitly say they have no such problems is comparable to the numbers of those who say they do. Moreover, the problem reports differ greatly -- quite a few aren't even problem reports as such but expressing concern about one or another kind of memory use. There isn't even a general consensus about which version or versions of OS X exhibit these problems.

Granted, this proves nothing but it very strongly suggests that the problems are not inherent in the OS itself & instead are due to one or more other factors. Whether that is true in all cases really doesn't matter much. This is a user to user forum. We don't write the code nor can we change it. All we can do is try to identify & resolve the problems we as users can do something about. Complaining about anything else here is pointless, a waste of time & energy that just makes it that much harder to help those we can.

If you want to complain to Apple, use the feedback links mentioned many times in this long discussion. If you want to help try to identify & resolve the problems we can so something about, this is the place to do it.

Jon,

if you have so many issues I'd suggest to reinstall Lion... over Lion. It fixed few things to me (Mail was showing in half Italian and half English for example. Reinstalling it got fixed) and other people that tried doing that had better results. Like the boot time that in some cases had a bad slow down went back to normal.

It only takes half an hour or so. Just click on your "Lion Install" application from MAS and let it resinstall. If your MAS Lion copy is not updates Software Update will do the rest (or you may want to download the 10.7.3 combo update from the Apple site). It seems to be a pretty safe procedure. Still you may want to do a backup first.

R C-R wrote:

If you want to complain to Apple, use the feedback links mentioned many times in this long discussion. If you want to help try to identify & resolve the problems we can so something about, this is the place to do it.

I did more than that. As we well discussed it in this discussion I had 10.7.0 trashing since day 1. The swap used was jumping sky high in few seconds. I opened a Bug Report and that has been followed by the Apple developers whom kept requiring data (the sysdiagnose output) from me anytime it happened . The trashing disappeared indeed in 10.7.2. Still someone kept denying that such trashing could be even possible. As if Apple is God and OS X can not have bugs.

Because of that Bug Report (I presume) months ago I have been selected for the AppleSeed program. I can assure you that even in its (private) forum many people, included the ones skilled in Unix internals, wonder (to use an understatement) about the OS X memory management.

To add injury to the pain, it seems that WIndows 7 memory management system is much clearer (and maybe more efficient). At least the numbers have a meaning as shown in this article. If you go through the gallery you'll see how in a 1GB RAM system the memory allocation changes starting and then quitting a process. Going back to the initial state when no user processes are running. Which is what people expect to see and what doesn't always happen in OS X, stealing at the end RAM for other processes that must be found paging out to the swap files.. So much for the Apple's self defined "best desktop OS in the world" (the GUI, sure. No doubt about it). At this point I wonder how Linux, which is more Unix like, is performing. I wouldn't be suprised if it uses half the RAM running the same open source applications.