Lion - Memory Usage Problems

Mac_Boston wrote:

Admittedly I can't distinguish the start up disk from the RAM in this discussion - seems like the same thing.

Please believe me. They are very different things.

Real (physical) memory is RAM, the memory chips in your Mac that store code that is executed & data that is manipulated by that code when processes are run by the OS or applications. The contents of RAM is volatile -- it is not retained between startups & must be loaded from disk storage or created by the code loaded from it anew on every startup.

Virtual memory (VM) exists only in temporary files stored on the disk. It is also volatile -- any data or code it stores must also be recreated on each startup. The OS swaps out chunks of real memory to VM to make room for the real memory needs of running processes. In this sense only are they similar -- they hold the same kinds of things but are used very differently. To simplify a bit, RAM is working space; VM is storage space dedicated to the exclusive use of that working space.

But the drive also stores other types of temporary files besides those used for VM, for instance buffers applications might create to temporarily assemble & store large amounts of data that don't need to be accessed all at once. This conserves memory space.

It also stores many kinds of permanent (non-volatile) files, including application files, various kinds of application support files, user document files, preference files, font files, databases, logs, & so on. Some of these files are created or updated as needed by the OS or an application. Some are used by several different apps, some aren't.

So even if a user loads or creates no document files, there will be a large number of files besides program (application) files on the disk & some will change in size or content. This is true even if a user never opens an application because the OS runs processes of its own even when a user does not. (In fact, since the Finder, Terminal, Activity Monitor, etc. are all applications, you can't do much of anything besides stare at the machine without running an application.)

Thus, when you say, "The only thing that could have been taking up storage space were the loaded programs (not their processing but just the programs themselves and their content)," your assumption is incorrect. Likewise, there is no way to keep a machine "totally naked" even if you keep all your user files on another drive.

I'm not sure exactly what the Apple people were looking for, but your description of it doesn't make much sense, at least from a technical standpoint.

PS. I too am getting every response to everyone on this chain ...

If you mean an email notification every time someone adds a reply to this discussion, that's normal. You won't get just the ones in response to your posts. It's all or nothing in that respect.

Virtual memory (VM) exists only in temporary files stored on the disk. It is also volatile -- any data or code it stores must also be recreated on each startup. The OS swaps out chunks of real memory to VM to make room for the real memory needs of running processes. In this sense only are they similar -- they hold the same kinds of things but are used very differently.

It seems to me like you yourself is just completely confused about what VM is. http://en.wikipedia.org/wiki/Virtual_memory please check here if you really interested. In short VM is address space which belong to application. This address space could be mapped to physical ram or paged to disk. Applications can share parts of their VMs - and that would be shared memory (normally used for inter-process communication), also all applications can have access to shared libraries through VM, so one library would be loaded only one time and then shared among the all application which use it. Part of VM address space of each process is reserved for system use.

R C-R wrote:

Because the purge command doesn't purge inactive memory. It just purges the disk cache. Remember, this simulates a 'cold' (empty) cache. That in turn forces the OS to rebuild the caches as needed to support all running processes. Also remember that there is never just one process running, but a mix of active & inactive ones. That's why it takes time.

Dude that's just not true, look at my post and watch at the end of the movie when I run purge (~12secs) INactive is turned Free and the VM size stays constant (VM being disk right...!?)

If anyone would bother reading my post and stop haggling over sematics, you'd see that the VM goes down as I quit apps, and then stays steady (along with Page Outs) as I try to exhaust memory. Inactive does give back, but seems to incur a performance hit.

But saying that purge doesn't free inactive is just plain wrong, have you ever run the command?

BlackNova wrote:

In short VM is address space which belong to application.

Yes, that is one way to describe it. However, do you think that is an adequate way to describe the differences between RAM & VM files stored on a drive to someone who thinks they are the same thing?

Message was edited by: R C-R

Joel Bruner1 wrote:

But saying that purge doesn't free inactive is just plain wrong, have you ever run the command?

Read more carefully. I did not say that purge doesn't result in inactive memory being freed, just that it isn't what the command does. Do you understand the difference?

R C-R wrote:

At this point what's the correlation between the resident memory and wired/active/inactive memory isn't given to know.

Active, inactive, wired, & free memory is allocted to real (or if you prefer resident) memory according to complex algorithms that are well documented but not easy to understand. If you are interested in that, study the developer document that has been mentioned here many, many times.

Yeah, sure. Let's always blame the complexity of the system. The question here is clear: why the total amount of resident memory is smaller than the active memory. There is no need to go into the alghoritms internals to answer what those numbers really represent. And the public Apple documentation not only doesnt answer that question it can also be plain wrong like the unfamous KB about Activity Monitor that everybody keep quoting as if they were parrots.

R C-R wrote:

Joel Bruner1 wrote:

But saying that purge doesn't free inactive is just plain wrong, have you ever run the command?

Read more carefully. I did not say that purge doesn't result in inactive memory being freed, just that it isn't what the command does. Do you understand the difference?

Actually you did:

R C-R wrote:

Because the purge command doesn't purge inactive memory. It just purges the disk cache.

----

It seems you are just repeating what the purge man page says:

purge(8) BSD System Manager's Manual purge(8)

NAME

purge -- force disk cache to be purged (flushed and emptied)

But this is what purge actually does:

BlueMoon:~ Michele$ vm_stat

Mach Virtual Memory Statistics: (page size of 4096 bytes)

Pages free: 13503.

Pages active: 440753.

Pages inactive: 233184.

Pages speculative: 12757.

Pages wired down: 85969.

"Translation faults": 326493862.

Pages copy-on-write: 1474994.

Pages zero filled: 164106396.

Pages reactivated: 471443.

Pageins: 441819.

Pageouts: 130014.

Object cache: 19 hits of 85259 lookups (0% hit rate)

BlueMoon:~ Michele$ purge

BlueMoon:~ Michele$ vm_stat

Mach Virtual Memory Statistics: (page size of 4096 bytes)

Pages free: 247750.

Pages active: 359556.

Pages inactive: 72795.

Pages speculative: 17621.

Pages wired down: 88403.

"Translation faults": 326536168.

Pages copy-on-write: 1478702.

Pages zero filled: 164122179.

Pages reactivated: 471443.

Pageins: 447289.

Pageouts: 130014.

Object cache: 19 hits of 85335 lookups (0% hit rate)

Funny enough, the number of speculative pages (the disk cache indeed) raised, while the number of active (about 320MB) and inactive pages decreased.

Thus, again, it's either the purge man page wrong or the active memory (other than the inactive memory, obviously) keeps part of the disk cache. That would partially explain why the resident memory is smaller than the active one. But it would contradict the paradigm that disk cache is used to reduce the number of accesses to disk, since when the free memory gets low (also due to the disk cache) the kernel starts paging. Like if it is paging out the cache. Nonsense.

Michelasso wrote:

The question here is clear: why the total amount of resident memory is smaller than the active memory.

Because RSIZE isn't quite what you think it is. If you don't care about the complexities of how it's computed there is no point in saying more than that. If you are willing to consider them, this discussion might be of interest to you.

R C-R wrote:

Michelasso wrote:

The question here is clear: why the total amount of resident memory is smaller than the active memory.

Because RSIZE isn't quite what you think it is. If you don't care about the complexities of how it's computed there is no point in saying more than that. If you are willing to consider them, this discussion might be of interest to you.

Interesting reading. Basically the author, whom I suspect is an Apple developer, is confirming that RSIZE is the Resident Set Size, in other words the amount of RAM that is mapped (for use by the MMU- The "real memory" used indeed) and that its value for each process is obtained true a call to the (Mach) system call task_info.

He's is also confirming that RSIZE should be equal to RPRVT+RSHRD. These last two values are computed walking through the page list and thus they are an approximation. It isn't very clear if RSIZE is an approximation itself. At least for the total value of RSIZE the kernel should have a clear idea.

Still that doesn't answer the question. Yesterday just to check a couple of things I started a virtual machine, then Chrome and other stuff just to raise the used memory. At the end instead of rebooting (as I always do, even if I shouldn't) I just logged out disabling Resume and logged back in, starting only Terminal to run top. This is what I've got:

MemRegions: 10709 total, 400M resident, 42M private, 139M shared.

PhysMem: 330M wired, 1671M active, 320M inactive, 2321M used, 750M free.

The Active memory was 4 times the Resident memory.Since only few user processes were running, as it is normal after logging out, the Resident memory value is the one making more sense than the Active memory one. With in any case is out of proportion, because as we well know

the Active memory is the memory currently in use by applications and processes

Someone should explain which processes were supposed to eat up all that memory since few were running. The Active memory (and so the Resident memory) still raised a couple of hundreds megabytes after launching Safari. Thus that wasn't a cache or similar either. And in any case that was supposed to be stored in the Inactive memory, not the active one.

These are the numbers the system is giving. Quoting another author (Mike Smith. He replied in the discussion as if he was another developer):

it is documented by the (freely available) source code and the activity of your system.

Well, the activity of my (and others) system is showing that for some reasons OS X likes to retain a lot of memory without making much use of it.

You could see which process is holding that memory, if you run

top -o rsize

That will sort the top display by the rsie of each process. There are number of useful options to top, see

man top

You need to remember, that OS X is an multiuser, multitasking UNIX OS. As such, it does not matter much if you 'logout', as that is likely to only stop 'your' processes, and under certain settings even that is not sure.

Michelasso wrote:

Basically the author, whom I suspect is an Apple developer, is confirming that RSIZE is the Resident Set Size, in other words the amount of RAM that is mapped (for use by the MMU- The "real memory" used indeed) and that its value for each process is obtained true a call to the (Mach) system call task_info.

I don't understand how you came to that conclusion. He says the resident memory size indicates how many valid page table entries there are for the process's current use of its virtual address space, that the number is a "reasonable approximation" of the pages immediately in use by a process or available for use by the process without faulting the MMU.

He's is also confirming that RSIZE should be equal to RPRVT+RSHRD.

Where does he say that? Among other things he says that "Shared region pmaps are a part of every pmap that contains them, so there is no way to distinguish page-table entries from one process to another," that "Private and shared memory sizes are an even greater approximation than the resident memory size," & that "it is almost never going to be the case that they add up to each other - even in the simple cases with no sharing and mapping in entire objects."

It isn't very clear if RSIZE is an approximation itself. At least for the total value of RSIZE the kernel should have a clear idea.

I don't understand how can not be perfectly clear from the above that it is an approximation, or for that matter that RSIZE is meaningful only on a per-process basis, not for a total of them.

If it isn't clear enough from that, how much clearer could it possibly be than Mike Smith's statement that, "Top is not telling you, except in the crudest sense, how much real memory your process uses"?

Have you at least browsed through the developer documentation that has been mentioned here so many times now? I understand the desire to avoid the complexities of the subject if possible but if you want to understand it in any depth you really do need to study that documentation.

R C-R wrote:

It isn't very clear if RSIZE is an approximation itself. At least for the total value of RSIZE the kernel should have a clear idea.

I don't understand how can not be perfectly clear from the above that it is an approximation, or for that matter that RSIZE is meaningful only on a per-process basis, not for a total of them.

If it isn't clear enough from that, how much clearer could it possibly be than Mike Smith's statement that, "Top is not telling you, except in the crudest sense, how much real memory your process uses"?

Because I was talking about the total value indeed, the one that top reports as "resident". To be compared with the Active memory. That value is needed by the memory management alghoritm so it should be easily accessible. Or at least so it used to be. Also it must be lower than the sum of all RSIZE, since they include the memory shared by more processes. Anyway this is what i was referring to:

1. Resident Memory size: this counts (to a first approximation) physical pages that are currently mapped into the task address space.

2. Private Memory size: this counts (to a first approximation) pages that are private to the task (not shared with other tasks)

3. Shared Memory size: this counts (to a very coarse approximation due to several factors) pages that are shared with one or more other tasks in the system.

1 counts physical pages. I am assuming that with 2 and 3 you are referring to the RSHRD and RPRVT fields, which are resident shared and resident private pages.

You might more reasonably assume that 1 = 2 + 3, but you'd still be bitten by the fact that things are not counted in quite the way you expect.

(sorry, it had been Mike Smith saying that. I confused the two posts). Again, everything is just vague. It isn't possible to find a single picture that explains the OS X memory management module. because, to be clear once for all, I DID read the Apple documentation you suggested. That doesn't explain why in 10.7.0 the Inactive memory was always much bigger than the Active one and in 10.7.3 it is the other way around.

As it doesn't answer the question "why the Active memory after a logout is 4 times the resident set size (rss) and twice the memory after the first login". Wherever I looked no one has been able to give an answe, not even the two devs in that discussion. And the documentation you keep suggesting is outdated, it doesn't represent what the memory management algorithm, which supposely changes with every new OS X update (if only just to fix bugs), really does.

Personally I want those numbers explained. I don't want to read the internals. I want to know where those numbers come from, how the RAM gets allocated in term os rss for the processes, wired, active and intactive memory.

And one last thing: an approximation is something that differs 5%, 10% the most from the real value. Anything above that is a random number.

dkalchev wrote:

You could see which process is holding that memory, if you run

top -o rsize

That will sort the top display by the rsie of each process. There are number of useful options to top, see

man top

You need to remember, that OS X is an multiuser, multitasking UNIX OS. As such, it does not matter much if you 'logout', as that is likely to only stop 'your' processes, and under certain settings even that is not sure.

Or, since top can't list all processes since its output has to fit into the window, I can run the command

ps -u <UID> -o user,rss,command

for getting all user processes with their memory usage (rss) running on the system. Which actually I did. Together with

ps -eo user,rss,command

for listing all processes and their memory usage. See

man ps

for better references. In any case if nothing is running in the background all processes started by the user when logging out must quit.

Michelasso wrote:

Because I was talking about the total value indeed, the one that top reports as "resident".

What specifically is the name of what you see in top that reports that total, other than the per process totals (IOW, other than RSIZE)?

Again, everything is just vague.

It isn't vague. It's just so complicated that you cannot assume anything means anything other than exactly how it is defined in those internals you don't want to read.

This is really driving me nuts. I am seeing more colored spin on my IMAC 2011 with I5 processor than my 2009 macbook pro. Everytime i try out Chrome which takes much less memory for sure but i am forced back into using Safari since some websites would invariably crash in chrome ; its plain and simple not a very consistent browser specially for mac.

Safari starts of well but quite fast starts hogging memory..

Well i do not use Final Cut Pro and currently even my Iphot is not turned on.

Go figure U consumer i guess is the attitude with Apple !!!!