Lion - Memory Usage Problems

R C-R wrote:

I suspect a lot of the confusion is the result of considering only part of the kernel environment & how it relates to memory management in OS X.

As explained here, in OS X the kernel is more than the Mach microkernel. Mach provides among other things low level VM support & memory protection. But the BSD & I/O Kit components can't be ignored since for instance the BSD component provides higher level support for both & the I/O Kit provides the kernel’s enforcement of protected memory spaces.

I never said otherwise. It is well known that OS X has an hybrid kernel based on Mach microkernel with BSD components. From a different perspective it is like if Darwin is a BSD kernel where components like the virtual memory have been replaced with the ones from the Mach microkernel and the Unix file based drivers with I/O kit components .

Still all of this is irrelevant like 90% of this discussion, even if somehow instructive.

Let get back on topic: what matters is to answer the original question mightymilk made when he opened this thread: Why is Lion using all 4GB of RAM running Mail, Safari (2 tabs), and iTunes?

Which can be refined using my situation: why in the same scenario but with 3GB of RAM Lion sometime pages out up to 2.7GB in "swap used" size?

Now, do we have an answer? Any theory invoking the concept of any application caching its data, conflicts with the large page outs Lion produces. Here is an example of my memory usage just now. I had to restart Safari because of the paging so the free memory went up. Still:

Where:

Occupata is Wired memory

Attiva is Active memory

Inattiva is Inactive memory

Libera is Free memory

Spazio di swap is Swap Used: 1.84GB out of 3GB of swapfiles allocated.

Can you please explain what in Earth Lion or Safari are "caching" if they produce 1.84GB of page outs? Why the swap used is not taken back? How can Apple sell systems with 2GB of RAM only, if not even 3GB are enough?

How does that fit with your theory that Safari is right using as much memory as possible because "free memory is wasted memory"? I am really curious about this one. The numbers speak for themselves and, sorry, contradict everything you said. Those numbers say that Lion with Safari need at least 5GB of RAM to run without paging out. Many of us think they are far too many and I don't understand how it can be said otherwise.

Michelasso wrote:

Which can be refined using my situation: why in the same scenario but with 3GB of RAM Lion sometime pages out up to 2.7GB in "swap used" size?

"Swap used" is not the same thing as page outs. Cache files & VM swapfiles are two different things. Darwin is not like the BSD kernel with its VM management component replaced with the Mach one -- elements of both are present & interact at different levels. That free memory is unused memory does not imply that a process should use "as much memory as possible."

What a user process caches & its use of swapfile space depends on what a user does & will vary from user to user. If you want to see how a specific process uses memory (real or virtual) you must use the appropriate software tools & understand how to interpret their output. That can't be done without a good understanding of the OS X kernel, the services the OS X frameworks provide to the process, & what the process itself does.

This is not simple stuff. There are hundreds of pages of Apple documents that cover the Apple-specific aspects of OS X in varying amounts of detail & dozens of books that provide in depth discussions of the Mach, BSD, & UNIX foundations it builds on. There is nothing to be gained by trying to ignore the complexity & hoping that a few imprecise generalities & extreme oversimplifications are somehow enough to explain it all.

They aren't.

Either you accept that or you don't.

R C-R wrote:

Michelasso wrote:

Which can be refined using my situation: why in the same scenario but with 3GB of RAM Lion sometime pages out up to 2.7GB in "swap used" size?

"Swap used" is not the same thing as page outs. Cache files & VM swapfiles are two different things.

Really? Then explain what the swap files are supposed to be, because all documentation I've read define them as the backing store. Just check the dynamic_pager man page : The dynamic_pager daemon manages a pool of external swap files which the kernel uses to support demand paging.

The "swap used" is the total amount of data in those files that are larger than that. The application caches are elsewhere, in the ~/Library/Caches to be precise. You keep mentioning these fantomatic caches without ever explaining what you think they do and how.

In any case there were 225.707 "pagine in uscita" (page outs). More than 900MB of pages "swapped" out to the backing store just running Safari. And that has not been my worst scenario. To have the virtual memory system writing out up to 3GB of data in a shot as sometimes it happens, because the system is low in free memory it is called trashing in any 3GB RAM system with any kernel. Whatever that data is and comes from.

To dig more and more into the OS X internals, keeping giving a personal creative interpretation, not only doesn't explain why doing the same activties Lion needs twice the RAM than Snow Leopard, it is wrong. For the simple reason that XNU, BSD, Mach or whatever architectures should be quite similar in Lion and SL. If they aren't then all documents posted are useless.

So, why with Snow Leopard I could run Safari 5.05, Entourage and a 750MB Windows XP virtual machine with 0 page outs for days, while in Lion after a couple of hours sometimes I get massive page outs just running Safari 5.1 and Mail? Why Safari in Lion is using 1 to 3GB of RAM if not even more as many people report?

To say that "it is working as designed" or worse because "Safari is caching to enhance performances" is not an answer.

Michelasso wrote:

To say that "it is working as designed" or worse because "Safari is caching to enhance performances" is not an answer.

Then I guess you don't want an answer.

There may well be bugs in the VM system in the current release, but that doesn't mean it's not working as designed, an answer you don't seem to be willing to accept.

That means either just wait to see what happens in future releases or revert to Snow Leopard.

Don't ask "why" if you're not willing to accept the answer, as even if future releases don't use the VM subsystem as heavily that doesn't necessarily mean Apple fixed a bug, but it may be that Apple simply tuned the VM subsystem differently.

Don't get me wrong; I think the VM subsystem needs a serious rewrite as I think Mach's vision of how it all works was fine for how VM systems worked when introduced, but it's not quite state of the art today compared to the way VM is handled in more mature operating systems like Solaris.

So it's actually designed to slow the system to a crawl? Wow, Apple must be out of their minds.

Michelasso wrote:

Really? Then explain what the swap files are supposed to be, because all documentation I've read define them as the backing store.

As has been mentioned several times, the the backing store is the various swapfile* files, by default in /private/var/vm/, just as the dynamic_pager man page says. But it should be obvious to anyone that (just as you noted) the "Page outs" & "Swap used" size numbers are not the same. Thus, "Swap used" can't possibly be a measure of the amount of data paged out, if for no other reason than "Swap used" is much larger than "Page outs," which is a cumulative measure of all the 4 KB sized page outs since startup. Even if "Swap used" measured just the amount of changed data in each 4 KB page that was paged out (like say just 20 changed bytes in some 4 KB page), it would still have to be smaller than the page out measure if it really represented paged out data.

In fact, if you check /private/var/vm/ with the Finder or any other directory tool you will see that "Swap used" is not even normally the same as the total size of all the swapfile* files.

All Apple tells us about "Swap used" in Reading system memory usage in Activity Monitor is that it is "the amount of information copied to the swap file on your hard drive." Obviously, this article is highly simplified & intended for non-programmers, & for more detailed info about OS X memory management one must consult the developer's link it mentions, the same one mentioned countless times in this discussion.

But if you do that, you won't find any mention of "Swap used" at all (which might suggest its value to programmers is insignificant); however, you will find info about how to discover how much memory actually is associated with any process using the vmmap tool. If you study this info carefully, you will see that the size numbers (there are several of them) represent the virtual memory pages reserved, but not necessarily allocated.

With this clue, it should not be too hard to work out what "swap used" really measures, why it is larger than "Page outs," & in general that <ahem> an overly simplistic view of a complex process is much more likely to confuse rather than enlighten. Simply put, the details really do matter. Ignoring them won't make them go away or any less crucial to understanding what the numbers actually mean.

William Kucharski wrote:

There may well be bugs in the VM system in the current release, but that doesn't mean it's not working as designed, an answer you don't seem to be willing to accept.

I don't follow the logic. If it is bugged (again, "if") it is not working as designed by definition. So why should I (we) accept "it works as designed" as suggested?

To me, there is something wrong in the code. Look, this is my memory usage now, after one hour or so using my system as usual:

Nearly 1GB of RAM free (libera). Which is what I was used to get in Snow Leopard even after 2-3 weeks without rebooting. Starting and quitting Windows virtual machines as well. So why after few hours in Lion sometimes the free ram goes down to 0 and I get gigabytes of RAM's data written out in the swap files? Other people with twice the RAM instead just get the memory fully filled.

Regarding this issue I have seen all over the Internet people suggesting the following:

- Disable the Safari extensions: Which is correct, AdBlock increase indeed the amount of RAM WebProcess uses noticeably.

- Restart Safari once in a while: that frees the RAM indeed

- Kill WebProcess!! Again that frees the RAM after few seconds and restarts. I just wonder how stable it is.

- Run the "purge" or similar command: not really a solution since it should clear just the disk buffer cache and hangs the system for few seconds when executed.But hey, if it works, why not?

- Reboot the system all together: just great. That was one of the reasons why I bought a Mac. Not need to reboot in weeks.

- Last but not least: Downgrade to Snow Leopard.

Which is kind of sad for the "best desktop OS in the world". I don't suggest it, but I can understand the reasons of some people.

R C-R wrote:

As has been mentioned several times, the the backing store is the various swapfile* files, by default in /private/var/vm/, just as the dynamic_pager man page says. But it should be obvious to anyone that (just as you noted) the "Page outs" & "Swap used" size numbers are not the same. Thus, "Swap used" can't possibly be a measure of the amount of data paged out, if for no other reason than "Swap used" is much larger than "Page outs," which is a cumulative measure of all the 4 KB sized page outs since startup. Even if "Swap used" measured just the amount of changed data in each 4 KB page that was paged out (like say just 20 changed bytes in some 4 KB page), it would still have to be smaller than the page out measure if it really represented paged out data.

In fact, if you check /private/var/vm/ with the Finder or any other directory tool you will see that "Swap used" is not even normally the same as the total size of all the swapfile* files.

You want it smaller? I make it smaller! Et Voilà!

14.843 page outs (64MB), 832KB of swap used. How did I make it? I started a virtual machine, I waited for it to raise the swap used value to 130MB and then I logged out. After that the cumulative page outs obviously didn't change and the swap used went back to less than 1MB. I did it without even thinking!

This is what it is stored in /var/vm:

$ ls -l /var/vm

total 6815744

-rw------T 1 root wheel 3221225472 20 Set 10:12 sleepimage

-rw------- 1 root wheel 67108864 20 Set 14:31 swapfile0

-rw------- 1 root wheel 67108864 20 Set 18:22 swapfile1

-rw------- 1 root wheel 134217728 20 Set 18:22 swapfile2

Which matches the 256MB total size reported in iStat Menu.

How does it happen that everytime I try something it fits with my analisys and it always conflicts with everything you say?

You miss the fact that the pages are not written one by one in the backing store, but they are first clustered, when possible, in groups of 4, all adjacent. So when there is a page out we can actually get up to 4 pages written in the backing store. This is what the source codes (yes, I can read the source codes. They are the only real documentation) say:

/*

* vm_pageout_cluster:

*

* Given a page, queue it to the appropriate I/O thread,

* which will page it out and attempt to clean adjacent pages

* in the same operation.

Where the 4 pages MAX per cluster comes from an header file that now I can't find.

As we know "to clean" a page also means to page it out in the backing store if dirty. It's interesting to notice how a very similar concept (and function) has been implemented in BSD Lite:

/*

* Attempt to pageout as many contiguous (to ``m'') dirty pages as possible

* from ``object''. Using information returned from the pager, we assemble

* a sorted list of contiguous dirty pages and feed them to the pager in one

* chunk. Called with paging queues and object locked. Also, object must

* already have a pager. */

void

vm_pageout_cluster(m, object)

When there are no dirty pages it stops. That's the reason why after the first page outs the "swap used" is lower than the multiple of 4 of the page outs number * 4KB.

That is why my orginal numbers that were:

Page outs: 255.707 (about 900 MB)

Swap Used: 1.84GB

make sense since I took the picture just after the kernel started paging out and was still paging. Not like now, to be clear. The swap use is

page outs * 4KB < swap used < 4 * page outs * 4KB

The total size os the swap files is 3GB > 1.84GB the value of swap used. All makes sense.

Honestly, does it take so much to try to think that it may not work as you believe?

Michelasso wrote:

How does it happen that everytime I try something it fits with my analisys and it always conflicts with everything you say?

Because you ignore in your analysis anything that doesn't support your convoluted & frequently changing guesses about how VM actually works in OS X.

In fact, you managed to overlook that you just demonstrated pretty conclusively that "Swap used" is not a measure of the amount of data actually paged out.

Mark Free wrote:

So it's actually designed to slow the system to a crawl? Wow, Apple must be out of their minds.

Nice try, but no.

Performance and design are two different issues which may or may not be related.

The VM subsystem may well be operating as designed, but performance may not be as good as it should be, and that's what VM tuning is all about - it improves performance but doesn't change the underlying code or design.

R C-R wrote:

Because you ignore in your analysis anything that doesn't support your convoluted & frequently changing guesses about how VM actually works in OS X.

In fact, you managed to overlook that you just demonstrated pretty conclusively that "Swap used" is not a measure of the amount of data actually paged out.

Then there must be a communication problem (English is my second language after all). Because I just proved the opposite. I proved that after logging out, and thus killing all user processes the swap used decreases nearly to zero, as it is supposed to do. I give you other news: that document people keep referring to, about how to read the Activity Monitor is plain wrong in some sections. This is what the Activity Monitor help page says:

Which is better explained saying that the inactive memory is memory with valid data that gets moved from the active memory when it is not recently used. The choice to select the new inactive pages is performed by the kernel's LRU (Last Recently Used) algorithm. Actually I believe that OS X uses the NRU (Not Recently Used) algorithm, but that is not that important.

That thing about inactive memory being used to be reassigned to processes that have been killed and then restarted is Science Fiction. There is where you and others probably got the wrong idea about all these caches. The best the kernel can do is to assign to the speculative pages list the process' pages that came from disk reads (not in the backing store). Unless at the time the document was written the kernel was keeping those pages in the inactive list, sure.

Also, and more important there is the definition for

Current Virtual memory [on disk] that as I showed gets written in clusters of up to 4 pages. In all IT literature that means the amount of valid data that is currently paged out from RAM to disk. If it isn't what I said and what is written here then what is it supposed to be?!

Regarding your constant accusation of me generalizing, I'd like to remind you that in IT many things are divided in layers, following the "onion skin" model. If I am thinking at the application level, in this case Safari, I shouldn't be forced to dig into the kernel level. The two things are totally separated, although interfaced, as much as the BSD and Mach modules in the kernel are. Something that you seem to have troubles to believe.

That's why about Safari I agreed that you were right saying that Safari, like any other web browser, is caching in disk the web pages it downloaded. It wasn't clear to me how it worked, so I went checking and I've found the HTML caching mechanism. Good, it makes sense. HMTL must advice for how long a page can be cached and when it has been modified. Still that works for the cache of Internet pages on disk. It is still a mystery why (and how) it keeps copy of all pages on its own memory.

At virtual memory level, instead, I can assure you that the applications have little way to interact with the VM system. The most they can do is to define some flags when asking for virtual (logical) memory. Flags that demand the kernel, or better Mach, to treat that memory in a particular way. Like to have the corresponding RAM pages wired, for example. The BSD VM layer, instead, is just a set of routines needed to provide the system call interfaces as defined in the UNIX and POSIX specification to the applications, in a way that the same application can be recompiled in any Unix system without many modifications. Because what manages the memory underneath must not be a general application concern. The same for I/O kit which doesn't provide the memory protection, which is provided by Mach, but instead it supports it.

Mixing these levels, is what generates the unfamous confusion. It is like discussing about a "spaghetti OS" (terms derived from spaghetti code) where all the structures and modules are intermixed. They are not.

William Kucharski wrote:

The VM subsystem may well be operating as designed, but performance may not be as good as it should be, and that's what VM tuning is all about - it improves performance but doesn't change the underlying code or design.

Ok, let's assume you are right. Seriously, I do not pretend to have the crystal ball. But could you please explain this? I made it in Engish for better understanding:

This happened after I started a virtual machine (I had 1.1GB free), which as you can see generated nearly no page outs, I closed Parallels, Win7 got suspended (writing its image on disk) and paralles eventually quit. The Parallels process wasn't present anymore. It did not release its memory at all.

I saved the Activity Monitor output, and loaded it in Excel. Computed the total real memory used (which should be the larget RAM number). It was 1.5GB. 1GB of RAM simply disappeared. Checked with top, vm_stat. I used "sysdiagnose" as well, which runs "allmemory" among other things. Nothing. I couldn't find it.

Now, in any Unix system (and if XNU stands for X is Not Unix, being OS X Unix 03 certified since Leopard, it should follow that OS X is not XNU anymore...), when a process gets killed its memory gets released. No matter what. Where did it go?

To be clear, after starting Parallels again I've got the system paging out. Up to 2Gb of swap used today (not running anything new. I left the Mac on during the night to update Win7). If the old Parallels memory was cached anywhere it had to be reused. That obviously didn't happen since the kernel paged out a massive number of data. Also Safari became half unsuable, some tabs where not loading pages. after generating a second sysdiagnose report I had the only choice to reboot OS X.

Obviously I updated my bug report with the sysdiagnose outputs as I have been asked by the developers to do as soon as I face another issue. Which means that Apple is not considering this issue a fantasy.

Michelasso wrote:

I proved that after logging out, and thus killing all user processes the swap used decreases nearly to zero, as it is supposed to do.

That does not prove that "Swap used" is a measure of page outs or, to put it another way, that they are somehow interchangeable or equivalent measurements of the same thing. In fact, you have shown that they are two different things. But several times, you have referred to "Swap used" numbers as if they were page out numbers. Examples (emphasis added):

Which can be refined using my situation: why in the same scenario but with 3GB of RAM Lion sometime pages out up to 2.7GB in "swap used" size?

Can you please explain what in Earth Lion or Safari are "caching" if they produce 1.84GB of page outs? (referring to the 1.84GB "Spazio di swap" number shown in iStat)

I don't know how to put it any more clearly than this: "Page outs" shows page outs; "Swap used" does not. Different numbers, different things. "Page outs" shows the cumulative total of page outs (& with some tools, the number of page outs in the last second). "Swap used" shows something entirely different, the amount of swap space currently in use. (Some tools also show the total size of all swap areas or the amount of swap space available.) So when you say something like:

To have the virtual memory system writing out up to 3GB of data in a shot as sometimes it happens ...

and you seem to be referring to up to 3 GB of "Spazio di swap," I have to assume that you are not only confused about the difference between swap space & page outs but also that you are also confused about the difference between the total amount of swap space available & the amount currently in use, and that you think it is all written to in one shot.

Yet later you say that you "took the picture just after the kernel started paging out and was still paging," & start talking about delayed (4 page clustered) writes to the backing store, which contradicts the "one shot" idea. You then do some math that clearly shows "Page outs" & "Swap used" are not the same thing, & conclude that your analysis somehow proves that they are, or at least that they show equivalent information, using some impossible to follow logic that seems more confused & convoluted than ever.

You frequently complain that the documentation is wrong, but this seems mostly the result of trying to interpret it as saying something it does not, usually in an attempt to make something seem simpler than it really is. You tend to skip over important details, substitute vague terms for precise ones, & rely too heavily on superficial & often inappropriate comparisons.

You do the same with comments you get from other users, for instance by interpreting detailed explanations of memory use as saying processes should use as much memory as possible, or references to the 3 components of the OS X kernel as saying it is like BSD with some parts replaced by Mach ones.

In short, the info is there. It just doesn't say what you want it to.

R C-R wrote:

Michelasso wrote:

I proved that after logging out, and thus killing all user processes the swap used decreases nearly to zero, as it is supposed to do.

That does not prove that "Swap used" is a measure of page outs or, to put it another way, that they are somehow interchangeable or equivalent measurements of the same thing. In fact, you have shown that they are two different things. But several times, you have referred to "Swap used" numbers as if they were page out numbers. Examples (emphasis added):

Which can be refined using my situation: why in the same scenario but with 3GB of RAM Lion sometime pages out up to 2.7GB in "swap used" size?

Can you please explain what in Earth Lion or Safari are "caching" if they produce 1.84GB of page outs? (referring to the 1.84GB "Spazio di swap" number shown in iStat)

I don't know how to put it any more clearly than this: "Page outs" shows page outs; "Swap used" does not. Different numbers, different things. "Page outs" shows the cumulative total of page outs (& with some tools, the number of page outs in the last second). "Swap used" shows something entirely different, the amount of swap space currently in use. (Some tools also show the total size of all swap areas or the amount of swap space available.)

Indeed. I still stand correct: page outs shows the number of times the paging out process has been called since boot time multiplied by the page size (4KB). Where for each page out call 1 to 4 pages get paged out in a cluster. Swap Used is, according to the Activity Monitor help page itself, the amount of disk space currenty used as virtual memory. Which it can't be anything else than the total of RAM pages currently swapped out to disk. And yes, that area can be bigger than the total RAM, because the same pages can be reused and paged out more times.

As I wrote already, the confusion lies elsewhere. That Apple document is wrong, it doesn't say the same of the Activity Monitor help page, underlined above.

Then you mentioned some tools. Good. Please show me which one because I can't find anything. In any other Unix system (because OS X is Unix 03 certified, isn't it? So much for XNU stand for "X is Not Unix") to effectively check the swap details it is as easy as running the command

$ swap -s

or similar depending if it is HP/UX, Solaris, AIX or even Linux, which gives an output like:

swapfile dev swaplo blocks free

/dev/dsk/c0d0s1 102,1 8 1220928 1220928

in OS X we are left with that dumb tool called Activity Monitor that doesn't even have a correct documentation published in the Internet.

It would be also nice if you would give your idea about what "swap used" means. I asked you already: f it isn't the total amount of data currently pafed out from RAM to disk, what else is it supposed to be?

R C-R wrote:

Yet later you say that you "took the picture just after the kernel started paging out and was still paging," & start talking about delayed (4 page clustered) writes to the backing store, which contradicts the "one shot" idea.

That is the problem, you always see contradiptions in anything that doesn't fit with your idea. The process is simple and I never mentioned any delay:

- One or more processes in the system keep requesting more and more physical memory

- The page out process selects clusters of pages, with 1 to 4 pages per cluster, to be swapped out

- These clusters get put into the I/O buffer for disk writing

- The clusters in the I/O buffer gets written into the swapfiles on disk. This is done in parallel with the page out process execution.

- Rinse and repeat

By the time a chunk of data gets written out to the swap files the page out process could put again in the queue hundreds of times the same amount of data. As soon as a chuck of data gets written on disk there are already many others waiting for their turn to be swapped out. That's why the disk activity, which I monitor with iStat menus, when thrashing is constantly nearly at the max. And that's why a trashing slows down the whole system, because processes may get put on wait until some memory has been freed and that doesn't happen until the previous chuck of memory has been effectively paged out.

These are few of the first basic IT principles that one gets to study in the Computer Science classes at University. It doesn't matter if it is VAX VMS, Unix, IBM OS or whatever. Principles that one should study consistently from books in a propaedeutic order, and not from collages of pieces taken here and there. As for man pages the documents one may find are just good later as references. They can't be and are not a substitute for a serious studying of the subject. Otherwise, what one gets in his or her head is just Confusion.