Lion - Memory Usage Problems

Krishh wrote:

To conclude inactive memory management is something not in our hands. In other words I would have had to wait forever for it to be freed up. Right ?

It isn't quite that cut & dried. Consider the following (somewhat oversimplified for the sake of brevity):

• The purpose of inactive memory is to retain data (including executable code) in memory that isn't currently being used but might be used again. This improves performance because otherwise the data would have to be fetched from the drive.

• Inactive memory won't immediately be released (freed up) until some process needs it. This may require writing some data that has been changed while in RAM back to the drive, causing a delay proportional to the amount of data that has to be written. If the delay is long enough, you get the spinning beach ball "wait" cursor.

• Inactive memory is very granular, meaning it is divided up into small chunks (pages) that can be processed individually or in groups. Some of it may contain data or code that has recently been used by more than one process (IOW, shared memory). Thus, even when you quit an app not all the inactive memory it has recently used may be freed up.

• Chunks of freed up memory will immediately be converted to active memory when some active process needs it, but very quickly thereafter may partially or completely become inactive memory. Activity Monitor only samples (updates) memory use information periodically (between 0.5 & 5 seconds, depending on its View menu > update frequency setting). Because of this, it may look like inactive memory use isn't changing much or at all when it actually is changing quite a bit.

Note: to get an idea of how quickly processes can switch between active & inactive states, set Activity Monitor's update frequency to 0.5 seconds & then select either "Active processes" or "Inactive processes" from the 'show' popup in its toolbar.

• Remember that it is impossible to quit every process. Even if you quit every application, there are many "faceless" (meaning no user interface) processes always running in the background in both the system & user domains. An "application" is basically just a process with a "face" (a user interface). In Activity Monitor, applications have an application icon preceding the process name; the other processes do not. Plus, some application processes need faceless 'daughter' processes to do some things & these are started up when needed. Examples include the Safari Web Content & VTDecoderXPCService (used by Quicktime Player in Lion). All these processes become active or inactive (& use varying amounts of active & inactive memory) as determined by the OS & their code.

• All this must be taken into account when trying to determine how much memory really is required at any particular time. Basically, it means you can't really consider apps individually -- they all rely on system processes & share memory to some extent. So while you can't control inactive memory use, you do not have to wait forever for it to be freed up. In fact, sometimes it is being freed up, made active, & then made inactive again faster than you can follow.

R C-R wrote:

• Chunks of freed up memory will immediately be converted to active memory when some active process needs it, but very quickly thereafter may partially or completely become inactive memory. Activity Monitor only samples (updates) memory use information periodically (between 0.5 & 5 seconds, depending on its View menu > update frequency setting). Because of this, it may look like inactive memory use isn't changing much or at all when it actually is changing quite a bit.

Note: to get an idea of how quickly processes can switch between active & inactive states, set Activity Monitor's update frequency to 0.5 seconds & then select either "Active processes" or "Inactive processes" from the 'show' popup in its toolbar.

There isn't really much correlation between the active processes in Activity Monitor and the active memory. Sure, if a process is always inactive its active memory should become a candidate for the inactive memory because it doesn't get referenced, but as long as there is enough free memory the kernel doesn't seem to bother.

If you wanna have fun try to do the same with "top" in Terminal. Press "s0" follow by ENTER which means top will update as fast as it can, and order the processes based on the real memory (write "orsize" ENTER). You'll see how in the header the values for the memory will slightly change if they'll change at all. Same for the processes' memory values, while the CPU utilization and other values will keep changing very fast. Also one can notice how still using CPU time some processes keep being shown as "sleeping" (which I suppose is the same of inactive in Activity Monitor) making me wonder how effective it is to report their status.

To make things more interesting you can have Safari playing a video. The result is about the same.

Michelasso wrote:

If you wanna have fun try to do the same with "top" in Terminal. Press "s0" follow by ENTER which means top will update as fast as it can, and order the processes based on the real memory (write "orsize" ENTER).

The most significant thing to me with top set to the minimum possible delay between updates is how much CPU usage this consumes (around 100% of one core on my system when no other CPU intensive tasks are running). This just about guarantees it isn't really measuring normal memory use & illustrates the significance of the "PERFORMANCE vs. ACCURACY" comment in top's man page.

Starting top with the -i 1 option (not too surprisingly, it doesn't seem to be supported in interactive mode) doesn't seem to change things much under these conditions, although it does very slightly improve the sleeping/running stats.

Also one can notice how still using CPU time some processes keep being shown as "sleeping" (which I suppose is the same of inactive in Activity Monitor) making me wonder how effective it is to report their status.

Keep in mind that with top updating this fast it is easy to miss a process very briefly switching to the running & then back to the sleeping state. But I do see that happening for every process whose CPU time increases, at least with the interval option set to every sample.

R C-R wrote:

The most significant thing to me with top set to the minimum possible delay between updates is how much CPU usage this consumes (around 100% of one core on my system when no other CPU intensive tasks are running). This just about guarantees it isn't really measuring normal memory use & illustrates the significance of the "PERFORMANCE vs. ACCURACY" comment in top's man page..

True, that's why I talked about "having fun", it doesn't pretend to be accurate. But that happens because all other processes are practically idle. I started a virtual machine later, which is CPU and memory intensive and the CPU usage got more or less split. Which is what the time sharing is for after all. But that reduce the refresh interval of top. Obviously we can't never win.

Still if the processes are mostly idle there shouldn't be much to do for the memory management system. Their memory has been allocated and there it stays. Unchanged on their wired/active/inactive status. What Windows 7 does for example is to free the memory that hasn't been used for some time. OS X doesn't seem to care. Which can be right, why to do extra work if not needed. My point is that the OS X kernel is not there worrying all the time about moving chunks of memory from here to there, unless there is one or more applications requesting more and more memory. Which is not the usual case in our end user systems. Most of the time they do just nothing.

So I tested it running a memory hungry program (the usual virtual machine in my case). One would expect the wired and/or active memory stealing RAM from the free memory, until the second gets low. At that point the memory management algorithm should kick, freeing memory from the inactive memory, swapping to disk the written pages. The wired and active memory would get the newly freed pages and some active memory would become inactive. Since the inactive memory stayed constant (few hundreds megabytes) the best I can think to make sense out of it is that a chunk with the same exact amount of memory gets freed from the inactive memory and another gets changed from active to inactive. Personally I expected the inactive memory becoming near zero in size before to have any page out and that, as we know, doesn't happen. It pages out with about 500MB of Inactive memory, more than 1GB in 10.7.0. Tested many, many times.

But that has been repeated far too long. One thing I really don't understand is how running the 32 bits kernel I can have System Preferences and another process (compiled by myself. But that shouldn't matter) reporting with top a VSIZE of 18GB. That has been noted by the devs in my bug report and I was obviously expecting a fix (either top is bugged or something is very wrong for the dev own admission. The 32 bits kernel can address up to 4GB of memory). Nothing changed.

Windows 7 and UNIX memory management is very different. As far as I am aware, VM settings in OS X are also quite different from other UNIX systems as well. So there is no point comparing them, at all.

But I see one flaw in your experiments -- you use an PC emulation program for emulating 'heavy demand for memory'. While this type of application indeed has heavy demand on memory, it's demands on memory have nothing to do with most other applications, for at least these reasons:

- PC emulation software always requests non-swappable memory. That will typically go to the 'wired' memory but it seems OS X accounts it for in the 'active' memory. Such application should never use swap and generally fail if there is not enough non-swappable memory available.

- PC emulation software is likely to grab much memory initially, sometimes as much as you have configured for the VM (depends on settings). This means no memory regions fragmentation.

- UNIX VM in general likes to touch swap, even if you have plenty of RAM. This is for many reasons, none of which is related to the topic. I didn't see "don't use any swap" sysctl on OS X, but this setting is available on FreeBSD for example and probably I didn't look hard enough.

It is not true, that 32bit kernels can address only 4GB memory. Architectures, such as PAE were available on Intel processors long before AMD invented the current 64-bit model. Also, UNIX operating systems were able to manage more than 4GB virtual memory (physical RAM, plus swap) for ages.

By the way, by default OS X Lion will encrypt the swap. If your computer is not fast enough, or you are concerned with the extra processing this takes and do not care if your swap is encrypted (like it's not on a laptop), then you can disable it by:

sudo defaults write /Library/Preferences/com.apple.virtualMemory UseEncryptedSwap -boolean no

sudo defaults write /Library/Preferences/com.apple.virtualMemory DisableEncryptedSwap -boolean yes

After reboot verify it's disabled with

sysctl vm.swapusage

That will provide some improvement...

dkalchev wrote:

It is not true, that 32bit kernels can address only 4GB memory. Architectures, such as PAE were available on Intel processors long before AMD invented the current 64-bit model. Also, UNIX operating systems were able to manage more than 4GB virtual memory (physical RAM, plus swap) for ages.

There is a lot of confusion about the limitations of the OS X 32 bit kernel. The key limitations are that the 32 bit kernel can only be used with 32 bit kernel extensions (so for instance you can't use 64 bit drivers with it) & that it can only address 4 GB per process. But the 32 bit kernel can run a mix of 32 & 64 bit apps, and each could (if you had enough RAM) use up to 4 GB of RAM.

I agree that the virtualizers seem to allocate all the virtual machine memory as wired memory (I still wonder why. There is not any possible deadlock since even if part of the Windows kernel is paged out by OS X - which would probably seldom happen if happening at all being often used - OS X will give it back once requested) and that makes this a special situation. Still if it allocated 1GB of wired memory it can end up with 2GB of paged out memory. Plus the 1GB RAM that was free already. It seems a bit too much to me. What was interesting is that the inactive memory doesn't seem to decrease when the free memory reaches the low water mark limit. My assumption has always been that it had to go to near zero, freeing the read only pages (the ones not requiring a page swap) first. But anyway..

If you want to disable the virtual memory in OS X it is pretty easy. Run in Terminal:

sudo launchctl unload -w /System/Library/LaunchDaemons/com.apple.dynamic_pager.plist

To re-enamble it just run:

sudo launchctl load -w /System/Library/LaunchDaemons/com.apple.dynamic_pager.plist

I did it few times for testing. The good news is that it never crashed. The bad one is that it obviously reaches a point were it keeps releasing pages allocating others, entering a loop that nearly hangs the system. A kind of thrashing with no backing storage involved.

Regarding the 18GB (why 18 and not, I don't know, 16 anyway?) of address space for System Preferences I honestly don't have a clue. System Preferences is a very small application. Still sometimes it reaches 200+MB in real memory size. And it's the only application (plus the self-compiled daemon I made) that has a virtual space above the 4GB. Also it was my understanding (but I may be wrong. So I understood) that in the OS X 32 bits kernels the single applications could never have an addressable space bigger than 4GB. That doesn't limit the total size of the backing storage to 4GB though.

In any case still not being happy with Lion memory management (3,221,225,472 bytes is an awful lot of memory for just running the OS, Mail, a web browser and few small applications and daemons. Still it can easily page out 1GB or more memory) I must say that at least compared to 10.7.0 the paging out is less intrusive. With 10.7.0 I had to reboot my Mac twice a day because it was getting unstable. But sometime a heavy paging out activity is there and to me that could/should be optimized.

Michelasso wrote:

What was interesting is that the inactive memory doesn't seem to decrease when the free memory reaches the low water mark limit. My assumption has always been that it had to go to near zero, freeing the read only pages (the ones not requiring a page swap) first.

How this works is explained in detail in Page Lists in the Kernel & Paging Out Process. Note that a pager object must exist that references the page before the kernel can do anything with it. Pager objects have their own memory overhead & are essential in handling page faults. Page faults are what cause page ins & thus are the initiating cause of free memory dropping below the low water mark threshold (which varies depending on the amount of RAM installed).

Page ins can come from the backing store (handled by the default pager) or from regular memory-mapped files in the file system (handled by the vnode pager).

So you have to look at what is causing page ins to see why the memory is hitting the low water mark. If you start there, it may help you understand what's going on with the inactive to free to active transitions that you can't figure out.

R C-R wrote:

How this works is explained in detail in Page Lists in the Kernel & Paging Out Process. Note that a pager object must exist that references the page before the kernel can do anything with it. Pager objects have their own memory overhead & are essential in handling page faults. Page faults are what cause page ins & thus are the initiating cause of free memory dropping below the low water mark threshold (which varies depending on the amount of RAM installed).

I know, I have read those documents a hundred times. The general idea like in any VM system is to make RAM pages available to processes that require it when the free memory is low. One thing the documentation doesn't explain is why OS X doesn't prioritize the assignment to free memory of the unmodified pages, delaying/avoiding the writing to the backing store (thus saving disk writing operations that are expensive). Also it doesn't explain what's the need of keeping a big Inactive memory list (even 1/5 of the total RAM in my configuration) while it is happily paging out. When the Free mmeory gets low the pages move in a loop like this:

Active Memory -> Inactive Memory -> Free Memory (some written to the backing storage) -> Active or Wired Memory

Sure, with this model the Inactive memory size can stay constant. The number of pages freed can be exactly the same number of pages coming from the Active memory list. But since the backing storage grows a lot it looks like most if not all pages freed were modified, not simply discardable to be read or recreated later.

In any case none of them explains what made the Inactive Memory rasing to 14GB or so with only a couple of applications running like in that user situation. One thing that should be sure is that once a process quit its private memory must be freed.

Michelasso wrote:

In any case none of them explains what made the Inactive Memory rasing to 14GB or so with only a couple of applications running like in that user situation. One thing that should be sure is that once a process quit its private memory must be freed.

It seems you have to read that part of the documentation again. It says explicitly, that with OS X you have several models of allocating and deallocating memory. As long as we have no clue how that particular program was written (and we already know why it had 14GB inactive memory), we cannot say for certain why it was not freed. For example, all that memory may have been allocated from the buffer space, in which case it does not really belong to the program and will not be "freed" as soon as the program terminates.

One of the things that Apple has tuned with OS X is the ability of the buffer memory to grow arbitrarily large. Unless otheriwse modified, the traditional UNIX model is to let buffer memory grow up to 10% of RAM.

dkalchev wrote:

One of the things that Apple has tuned with OS X is the ability of the buffer memory to grow arbitrarily large. Unless otheriwse modified, the traditional UNIX model is to let buffer memory grow up to 10% of RAM.

I am not sure what you mean with "buffer memory". If it is the disk cache that should be in the speculative memory (which is part of the free memory), as shown in vm_stat:

bash-3.2# vm_stat

Mach Virtual Memory Statistics: (page size of 4096 bytes)

[...]

Pages speculative: 2719.

Please note that I couldn't find any Apple document explain what the "Pages speculative" are. Not even the vm_stat man page. I had to dig into the OS X open source code to find it out. In other words the Apple documentation is not complete, if reliable at all. For example these are the real and only conditions when the pageout daemon will wake up (I just doubled checked now the code, so..):

/*

* Decide if we should poke the pageout daemon.

* We do this if the free count is less than the low

* water mark, or if the free count is less than the high

* water mark (but above the low water mark) and the inactive

* count is less than its target.

*

* We don't have the counts locked ... if they change a little,

* it doesn't really matter.

*/

if ((vm_page_free_count < vm_page_free_min) ||

((vm_page_free_count < vm_page_free_target) &&

((vm_page_inactive_count + vm_page_speculative_count) < vm_page_inactive_min)))

thread_wakeup((event_t) &vm_page_free_wanted);

Which is clearly different and more complex than what the available Apple documentation says. The pageout daemon stops moving pages from the active to the inactive queue when:

(vm_page_inactive_count + vm_page_speculative_count) >= vm_page_inactive_target

where without copying & pasting more code vm_page_inactive_target is 1/3 of the active+inactive+speculative number of pages. Thus on first approximation the inactive memory must be greater than half the active memory after the free memory gets too low. This to improve the efficiency of the LRU like algorithm apparently.

Oh well, at least it explains why the inactive memory never goes down to near zero.

Michelasso wrote:

One thing the documentation doesn't explain is why OS X doesn't prioritize the assignment to free memory of the unmodified pages, delaying/avoiding the writing to the backing store (thus saving disk writing operations that are expensive).

Consider that it isn't as simple as just Active Memory -> Inactive Memory -> Free Memory. It usually (we hope) is Active Memory -> Inactive Memory -> (soft fault) -> Active Memory -> Inactive memory, perhaps repeating several times before a page must be freed up on a hard fault to make room for a new active page.

IOW, it is a process driven directly by page faults, & only indirectly by the amount of free or inactive memory. Only when a fault occurs does the kernel need to go through the relevant VM object’s list of resident pages & mark any of its pages as modified (as discussed in the Paging In Process section).

Also consider that as well as VM objects mapping to the default pager, they may be mapped to the vnode pager, or to other VM objects (as discussed in the Details of the Virtual Memory System section).

So basically, the kernel has to do all this this VM object/pager object housekeeping stuff whether or not it prioritized unmodified pages, & there isn't any particular benefit in doing that.

I am not sure what you mean with "buffer memory".

Refer to the Wired Memory section. Wired memory includes virtual memory & I/O buffer caches, some of which the kernel has allocated to support application, framework, & other user domain software, & may not be released to the free memory pool even when that software quits.

The bottom line is pretty much that the kernel tries to make memory management as inexpensive as it possibly can, in large part by not touching any memory objects more often than is really necessary. If there was some way for it to look into the future to see if or when some object might need to be reused, it could do a better job of this, but since that isn't possible, all it can do is not release them unnecessarily, which would (at least in theory) end up being more expensive overall.

R C-R wrote:

Consider that it isn't as simple as just Active Memory -> Inactive Memory -> Free Memory. It usually (we hope) is Active Memory -> Inactive Memory -> (soft fault) -> Active Memory -> Inactive memory, perhaps repeating several times before a page must be freed up on a hard fault to make room for a new active page.

Sorry, but Active Memory -> Inactive Memory -> Free Memory is what the pageout daemon does. Inactive Memory -> (soft fault) -> Active Memory is the pagein algorithm. The first is well described in the source code I checked. It also should do it in two steps: it first balance the Active and Inactive memory making the Inactive memory about half of the Active memory (it's a bit more complex. The Inactive memory target is also function of the Speculative Memory) and then it goes on freeing memory from the inactive memory queue.The condition to trigger it is written in the piece of code I posted:

if ((vm_page_free_count < vm_page_free_min) ||

((vm_page_free_count < vm_page_free_target) &&

((vm_page_inactive_count + vm_page_speculative_count) < vm_page_inactive_min)))

thread_wakeup((event_t) &vm_page_free_wanted);

Even more interesting there is a section considering the option that the dynamic pager is not running. In that case, if an Active memory page is a candidate to become inactive, if the page is dirty it gets put back into the Active queue, with the pageout deamon keeping scanning the Active memory until if finds a page that is not dirty. Which explains the ugly degradation in performances when the dynamic pager (the virtual memory) is disabled.

Another thing, that finally explains why OS X may write in the backing storage while we still see plenty of Free memory in Activity Monitor or the likes: the free memory relevant to the kernel is not the one we are used to, i.e. the free memory reported in AM, but the one reported by vm_stat. This is the formula:

Free Memory (AM) = Free Memory + Speculative Memory

The maximum lower limit for free memory before to request extra pages, i.e. the value above which the pageout algorithm will NOT be called is 2000 pages, 8MB.

I made few tests and for the first time all numbers started to make sense. After in vm_stat the free memory went below 2000 pages the Active and Inactive memory have been balanced with the proportion mentioned (from 3-4:1 to about 2.1) and some pageouts occurred growing the backing storage from 0. But AM was still showing 800MB or so of Free Memory. Obviously the Speculative Memory was keeping the rest of RAM shown as free in AM. This is shown in here:

free active spec inactive wire faults copy 0fill reactive pageins pageout

23171 266678 274528 72453 149456 2587 1 1993 0 12658 0

1744 247058 268286 70677 198294 1395 2 361 0 48222 0

1744 246783 217320 70677 249657 2656 0 1564 186 46901 0

1744 245288 171390 71071 296880 1923 0 222 318 43037 13

1744 246029 118976 74631 344752 2660 1 1497 1 43615 0

2000 246549 78129 81932 377369 4136 0 1464 505 29601 107

[...]

1920 247681 18846 125092 392628 348 0 283 0 79 0

Where the last line shows the status after the pageout daemon stopped.

In other words the Speculative memory, which can (and does) keep unnecessary data read in advance from the disk can cause the kernel to page out. All of this happens in a concurrent fashion. So while the kernel is paging out another process can request (or release) memory, invoking the pagein process.

Again, this is what the source code says, and sadly that is not shown in Activity Monitor at all. The free memory reported can be considered a fake number at this point. It is not documented either, since the Speculative memory, which is a big player in the memory management module is not mentioned anywhere. Which, as we know, lead already to far too much confusion.

Michelasso wrote:

Sorry, but Active Memory -> Inactive Memory -> Free Memory is what the pageout daemon does. Inactive Memory -> (soft fault) -> Active Memory is the pagein algorithm.

What I think you may not have fully considerd is that page outs only happen if there is a hard fault. On a soft fault, no paging (in or out) occurs -- if the fault involves a write (to physical memory), the page is marked both as active & modified but paging doesn't occur until later, when that page data needs to be written back to the disk. That only occurs when physical memory needs to be freed, or possibly when the application explicitly frees it.

Something else you might want to consider is VM objects & memory objects are not the same thing. (See for instance Mac OS X VM Overview in the Kernel Programming Guide for more about that).

R C-R wrote:

Michelasso wrote:

Sorry, but Active Memory -> Inactive Memory -> Free Memory is what the pageout daemon does. Inactive Memory -> (soft fault) -> Active Memory is the pagein algorithm.

What I think you may not have fully considerd is that page outs only happen if there is a hard fault. On a soft fault, no paging (in or out) occurs -- if the fault involves a write (to physical memory), the page is marked both as active & modified but paging doesn't occur until later, when that page data needs to be written back to the disk. That only occurs when physical memory needs to be freed, or possibly when the application explicitly frees it.

It is fine with me that the pageout daemon can be called only after an hard fault. But that wasn't relevant. The point is that after it is called (which happens only if the conditions I copied from the source code are met) its first job is to balance the active and inactive memory and then to raise the free memory paging out if the selected inactive pages are marked as modified (and no matter what in such case It does page out. That's why sometime we see some MB in the backing storage still having plenty of free memory available in Activity Monitor). Once the free memory reaches 8MB or so (2000 pages MAX) it stops.

Regarding soft faults instead is the pagein process duty to manage them. I thought this was clear to us all. Basically a page in the inactive memory gets remapped into the address space of the process requesting it and getting moved into the active queue. From a point of view those operations can be consider as a paging in [the active memory].

It must be considered that this happens in a concurrent fashion. It may happen that when a CPU core is paging out, another can be paging in (or if you prefer, moving pages from the inactive to the active queue) for another process. With conflicts managed by semaphores. But what the pageout process (actually vm_pageout_scan() in the module vm_pageout.c) does is clearly documented in the code.

What is still not clear to me is how the speculative pages (that can be quite a large amount of RAM and are vital for the memory management module) get moved, (re)allocated or released. I didn't focus on them yet, but I'll get there. Also it's not immediate to understand when the pages gets removed from the backing storage. It still seems to grow a bit too much in some cases.

At least I am on peace with myself now considering the "real memory" values not reliable. They are an approximation (as you rightly pointed out) with a variance that is honestly far too big for me to be considered. As it is clear that OS X Lion can be a memory sucker when compared to other OSes executing more or less the same functions.