1 24 25 26 27 28 Previous Next 957 Replies Latest reply: Dec 1, 2013 1:28 PM by Jonathan Payne1 Go to original post
  • 375. Re: Lion - Memory Usage Problems
    Michelasso Level 1 Level 1 (65 points)

    BTW regarding the note. If "page outs" is the cumulative amount of data that Mac OS X has moved between RAM and disk space (so with cumulative I assume the total in time) and "swap used" is the amount of information copied to the swap file on the hard drive (again, I assume the actual amount of information) how comes that the second is bigger?? This is driving me crazy. Not in terms of solving an issue I have no power over it, just in terms of understanding.

  • 376. Re: Lion - Memory Usage Problems
    shuckster Level 1 Level 1 (0 points)

    We're not sitting here playing "spot the green slice" in Activity Monitor. I wouldn't bother opening it if I weren't driven to do so by real-world performance issues.

     

    Here's my experience of Activity Monitor: Inactive Memory grows until OS X starts swapping-out causing performance problems.

     

    Whether this is a problem with Safari (since that's the app I use most) or with Mach itself I don't give a hoot. I just want it fixed (if that's an "appropriate" topic for the discussion.)

  • 377. Re: Lion - Memory Usage Problems
    Mark Free Level 1 Level 1 (20 points)

    Couldn't agree more. We all know what Apple says about how the memory system should work. But this thread wouldn't even be here, if everything worked as it should and people were not having performance issues.

  • 378. Re: Lion - Memory Usage Problems
    R C-R Level 6 Level 6 (14,160 points)

    Michelasso wrote:

    It is labeled as "swap used", as you know already. That is the total amount of memory actually stored inside the /private/var/vm swap file(s).

    That is not exactly true. AM's "swaps used" shows the amount of information actually swapped out to the hard drive. See for example this KBase article for a simplified explanation of how to interpret AM's memory info.) The swapfile(s) in private/var/vm/ are created with fixed sizes. It is important to understand that these swap files are organized into pages of 4 KB each, & each page may be swapped in & out individually.


    So for example, you might have just one swapfile0 in private/var/vm/ with only some of its pages actually ever used to store anything, with some pages used over & over again when their previous contents are no longer needed. Or you might have also have a swapfile1, created when swapfile0 wasn't big enough to hold all the pages that needed to be swapped out, but at some later time its pages also might be only partially used, or not used at all.

     

    It might help to think of the swapfiles' size(s) as something like a measure of the capacity of the box(es) used to store swaps & "swaps used" as a running total of what was put into them. Obviously, if you take something out of the box you can put something else in its place.

     

    Where your description of virtual memory fails in Lion is that sometimes Lion simply does not release the active memory.

    Lion should never release active memory. Memory pages must first be moved to the inactive list before they can be paged out. Did you mean to say inactive memory instead? If not, refer to the Paging Out Process section of the developer doc for an explanation of the steps in the process.

     

    As you already know, before to start the resource's expensive job of paging out, the pager should first release the inactive read memory.

    I'm not sure what you mean "read memory." It doesn't matter if the page holds code, data, or whatever. If it has to be paged out, it takes resources to do that.

     

    What wouldn't make sense is why such large amounts of memory, mostly junk, have been marked for retention and not released already.

    What makes you think that the retained pages contain mostly junk? If say Safari has caused some code to be initialized in RAM that hooks into a system API (say to run a movie), it would not make sense to release it as long as Safari is running unless that RAM is needed to run something else.

     

    Logic says that since only the inactive memory can be paged out (it's one reason why it's there after all) and since in my case it is always less than 1GB I shouldn't have more than 1GB of data paged out.

    I don't follow your logic. As mentioned above, many more pages can be swapped out than the total of what can fit into all the available RAM at any one time. That's the point of using VM.

     

    Last thing: you say that it is ok for Safari to retain the memory used by all pages.

    That's not what I'm saying. I'm saying that the contents of some of the pages retained in RAM are likely to be reused, so they are not discarded unless & until that is necessary.

     

    It may help at this point to refer to the "Paging In Process" section of the developer doc. Note that page ins can occur from either the backing store (the swapfiles) or from ordinary files. But even if some pages are not currently mapped into the app's virtual memory space, they still may be resident in physical memory because they were previously used & not yet released. In that case, a soft fault occurs & the pages just need to be marked as active again. But if they are not in physical memory a hard fault occurs & they must be paged in, from either the backing store via the default pager or from the file on disk via the vnode pager. In either case, if the page is involved in a write operation, the page is marked as modified & must be written to disk to preserve the newly written info before the page can be freed up for other processes. If it is never modified while it is in physical memory (not written to), it may be released without having to be written back to disk. Obviously, if the system has to write a page out before it can read another one in from the disk, the performance impact is much worse than if it just has to read the page.

     

    But also note that different processes or even different sections of code within one process can share pages, as explained in the third paragraph of the Details of the Virtual Memory System section of the doc. As long as no process writes to those pages, they do not have to be written back to disk either.

     

    I hope all this helps explain why understanding why the VM system retains some things in RAM longer than seems necessary or desirable can be a formidable challenge, even with the system-wide statistics Activity Monitor, vm_stat, or similar tools provide. They just do not provide the detailed, process-by-process info necessary to do this, for instance what the contents of some specific page holds, which (if any) processes may have modified it, or what occurs between sample intervals -- which can be a great many things per second, especially for soft faults that never have to write anything back to disk.

  • 379. Re: Lion - Memory Usage Problems
    Michelasso Level 1 Level 1 (65 points)

    R C-R wrote:

     

    I hope all this helps explain why understanding why the VM system retains some things in RAM longer than seems necessary or desirable can be a formidable challenge, even with the system-wide statistics Activity Monitor, vm_stat, or similar tools provide. They just do not provide the detailed, process-by-process info necessary to do this, for instance what the contents of some specific page holds, which (if any) processes may have modified it, or what occurs between sample intervals -- which can be a great many things per second, especially for soft faults that never have to write anything back to disk.

     

    No, unfortunately it doesn't help much. We keep repeating more or less what it is written on those 2 Apple documents. Still they don't explain some key questions:

     

    • What is really stored in the "swap used" area? The Apple keynote explains little or nothing about this
    • Why it can be order of magnitudes bigger than the page outs value? Right now 10.6MB vs 336KB
    • Why still having plenty of free memory (hundreds on MB) one can get some page outs activity?
    • Why it happened to me twice that I've got a single swapping activity of 2.7GB in terms of "swap used" space? From 0 to 2.7GB I mean. Uninterrupted according to the swap used space growing constantly.
    • Why more often I get similar swapping activities of 4-500MB? And keeping raising the "swap used"
    • And a little bit out of the topic, why safari memory usage sometimes keeps raising just statying in the same web page?

     

    Now, from my understanding of the old Unix System V virtual memory algorithm I had the opportunity to put my hand on, generally speaking the pagers in Unix (but probably in any other OS, even in Windows) work in this way:

     

    - There is a Low Water Mark and an High Water Mark

    - When the free memory goes below the Low Water Mark the pager kicks and frees pages until the free pages reach the High Water Mark

     

    Considering the values in bytes

     

       High Water Mark - Low Water Mark

     

    can't be 4-500MB in a 3GB RAM system. It should actually be a relatively small value for obvious performances reasons.

     

    Also this makes the small page outs when having hundreds of MB free (even more than 1GB) a mistery. Until the free memory doesn't go below the Low Water Mark value the pager just has to stay quite. And that doesn't show in my memory monitor graph. Unless, sure. We have a peak of memory usage (not allocation) that lasts less than a second to be immediately released. More than unlikely it would be nearly impossible to happen, because to fill 1GB or so of memory would take more than 1 second in a slower system like mine. And what application would need to grow so much in size in such a short time? Thus that peak should show in the memory graph. Obviously it doesn't.

     

    Now, maybe it is working as designed. But if so it would nice from Apple to update that document and explain to people like us what really happens in terms of memory inside our most beloved operative system. My idea is, since I even got some weird error messages in the kernel log from the virtual memory system (the same I published here and I opened a bug report for), that something is bugged. But sure, I may be wrong. In that case, though, I would be right saying that the competition manages the memory in a much more efficient way and Apple should be better to stick with the stable old solutions instead of experimenting something that works like junk. Either way it must be fixed.

  • 380. Re: Lion - Memory Usage Problems
    John Kitchen Level 3 Level 3 (635 points)

    Some answers...

     

    The "Swap Used" contains changed pages which had to be swapped out due to RAM shortage. 

     

    The "Swap Used" can exceed the "Page outs" since swap used is allocated in large chunks.  IIRC, it's 128KB chunks.  Much larger than the 4K page size.

     

    Why does Page Out happen when there is (apparently) lots of Free Memory?  The "Free" memory is a snapshot in time.  RAM demand fluctuates on a microsecond basis, whereas Activity Monitor is only updated once a second or slower.  It is highly unlikely that the pie chart will show you peak demand.  On my 12GB iMac, I have seen the pie chart telling me that about 6GBs are free, but suddenly see some page outs have occurred.  This is normal in any virtual storage system.  It is also why relatively underpowered Macs which are equipped with SSDs often behave better than high powered Macs with HDDs.

  • 381. Re: Lion - Memory Usage Problems
    Michelasso Level 1 Level 1 (65 points)

    John Kitchen wrote:

     

    Some answers...

     

    The "Swap Used" contains changed pages which had to be swapped out due to RAM shortage. 

     

    The "Swap Used" can exceed the "Page outs" since swap used is allocated in large chunks.  IIRC, it's 128KB chunks.  Much larger than the 4K page size.

     

    Why does Page Out happen when there is (apparently) lots of Free Memory?  The "Free" memory is a snapshot in time.  RAM demand fluctuates on a microsecond basis, whereas Activity Monitor is only updated once a second or slower.  It is highly unlikely that the pie chart will show you peak demand. 

     

    Oh, thanks. The 128KB chunks would be compatible with the numbers I had indeed. I am not sure how it would save the swapped pages thou, but for now this would do.

     

    Regarding the microseconds thing I explained why I don't buy it. We are talking about hundreds of MB in the free memory that should get allocated, processed, written and then released in less than a second as if nothing happened. Apart that I wonder if it would be possible (the maximum bandwidth for 667MHz DDR2 RAM is 10.7GB/s), what would be the need for anything like that? The processes are busy elaborating data, not copying or filling memory all around.

  • 382. Re: Lion - Memory Usage Problems
    John Kitchen Level 3 Level 3 (635 points)

    When an application decides it needs working space in RAM, it asks the OS for allocations.  The time taken to allocate the RAM is not dependent on RAM bandwidth, just the amount of time it takes for the instructions involved in the call to the OS.  The amount of time that the RAM is held depends on what functions the app is performing, but may be very brief, and doesn't require that app actually modify all the RAM acquired.  When the app no longer needs the RAM, it calls the OS to release the RAM.  Again, that is very fast.

     

    PS The swapped pages (each 4K in size), are written to the swap areas (within the 128K allocations)

     

    Message was edited by: John Kitchen - added PS

  • 383. Re: Lion - Memory Usage Problems
    Michelasso Level 1 Level 1 (65 points)

    It doesn't work like that. When a process calls malloc() or any equivalent the kernel just adds the memory to the virtual address space.and returns the pointer to the virtual address where the virtual memory area begins. Only when the process first access an address in that virtual address space a fault is generated to have the relative physical page assigned to the process for its use.

     

    So what it grows when a process allocates more memory is the virtual memory size, not the real memory used. The total for all processes is the VM Size in Activity Monitor, usual hundreds of gigabytes. The first virtual memory principle is that physical memory must not be assigned to a process until it is actually used.

  • 384. Re: Lion - Memory Usage Problems
    John Kitchen Level 3 Level 3 (635 points)

    Michelasso wrote:

     

    Only when the process first access an address in that virtual address space a fault is generated to have the relative physical page assigned to the process for its use.

    Quite correct, I should have added the access step in my description.

  • 385. Re: Lion - Memory Usage Problems
    R C-R Level 6 Level 6 (14,160 points)

    Michelasso wrote:

    Now, maybe it is working as designed. But if so it would nice from Apple to update that document and explain to people like us what really happens in terms of memory inside our most beloved operative system.

    If you mean the "About the Virtual Memory System" developer document, note that it says it just provides a brief overview of the virtual memory system, & for more detailed information on how the virtual memory system works, you should see Kernel Programming Guide . As as that document says, its sections are primarily intended for programmers writing file systems or modifying the virtual memory system itself, but you might find several of them of interest.

     

    For example, the Other VM and VM-Related Subsystems section has detailed info on the pagers. There are several of them to consider, not just the default pager most people think of when they think of VM systems, high-water marks, & so on.

     

    You should probably also study at least the Mac OS X VM Overview section, which covers things like the difference between a memory object & a VM object, VM calls available to apps, etc.

  • 386. Re: Lion - Memory Usage Problems
    Michelasso Level 1 Level 1 (65 points)

    R C-R wrote:

     

    for more detailed information on how the virtual memory system works, you should see Kernel Programming Guide . As as that document says, its sections are primarily intended for programmers writing file systems or modifying the virtual memory system itself, but you might find several of them of interest.

     

    Oh, blyme. It seems to be exaclty what I was looking for. It is also updated to August 2011. No matter how much I looked around I couldn't find it. I will read it now!

     

    Well, thanks!! Finally some instructive documentation.

  • 387. Re: Lion - Memory Usage Problems
    Michelasso Level 1 Level 1 (65 points)

    Oh well, just a couple of things if anyone is interested:

     

    Thanks to the new link I have finally understood what the low and high water marks are in OS X. They do not apply to the physical RAM as it was common, but to the swap files. When the swap used area goes above the High Water Mark the dynamic pager allocates a new file. When it goes back below the Low Water Mark the dynamic pager reorganizes the data in the swap files and removes one flle (the largest one I suppose). Sure I couldn't make any sense out of it since I was thinking in terms of RAM. I still have to understand when the default pager gets called (probably anytime there is a memory fault at this point), but frankly I'd rather wait for the new update and see if anything improves.

     

    The Safari process just jumped from 200MB to 1GB+ opening the activity window. Web Content was already at 800MB. It's useless to mention that OS X went on just swapping... Ok, this is a rogue bug, but still a sign that something is very screwed up in the way Safari allocates memory. Not that Chrome seems much better to be honest. But at least it releases memeory when closing the tabs. Again, let's hope in the incoming update. It should be a matter of days at this point.

  • 388. Re: Lion - Memory Usage Problems
    Gabriel Couto Level 1 Level 1 (0 points)

    Hi, usually I would say I'm very experienced with OS workings... I study computer science and stuff.

    But I see people here talking about things beyond my mind knows about.

     

    I just wanted to report that I think my memory use in Lion is not normal. I have around 5 programs running(dashcode, textedit, excel, safari and sparrow)... almost none of them is eating memory, except Safari.

    I had 5 tabs open in Safari, I'm using ClickToPlugin extension, so no plugins are running... and they are simple pages, no javascript animations and such things. My Safari is eating almost 1.5gb ram. That's crazy, and sometimes is way above it.

    (Sometimes I close all windows and it doesn't release any memory at all - only exiting the app)

     

    My Activity Monitor Screenshot:

    http://cl.ly/3v2G3A0J1G3j3D0H2t3h/o

     

    One more question, if I sum all the "Real Mem" column, it would be equivalent to the "Active Memory" or "Used Memory"? I did sum, and it appears it is the "Active Memory"... but how can I know who is reserving inactive memory then? Sometimes I have 4GB of inactive memory filling my RAM.

    And 1gb of wire coming from nowhere? I expected the wired memory was the kernel, and something like that.

    I mean, If the sum of that column was supposed to be equivalent to other value than only "Active Memory", my computer is using much more memory than my processes are using.

  • 389. Re: Lion - Memory Usage Problems
    Michelasso Level 1 Level 1 (65 points)

    Hello Gabriel.

     

    Welcome to the nightmare of Virtual Memory Management in OS X!

     

    Your observations are correct. At least it is what many of us are experiencing in this thread. To answer your question about how to read Activity Monitor values, from what I understood it should work more or less like this:

     

    • Real Mem: The total amount of RAM actually used by the process. This should include the memory from dynamic libraries and similar
    • Private Mem: The amount of Active RAM used by the process itself.
    • Shared Mem: The amount of Inactive RAM used by the process. "Shared" because it can be used, when reallocated (or "stolen") by other processes. I know, it is absolutely confusing.

     

    To see those values you must enable the equivalent columns in AM.

     

    So the Active memory should be the sum of all Private Mem, and the Inactive memory the sum of all Shared mem. Obviously those numbers don't match, so that could be completely wrong. But that's what I've read anyway.

     

    Regarding your Safari numbers tha's what I get sometimes as well. Getting worse with AdBlock extension enabled. Safari 5.1 is a new release making use of many new APIs (again). Chances that it is bugged are not low. And my strong belief is that the memory management system is not in much better conditions.

     

    Since your system still doesn't really page out, I'd say that the best thing at this point is not to worry and wait for 10.7.2. It should be a matter of days, now. Hopefully these very annoying teething troubles will be over.

1 24 25 26 27 28 Previous Next