1 26 27 28 29 30 Previous Next 957 Replies Latest reply: Dec 1, 2013 1:28 PM by Jonathan Payne1 Go to original post
  • 405. Re: Lion - Memory Usage Problems
    Michelasso Level 1 Level 1 (65 points)

    Sjazbec wrote:

     

    If your apps ran succesfully on Snow with 2GB of Ram ( Lions minimum and double the minimum of Snow , because Lion runs by default in 64bit mode while Snow ran by default in 32bit mode) , then they will still in Lion with the same amount. The more, the better is a natural thing.

     

    You complain that "all the RAM" is occupied  with just the handful of Apps open you mentioned, but have you de facto tried opening more apps and see if there are problems related to "low on memory" sort of errors ?

    Well, actually having the EFI32, Lion on my MacBook is forced to still run at 32bits.

     

    To answer your question, yes I did many tests. 10.7.1 just paged out up to 2.8GB of RAM in a single chunk sometimes. WIth the dynamic pager also reporting errors about "not enough available paging space" and aborting.

    The minumum chunk has always been at lest 400MB. I opened a bug report about it, and I have been asked today to send the log file generated by "sysdiagnose" (I still have to take a look at it, but it is nice to know some new commands for OS diagnostic!). At least it is good to see that at Apple they take the bug reports seriously.

  • 406. Re: Lion - Memory Usage Problems
    Michelasso Level 1 Level 1 (65 points)

    William Kucharski wrote:

     

    Most operating systems take a lazy approach to freeing up memory, so once again it doesn't matter how much memory anything is taking up until it's needed elsewhere.

     

    So for example, if there are cached disk blocks that are no longer needed, that memory isn't actually freed up until low water marks for free memory are reached, or you would need a constantly running thread that did nothiing but free up unused memory.

    That is fine with me. As long as the VM system and the OS in general maintain the goal of keeping all disk accesses to the minimum.

     

    Does the OS use all the available RAM to achieve that goal? Great!! But if an unusual high inactive memory is a sign that sooner or later the OS is going to swap out large chunks of memory, not only I am not happy anymore, I am sure that something is not working as designed.

     

    Regarding the I/O buffers I agree. They are like a dynamic cache. They raise and diminish in size according to the available free memory. If I understood correctly they are shown as part of the free memory indeed and not of the inactive one (all documentation I've read didn't make it very clear). Plus other buffers belonging to the kernel. So yes, when an Application needs some pages they should be pages taken from the I/O buffers reducing their sizes. Most free RAM can be used for the I/O buffers still being transparent to the users because it still shows as free memory. When the free memory gets low, pages not marked as modified should be taken from the inactive list.

     

    This makes sense because the other only solution is to swap out inactive pages marked as modified and that contradicts the goal of I/O buffers which is, as well, to reduce the disk accesses to the minimum (it would be better to page out to disk than to reduce the I/O buffers). Until the minimum size for the I/O buffers has reached, sure. Otherwise the system would trash.

     

    This is how I am pretty much sure Snow Leopard works. I don't believe the same is true for Lion 10.7.1.

  • 407. Re: Lion - Memory Usage Problems
    R C-R Level 6 Level 6 (14,175 points)

    Michelasso wrote:

    1. The amount Windows 7 is using?

    Why do you think comparing apps with different feature sets running on two different OS's with different memory management systems & entirely different API's will tell you anything useful? It is an apples-to-oranges comparison.

    2. I mean that the content of the VM pages has been marked as modified. In the paging and swapping algorithm that means they will be paged out when "stolen" to raise the free memory.

    I still don't understand what you mean. You seem to be confusing real & virtual memory pages -- modifications take place in RAM, not VM.

    3 It is wrong because it is degrading performances. One golden rule can be that "unused memory is wasted memory" but before that there is the rule "avoid disk access [for memory] as much as possible". Especially the writes that  take longer.

    This seems to be an argument for keeping things in RAM once they are placed there, not for discarding them.

     

    At least that's what they taught me at the Flinstones age of Computer Science.

    This may be part of the problem. As you noted earlier, memory management in OS X is quite complex & can take months of study to understand even partially. There are (among other things) three different pagers, both the VM-enhanced Mach kernel & BSD memory calls, & OS X's unique division of kernel & I/O Kit low level tasks to consider.

     

    Just to really understand how a process actually is using memory requires the use of tools like vmmap, which among other things shows you info about what happens when a page is modified. Unless you study this detailed info carefully, all you can do is guess about the process's memory use. And it is very easy to make wrong guesses, especially if you don't know much about the tasks processes must support (like interpreting web page markup languages) or system-specific API's & libraries.

     

    Worse, it seems like the more you study, the more there is to learn, with no end in sight. The temptation to generalize & oversimplify can be overwhelming. Believe me, I know -- I've devoted a great many hours to studing the developer docs & all I really have learned is how little I know.

  • 408. Re: Lion - Memory Usage Problems
    Michelasso Level 1 Level 1 (65 points)

    1. If there is 3 times the difference in RAM used by the same application doing the same job in two different OSes I think I have the right to believe that such application is not optimized in the second OS. Not in terms of memory, at least.

     

    2. Yes, the physical pages that have been marked by the VM system as modified. The marks are contained in the VM maps not in the physical pages themselves. The basic concept is that the RAM content that is mapped by the VM system on disk, and so in a sense not modified, doesn't have to be paged out.

     

    3. Unless it is bugged, filling up the inactive memory, causing page outs that trash the system. Now I know for sure that 10.7.1 is, but I can't say why for not breaking the NDA.

     

    The tools we have available  give a valid idea about what's going on in the systems. More "top" than AM, anyway. And vm_stat. 1 second granularity is more than sufficient for a fast analysis. If in a idle system the memory used by a process keeps growing 1MB per second, something is very suspicious. If the used memory takes all free memory, so much that the kernel starts paging out, I can claim a bad allocation of resources the least, a memory leak the worst. 

     

    I give you an example: In Unix any application has the right to use all memory it likes (likes, not needs). It's very easy to write a malicious program made with the only purpose of thrashing a system. Just an infinite loop that allocates memory and fills it. The allocated memory blocks get used only once, but  still they stay in RAM first and since they have been written they will be paged out later because the process itself keeps requesting new memory.

     

    To make things even worse it can have a second loop that in parallel it reads back from the first memory allocated (and so from the less recently used pages, which are in inactive memory) all values. Result? What has been paged out must be paged in, while it keeps paging out. The swap used keeps growing ad infinitum. The inactive memory decreases being often accessed, raising the active memory. The only solution is to kill such a process. Sure, assuming the system is responding enough to allow to run any told to identify it.

     

    And guess what: that is not a memory leak. vmmap, leaks, any garbage collector, etc can't detect anything wrong. For the simple reason that nothing is wrong. The process is making effective use of all its memory. But a simple look at the processes memory usage with "top" or Activity Monitor will identify the culprit. Even before it will start thrashing. That's why those numbers in big organizations are monitored every few minutes (not seconds) on each server by tools like HP OpenView and create tickets to be checked by the system administrators. And yes, those processes can (and in various cases will) get manually killed.

     

    Which is exactly what many of us short in RAM are doing with Safari. When it takes too much memory it is better to close it. After restarting it we still get hundreds of megabytes, sometimes more than 1GB of extra free memory. So much for its beautiful mostly useless caches: gone, finally. At least for a while. See the first example.

  • 409. Re: Lion - Memory Usage Problems
    shuckster Level 1 Level 1 (0 points)

    This is all getting very complicated. Lets simplify things a bit:

     

    1. Under Lion, Safari eats up all my RAM.

     

    2. Under Snow Leopard, it didn't.

     

    Now, unless OS X has undergone a major rewrite of its memory-management code -- no small feat judging from what you have been teaching us about it -- then the changes that have been made that are causing problems are localised to new or refactored code.

     

    Now, I've ready Siracusa's Ars Technica article explaining how reference-counting has become an automatic compiler step, and I know that Safari now incorporates Webkit2. So here we can see two obvious paths of inquiry which I am sure the dev-team at Apple are hard at work on.

     

    So enough with the apologetics: There's new code in Lion and there's bugs in it. Bring on the fixes.

  • 410. Re: Lion - Memory Usage Problems
    R C-R Level 6 Level 6 (14,175 points)

    Michelasso wrote:

     

    1. If there is 3 times the difference in RAM used by the same application doing the same job in two different OSes I think I have the right to believe that such application is not optimized in the second OS. Not in terms of memory, at least.

    Except that it isn't really the same app. The feature sets are different & they have been compiled for two different OS's with different memory management systems & with different API's. They don't even call the same routines to measure RAM use.

    2. Yes, the physical pages that have been marked by the VM system as modified.

    Why are all these pages marked as modified? Earlier you said, "Also considering that all web pages content in RAM are modified pages that will be swapped out when the free memory gets low." You still haven't explained what you mean by that, nor have you considered anything besides web page content.

    3. Unless it is bugged, filling up the inactive memory, causing page outs that trash the system.

    You are oversimplifing again. Don't forget that not everything in an inactive memory region has to be paged out -- if it has not been modified since being paged in, there is no reason to page it out before marking the region as free memory. As you (now) know, apps like Safari maintain cache files seperate & apart from the VM backing store. Since they are ordinary files, they are handled by the vnode pager, not the default pager, just like code. So even hard faults don't necessarily cause page outs. Understand?

     

    The tools we have available  give a valid idea about what's going on in the systems. More "top" than AM, anyway. And vm_stat. 1 second granularity is more than sufficient for a fast analysis.

    You misunderstand what I mean by granularity. You still seem to have the idea that all objects in memory are large, for instance that web pages are made up not of many different smaller parts that each load separately but of one big thing that either is or isn't in memory at any given time & it  takes substantial time to load if it isn't. That isn't true. It is the same for code segments -- much of it is quite small & it takes very little time for the vnode pager to load it into memory. Study Safari's Activity window & vmmap's output & you should see that this is true.

     

    Because of this, a 1 second sample rate will miss much of what is going on -- thousands of memory changes could occur in that interval. In fact, overview tools like AM or top only loosely approximate memory use because an accurate analysis takes considerable system resources. (See for example the "PERFORMANCE vs. ACCURACY" section of the top man page.)

     

    For AM's per process memory statistics it is even worse. If you have used vmmap (which to get accurate numbers walks through each page, checks its pmap entries, & so on), you probably have noticed that it can take as much as a second or two just to compile statistics for one process. Now imaging how much time it would take AM to do the same for all the processes it displays. Obviously, AM does not do this. (If it did, AM would show its own use of CPU time soaring).

     

    AM makes no attempt to sort out what is in the memory space of a process, what put it there, or its purpose. In fact, it doesn't even try to estimate the use of active, inactive, or wired memory on a per process basis, which is part of the reason you can't get the per process memory numbers it shows to add up to the totals AM shows for all processes.

     

    So what you call a "fast analysis" is in reality a guess based on knowing only a small part of what is really going on, some of it an estimate, & most of it on what you admit is hard to understand & confusing. And yet, even after you have read through Apple's doumentation explaining in detail what is needed for a real analysis, you ignore that, in part because it is "too detailed."

  • 411. Re: Lion - Memory Usage Problems
    Michelasso Level 1 Level 1 (65 points)

    This is never ending.. What I really want is to make this concept clear once for all:

    Because of this, a 1 second sample rate will miss much of what is going on -- thousands of memory changes could occur in that interval. In fact, overview tools like AM or top only loosely approximate memory use because an accurate analysis takes considerable system resources. (See for example the "PERFORMANCE vs. ACCURACY" section of the top man page.)

    This is the manual section for top you just referenced:

     

    PERFORMANCE vs. ACCURACY
           Calculating detailed memory statistics is fundamentally resource-intensive.  To reduce the cpu  usage in  top,  the -i parameter has been introduced to allow the user to tune this tradeoff.  With the default value of 10, framework stats will be updated once every 10 samples.  Specifying -i 1  will  result in the most accurate display, at the expense of system resources.

     

    It's talking about a default interval of 10 samples. The reason why it uses 10 samples is to reduce the impact on CPU usage. Nowhere it is written that 1 sample, which I use, is not accurate enough. Actually it is stated to be the most accurate. Most, not "more" or "accurate enough". Your own link totally contradicts your thesis. Even 10 samples, being the default are good for an accurate enough monitoring.

     

    The point is that what happens every micro or millisecond does not matter. I really don't understand why you insist.

     

    What it matters is that Safari after some hours (not milliseconds) uses 2,3 even 4 times the initial memory having the exact same tabs opened at the beginning. It matters that its high memory usage can trigger a bug in 10.7.1 that makes the VM system thrashing. Unless you consider 3GB of page outs in a shot a normal behavior, sure.

  • 412. Re: Lion - Memory Usage Problems
    Dperr23 Level 1 Level 1 (0 points)

    All of the theory is great, but nothing is explaing why my machine is considerablly more sluggish than it was under SN.  I used to brag to my friends how fast my Mac was up and running while my Windows machince was still booting.  Now, it is the other way around.  It takes a great deal of time to get to the login screen.  When it finally comes up, I click on my login, and it takes forever for me to get to a point where I can input my password.  From the password input, my machine is very sluggish getting to my desktop.  Running applications is very sluggish even if the app is not resource intensive.  I maxed it out on RAM, and the problem has not improved.  Without a bunch of theory, can anyone provide something that will improve my situation?  I am a lover of all things Apple, but Lion has not lived up to the hype, yet.

  • 413. Re: Lion - Memory Usage Problems
    Atomic Al Level 2 Level 2 (415 points)

    Dperr23 wrote:

     

    All of the theory is great, but nothing is explaing why my machine is considerablly more sluggish than it was under SN.  I used to brag to my friends how fast my Mac was up and running while my Windows machince was still booting.  Now, it is the other way around.  It takes a great deal of time to get to the login screen.

    This could be Resume starting all of your Apps in the background?

  • 414. Re: Lion - Memory Usage Problems
    Michelasso Level 1 Level 1 (65 points)

    Dperr23 wrote:

     

    It takes a great deal of time to get to the login screen.  When it finally comes up, I click on my login, and it takes forever for me to get to a point where I can input my password.  From the password input, my machine is very sluggish getting to my desktop.  Running applications is very sluggish even if the app is not resource intensive.  I maxed it out on RAM, and the problem has not improved.  Without a bunch of theory, can anyone provide something that will improve my situation?  I am a lover of all things Apple, but Lion has not lived up to the hype, yet.

    Happening at the boot time that shouldn't be memory related. I had the same issues (like many) and I managed to fix them. This is what you should do (not necessarily in the order):

     

    • Reset the PRAM and NVRAM
    • Reset the SMC
    • Repair the file permissions with Disk Utility
    • Download Onyx, start it, go to Automations. Select everything (I did) and execute. That will clear all caches and anything that is possible wrong that can be automatically rebuilt. The step above is actually included in Onyx. So you may skip it.

    These commands are not intrusive. The first boot will take quite some time because of the caches, the Spotlight index and maybe other stuff that needs to be rebuilt. So give it time to finish. The Finder's folders view will be restored to default and if you have the size column in Finder that will be computed again.And that's the easy part. Then I went as far as removing the preferences files in the ~/Library/Preferences folder.

    • In Finder click on the Menu Go/Go to Folder.
    • Type or paste "~/Library".
    • Drag the Preferences folder into the Desktop
    • Reboot.

     

    It will be better to do it booting first in single user, but personally I don't care, it shouldn't mess up anything. This operation is very intrusive. All your applications settings will be reset to default. Like when you create a new account.

     

    Last thing to check are the third party applications. In Terminal you can paste the following commands:

     

    echo "\nKERNEL EXTENSIONS:"

    kextstat -kl | awk ' !/apple/ { print $6 $7 } '

     

     

    echo "\nLAUNCHD LOADED JOBS:"

    launchctl list | sed 1d | awk ' !/0x|apple|com\.vix|edu\.|org\./ { print $3 } '

    launchctl list | sed 1d | awk ' !/0x|apple|edu\.|org\./ { print $3 } '

     

    echo "\n\nLIST OF THIRD PARTY COMPONENTS:"

    ls -lArt /Library/{In,Launch,Start}* 2> /dev/null

    ########## END SCRIPT

     

    The output of the last command is sorted from the oldest to the newest. You should check if you recognize anything that you are sure it isn't used anymore, identify the corresponding application and uninstall it. Also anything not owned by "root" would be suspicious.

  • 415. Re: Lion - Memory Usage Problems
    R C-R Level 6 Level 6 (14,175 points)

    Michelasso wrote:

    Nowhere it is written that 1 sample, which I use, is not accurate enough.

    Nowhere is it written that top (or Activity Monitor, which displays more or less the same info) is a suitable -- much less accurate -- tool for analyzing per process memory use, no matter what the sample rate. Refer to Apple's Memory Use Performance Guidelines, especially the section on Tracking Memory Usage, for what is written about the tools that are suitable for that purpose.

     

    Throughout this discussion you have made it clear that you don't know much about what the per process numbers either top or AM displays really mean, or for that matter much about how browsers or web programming work. You have in effect said that isn't important, that the details of how a process uses memory don't matter, & that you can somehow tell if a process isn't using memory efficiently without using the tools programmers must use to do that.

     

    It would be great for all concerned if this were possible, but it isn't. You really do have to study the details of how a process uses memory to know if it is using it properly.

  • 416. Re: Lion - Memory Usage Problems
    Michelasso Level 1 Level 1 (65 points)

    R C-R wrote:

     

    Michelasso wrote:

    Nowhere it is written that 1 sample, which I use, is not accurate enough.

    Nowhere is it written that top (or Activity Monitor, which displays more or less the same info) is a suitable -- much less accurate -- tool for analyzing per process memory use, no matter what the sample rate.

    Really? I copy & paste it so it's easier to read:

     

    "In addition to the Instruments [the one mentioned in the link you posted] application, Mac OS X provides the top, vm_stat, pagestuff, and vmmap command-line tools for viewing statistics about virtual memory usage. The Memory Monitor instrument in the Instruments application lets you view all sorts of information about virtual memory usage in your application. This information displayed by this instrument is analogous to what you would see running the top command-line tool."

     

    Now if you wanna say that "top" doesn't monitor the memory on a per process base we must use a different definition of the term, since "top" shows a global summary on the header and then a list of processes with their memory usage and other information that fit in the remaining space in Terminal. Maybe if you try to use it at least once, you'd know what it really does.

     

    I'd say that before to recommend to someone else to study, one should actually read, and better yet, understand the documents s/he posts. To just throw a bunch of links (often the same), rephrase their content and use some fancy words doesn't really show an understanding of the matter. As much as it doesn't prove anything.

     

    If you really believe that the bizarre to say the least memory issues people are having in this thread are due to Safari rightly caching, please feel free to use Xcode's Instruments or any ohter tool and show us why.

     

    In the meantime I will keep the common belief that Safari 5.1's memory usage in Lion is far above the top. That Safari high memory usage is thrashing my system since it makes the kernel to page out gigabytes of modified pages.

  • 417. Re: Lion - Memory Usage Problems
    R C-R Level 6 Level 6 (14,175 points)

    So exactly where (in which column, by name) does top show that Safari is paging out gigabytes of modified pages? Which column shows what those pages hold, or that they have been modified?

     

    Specifically, which column shows which resident memory pages hold for example mapped files or segments of library (framework) code (which do not have to be paged out because they are not modified) vs. those holding for example computed data or malloc working regions (which may have been modified)? Which column shows how much of the resident memory (RSIZE) allocated to Safari is actually shared with other processes?

     

    As explained here, top is a monitoring tool that is indeed an important starting point for identifying potential performance problems in a process. But that is all it is. It is not a substitute for the tools designed to do a detailed analysis of the process to see if it really does have memory or CPU use problems or what they might be.

     

    Gathering "all sorts of information" is not the same thing as analyzing it. If you just use top, & particularly if you don't know the precise answers the questions posed above, you are just guessing. It is that simple.

     

    If you really believe that the bizarre to say the least memory issues people are having in this thread are due to Safari rightly caching, please feel free to use Xcode's Instruments or any ohter tool and show us why.

    How could anyone possibly do this without access to those systems to examine the contents of the cache files (if that is really what you mean) on them, the web pages visited, or anything else that affects the memory footprint? That you would even ask this makes me wonder how much you really understand about memory use or how to analyze it.

  • 418. Re: Lion - Memory Usage Problems
    Michelasso Level 1 Level 1 (65 points)

    R C-R wrote:

     

    So exactly where (in which column, by name) does top show that Safari is paging out gigabytes of modified pages? Which column shows what those pages hold, or that they have been modified?

     

    Specifically, which column shows which resident memory pages hold for example mapped files or segments of library (framework) code (which do not have to be paged out because they are not modified) vs. those holding for example computed data or malloc working regions (which may have been modified)? Which column shows how much of the resident memory (RSIZE) allocated to Safari is actually shared with other processes?

    ?? Regarding how the real memory is allocated for each process t's well explained in the link you posted:

     

    RSIZE: is the total RAM accessed by a process. That includes memory shared with other process

    RPRVT: is the RAM accessed only by that process

    RSHRD: is the RAM accessed by the process and shared with others

     

    I don't think it is possible for the kernel to identify to which process belong the pages that are either marked as modified, in the inactive list, or that have been paged out. That is also confirmed by the author of a tool similar to top, htop.

     

    Apparently the kernel doesn't keep track (probably for performances reasons) of that kind of information. And if the kernel itself doesn't do so, it's very likely because it doesn't find that information much useful. The paging system it is transparent to the processes after all.

     

    Indeed note that I didn't write "Safari paged out gigabytes of memory". I couldn't anyway, since the page choice for paging out is application independent. I wrote that the high memory usage of Safari (monitored with the tools we have available), can trigger a bug in the kernel that generates a massive page out. Nearly as big in size as the total RAM available. When I monitored that situation, the only process that was raising in memory was Safari's WebProcess indeed (the sandbox). With no real need for it if I can say since I was just writing in this exact forum.


    The issue is that any OS paging algorithm pages out only small amounts of memory per time. So, either Safari, out of reasons, can suddenly decide to allocate (and modify) more and more memory, pushing the kernel to page out the same amount memory stolen from all processes for many times (up to the infamous 2.7-2.8GB), or the dynamic_pager in certain situations loses control and pages out everything it can and even more (which is a severity 1 bug).

     

    I opt for the second option, still believing that Safari 5.1 is allocating far too much memory (not a bug: inefficiency) that makes things even worse. Without the pager bug I could get a large backing store, but in few hours and not in a shot. That would be acceptable, because if the pages are not really going to be reused, the only overhead would be some disk space temporarely "wasted" and the nearly unperceivable time spent writing it.

  • 419. Re: Lion - Memory Usage Problems
    Michelasso Level 1 Level 1 (65 points)

    BTW Regarding the kernel caching pages...

     

    In all this discussion we are missing the speculative memory that I mentioned some psts ago. The one that is clearly shown in the vm_stat output, but that obviously is not explained in the OS X vm_stat man page. That is the memory that is used to preload the pages content from disk, to optimize disk performances when there is a page fault and the content must be loaded from file. Via mmap(), I suppose. Also it should keep the pages that can be reused after killing a process (if any are really kept) because one may restart the same process.

     

    It is not part of the inactive list, but instead it is part of what we always described as Free Memory. If you run vm_stat you'll see that the number of free pages (multiplied by 4K) can be much lower than the value in Activity Monitor. Or top. It's where the principle "free memory is wasted memory" is really applied. And it is where an high inactive memory is a bad sign. Because it steals RAM space that could be used to precache the pages that will probably be used in the future.

     

    In other words they take resources from the youngs to give them to the olds. Nothing new in this world, actually.

     

    Having the man page of a command not even describing the meaning of all its output values is just confusing. The meaning of that value doesn't appear in any Apple document I've read. Still it is there. I had to go checking the open source code files to find out what it was. Really, Apple could do a better job on keeping its own documentation.

1 26 27 28 29 30 Previous Next