Lion - Memory Usage Problems

R C-R wrote:

Removing/renaming the ~/Library folder would probably do that, but it would also remove all the account's user preferences, application-specific settings, & everything else normally stored in that folder. It can be quite tedious to reset/reenter all of that.

I know. But there are people so desperate with Lion that reached the point to do a clean Lion install from an USB stick (I made one myself, it works. Or they buy an overcharged one from Apple), erasing the partition, creating a new user, reinstalling one by one all applications and then copying back all data from the old account(s).

Which, at the end of the story, is the only sure way to have a brand new Lion installation without all the junk we have left during the years from previous versions of OS X, old applications that didn't fully uninstall, old caches lost in the dungeons of the file system, wrong/corrupted parameters and preferences setting and all the likes.

Tedious? Very. But so is keeping facing weird bugs for the reasons mentioned above. To avoid such a long job the fastest solution, or the best compromize in terms of costs/benefits is to just move the Library folder. But you're right, I gave the hint in a hurry, I could have specified that the cleanest way to do it is to move the user Library folder after booting in single user mode. I just didn't feel like explaining everything in detail.

As much as you are correct saying that the best procedure for troubleshooting is to just move selected items from the folder to the Desktop or another folder and test. But this task requires quite some knowledge and was not the goal of the users that just wish to have Lion running clean as soon as possible. Investigating which setting is causing trouble is a task much more time consuming than just redefining all settings from scratch. Because, I wish this to be clear once for all, we were not talking about testing (something that I did before anyway and I know that there is more than a configuration file causing troubles in Lion) but about fixing.

Now, back to OS X being not Unix, is an Unix 03 certification enough for defining OS X an Unix OS?

http://www.opengroup.org/openbrand/register/brand3581.htm

http://www.opengroup.org/openbrand/certificates/1194p.pdf

I can't believe people still argue about user directory. It has been said several times that it won't help with memory leaks problems. A clean install won't help either.

It was never an argument about user directories. It was about a suggestion to try to diagnose the cause(s) of unusually high memory use and/or performance problems.

This is not necessarily about memory leaks, be they in the core OS routines or in specific apps. Keep in mind that the OS is supposed to use the RAM that is installed to hold data that might be reused. Thus, small amounts of free memory is not by itself a sign of any problem, even if it seems unusual. What is a problem is this affecting the responsiveness of the system, but there are several possible causes of that, some of which have nothing directly to do with how much free (which is unused) RAM is available.

The point of creating a test user account is to cleanly separate the system & user domains, which may (or may not) provide clues about performance problems. We know that not all users are having these problems & that the details of the reported problems vary. From that it is reasonable to assume that there is something local to the affected systems that is at least part of the cause(s) of these issues. In some cases, it may be something only Apple can fix. In others, it may be something the user can fix relatively easily, for instance by replacing a corrupted system file or trashing a user preference.

But without testing, it is just a guessing game that doesn't help anybody.

Both Systems Clean installed. Lion for the OS does use more memory and allocates it intentionally in case it is need to make things seem to operate quicker search the apple Kbase articles, I dont have it handy at the moment so I do apologize. However I know this cant be the case with Safari and the Safari Web content process. Safari even with 3 windows open or less without flash or any type of media or fancy css on the pages can consume up to 1gb of my memory just with Safari alone and not its web content process and then then webcontent proces swith take up usually a range of 150mb - 450mb. I would say it goes higher but Ive only seen that once and that was running a heavy flash page.

When Im not in safari, My machine is fine, Yet I did upgrade from 4gb to 8gb in this machine to try to help and the os just allocates more, and safari seems to find a reason to use more and more with less windows open. Firefox has been fine, Chrome is killing memory too. The App store and all its fanciness with web loaded content in the application tends to take up a ton as well.

Opening Applications in 32 bit mode by right clicking them in the application folder > get info> and then checking the box seemed to help before 10.7.1 but now safari still ***** up all the memory.

Adobe Air applications have been bad as well.. Yet that related to Adobe products like flash being resource hogs as well.

Has anyone experienced large Keynote files being open or Numbers documents taking up more resources than usual? This is just a question because I have had reports from users where I head up iOS and OSX Support.

Sidenote:This opens another issue that HTML5 video should be more videly adopted but its less secure with anyone being able to get it without a third party program. Apple needs to help out and come out with some new secure standards for HTML5 along with the current.

I have an early 13' mbp core i5 with 8gb of ram.

After upgrading to lion 10.7.1 i have less than 500mb available after just a few hours since i turn it on.

I must mention that the only running apps are: safari, mail, app store, ichat, ms excell and itunes.

Upon quitting excell, itunes, appstore and restarting mail and safari i still only get 1,5gb of free ram.

My suspicion is that there is another ram leak somewhere, or else i can't explain why my computer user 128mb of swap with 8gb of ram installed.

PS

Whenever i restart or shutdown my mbp i always uncheck the option to restart all programs and such...

A fresh restart gives me 6.5 gb of free ram.

andreiil wrote:

I have an early 13' mbp core i5 with 8gb of ram.

After upgrading to lion 10.7.1 i have less than 500mb available after just a few hours since i turn it on.

The OS is designed to retain data in memory for as long as it might be useful. Free memory is memory unused for anything -- there is no benefit whatsoever in the OS freeing up memory while it still might be reused. If you add more memory to the system the OS will use it -- that's what it is for.

Among other things, this means data from any web page you have visited may remain in memory even after you close that page & that some code loaded into memory during the launch of Safari or some other app may remain in memory even after you quit it. (This is why relaunching an app a short time after quitting it is often faster than launching it for the first time after a restart or after a longer interval after quitting it -- some of the code it needs may still be in memory, already initialized & ready to execute.)

If the OS did not do this, almost everything would take longer because much more code & data would have to be fetched from the HD, which takes many times longer than just switching retained memory pages from inactive to active.

So if your goal is to increase free memory, you are in effect trying to prevent the OS from using memory as efficiently as it can & trying to slow down its operation.

My suspicion is that there is another ram leak somewhere, or else i can't explain why my computer user 128mb of swap with 8gb of ram installed.

A 128 MB swap file is not unusual. The OS always creates at least one swap file, regardless of how much RAM is installed. It may use only a tiny fraction of the file's space, which is organized into pages of a few KB each & swapped page-by-page as needed, but it must reserve HD space for this "backing store" to prevent very bad things from happening if there is contention for HD space.

If you are having performance problems, check the page out numbers in Activity Monitor. If they are high, then you either don't hve enough memory for the apps you are running concurrently or you have a memory issue (which may or may not be a memory leak). But free memory numbers by themselves don't tell you anything about this.

Thanks for the broader explanation of "memory" and OS X, but you again refrain from elaborating more about the subject of page-outs. Excessive page-outs for anyone with 8 GB of memory has been inexplicable so far. I read that the way to tell if you need more memory is to compare page-ins vs. page-outs. If page-outs are more than 10% of page-ins, you need more memory. I went from 4 GB in Snow Leopard to 8 GB in Lion and my paging ratio has exceeded 30% on occasion (more than Snow Leopard). This is crazy, my machine's memory is maxed out. I never have more than 5 or 6 apps open simultaneously and never had any memory/ performance issues in 64bit Snow Leopard.

R C-R wrote:

andreiil wrote:

I have an early 13' mbp core i5 with 8gb of ram.

After upgrading to lion 10.7.1 i have less than 500mb available after just a few hours since i turn it on.

The OS is designed to retain data in memory for as long as it might be useful. Free memory is memory unused for anything -- there is no benefit whatsoever in the OS freeing up memory while it still might be reused. If you add more memory to the system the OS will use it -- that's what it is for.

I think you rely too much on the saying "free memory is wasted memory". I don't know how in the name of God a web browser and few other small applications are allowed (allowed, not supposed) to fill 8GB (10-20 times the size of Encyclopædia Britannica?) of RAM in few hours. Actually they shouldn't be allowed to do that not even in few days. Like Snow Leopard never did to me. Lion memory management system is bizarre the least, and for sure less efficient than the Snow Leopard one.

Two days ago I reached the top: 2.7GB of pages out just playing Angry Birds! Having 3GB of total RAM. It went on forever. Also getting some nice funny messages in the kernel.log:

Aug 28 23:44:16 BlueMoon kernel[0]: (default pager): [KERNEL]: ps_select_segment - send HI_WAT_ALERT

Aug 28 23:44:21 BlueMoon kernel[0]: (default pager): [KERNEL]: Switching ON Emergency paging segment

Aug 28 23:44:30 BlueMoon kernel[0]: (default pager): [KERNEL]: System is out of paging space.

Aug 28 23:44:31 BlueMoon kernel[0]: (default pager): [KERNEL]: ps_vstruct_transfer_from_segment - ABORTED

Aug 28 23:44:31 BlueMoon kernel[0]: (default pager): [KERNEL]: Failed to recover emergency paging segment

Aug 28 23:44:31 BlueMoon kernel[0]: macx_swapon SUCCESS

Aug 28 23:55:35 BlueMoon kernel[0]: (default pager): [KERNEL]: default_pager_backing_store_monitor - send LO_WAT_ALERT

Aug 28 23:55:55 BlueMoon kernel[0]: (default pager): [KERNEL]: ps_select_segment - send HI_WAT_ALERT

Aug 28 23:55:55 BlueMoon kernel[0]: (default pager): [KERNEL]: ps_vstruct_transfer_from_segment - ABORTED

Aug 28 23:55:55 BlueMoon kernel[0]: macx_swapoff FAILED - 35

Aug 28 23:55:56 BlueMoon kernel[0]: macx_swapon SUCCESS

Aug 28 23:56:26 BlueMoon kernel[0]: (default pager): [KERNEL]: default_pager_backing_store_monitor - send LO_WAT_ALERT

Aug 28 23:56:26 BlueMoon kernel[0]: macx_swapoff SUCCESS

Aug 29 00:14:54 BlueMoon kernel[0]: HFS: Very Low Disk: freeblks: 0, dangerlimit: 77

Aug 29 00:17:00 BlueMoon kernel[0]: (default pager): [KERNEL]: default_pager_backing_store_monitor - send LO_WAT_ALERT

Aug 29 00:17:02 BlueMoon kernel[0]: macx_swapoff SUCCESS

Having 100GB of free disk space the two messages in bold look like a big mess to me. I actually opened a bug report about it. Hopefully the ticket will get an update. Closed as duplicated would do. That would mean that Apple is investigating already the issue.

Anyway, 500MB of free RAM out of 8GB running few small apps makes little sense. 2.7GB of active backing store just running a web browser doesn't make any sense at all.

It would be an efficient way, if the system would release the inactive memory to applications that need more than the free memory has. But, unfortunately, it takes a pretty long time. Sometimes, up to several minutes of staring at the beach ball of death. And that's even without any page outs.

What i find very odd is that I don't have any pageouts, and that high ram usage is totally random.

For example, today i've been using my computer for 5 hours and i still get 5.07 gb of free ram.

I use the same apps every single day... mail, safari, appstore, itunes, ms excell, and the number of open tabs or excell spreadsheets is roughly the same, as i read the same pages and review the same reports on a daily basis.

Sometimes it hogs up all the 8gb of ram memory and starts using and increasing size of swap file and some days it barely uses 3gb of ram...

andreiil wrote:

What i find very odd is that I don't have any pageouts, and that high ram usage is totally random.

If you don't have any page outs then you are not running out of available RAM for the processes your Mac is currently running & there is no reason for the OS to free up any of it. What may appear to be random RAM use is just a consequence of the OS not releasing chunks of it when there is no good reason to do so.

For instance, if a browser loads & initializes its code used to show a movie or display an image in any of the many formats it supports, tap into an RSS feed, or any of the many other things modern browsers can do, that code may remain in memory long after it was first used. If it did not, then every time any process needed to use that code it would have to be fetched from the HD again, dramatically slowing the responsiveness of the system.

There is no simple way for users to accurately determine what part of RAM is used for what process or when it should be freed up. RAM isn't used just for document data -- even a tiny JPEG file may require hundreds of MB of code & working space to decompress & render the image.

Sometimes it hogs up all the 8gb of ram memory and starts using and increasing size of swap file and some days it barely uses 3gb of ram...

What exactly do you mean by "swap file"? Do you mean the actual, normally hidden swap files created by the system or something displayed by Activity Monitor? If if is from AM, what is it labeled as?

R C-R wrote:

What exactly do you mean by "swap file"? Do you mean the actual, normally hidden swap files created by the system or something displayed by Activity Monitor? If if is from AM, what is it labeled as?

It is labeled as "swap used", as you know already. That is the total amount of memory actually stored inside the /private/var/vm swap file(s). The same that iStat Menus reports. Also giving the total size of the swap files between parenthesis.

The point Adreiil was trying to make, which is the same it happens to me (again) lately, is that the way Lion manages the virtual memory is simply not deterministic. I have had no problems for a couple of weeks. Than recently, after upgrading Flash I started to have large chuncks of memory (4-500MB each) swapped out.

You keep repeating the theory of how the virtual memory management system should work in OS X Lion. And that is correct. For sure it is how it works in Snow Leopard but I very much doubt in Lion it has been implemented exactly in that way.

Where your description of virtual memory fails in Lion is that sometimes Lion simply does not release the active memory. That can be seen by mean of vm_stat, reporting the memory usage every 1s (which, contrarely to your belief, is a short enough time to analyze what it is happening): the free memory gets low, the acive and inactive memory raise. When the free memory goes below the low water mark the kernel starts paging out.

As you already know, before to start the resource's expensive job of paging out, the pager should first release the inactive read memory. That simply doesn't seem to happen. So, to save its beautiful cache of mostly useless read data it takes the road of writing on disk. Unless, sure, all inactive memory had been marked as written. At that point the page out would make complitely sense. What wouldn't make sense is why such large amounts of memory, mostly junk, have been marked for retention and not released already.

The virtual memory is not consistent in numbers. I had it twice now the case where my swap used raised to 2.8GB. Once from nothing to it. That is basically all the RAM in my system, wired memory (which can't be swapped out) excluded. Logic says that since only the inactive memory can be paged out (it's one reason why it's there after all) and since in my case it is always less than 1GB I shouldn't have more than 1GB of data paged out. Instead I get up to 3 times that amount. Not only that. Most of it didn't get released as free memory anyway. This in a session where the only high memory consuming application is Safari.

Now, we can argue as much as we like about what it is really going on inside the kernel. That at the end doesn't really matter. What it matters is that nearly 3GB of RAM paged out just running Safari is simply not acceptable. That didn't happen in Snow Leopard and doesn't happen when I use Windows 7 in the very same machine.

Last thing: you say that it is ok for Safari to retain the memory used by all pages. We are talking about more than 100MB per page in some cases, out of 1 or 2 MB of data retrieved from the Internet. Most of that memory should be working memory. With an average Internet connection it takes much longer to download a page than to load in memory the pieces of code needed from the disk, operation that can be (is) done in parallel. And most data must be rewritten anyway because the web is dynamic. So what would be the reason to keep that memory which is actually garbage? Masochism? It would just make the memory management totally inefficient. As it seem it does, actually.

Have you read Apple's own explanation of how Mac OS X handles VM and their note on how to read the information Activity Monitor provides?

In short, inactive memory contains cached data that, if needed, is much more quickly retrieved from the swap area than from the file system on-disk, the Internet, or any other store.

What Apple has yet to get precisely right is proactive freeing of inactive pages and quickly making those pages available to memory requests, and that's a long argument on the merits and demerits of Mach's externel pager mechanisms that is inappropriate for discussion here.

Yes, I did read it as much as I noticed that it is 3 years old. So pre Snow Leopard. And it doesn't work as you described. The inactive memory is the pool of processes pages that "got old" because not used recently, ready to be reallocated to the free memory by the dynamic_pager when needed. When are they needed? When the free memory goes below a water mark (God knows which, since Apple doesn't say anything about it).

So the page out should be zero as long as there is a minimum of free memory available. Often it isn't. Once it gets low the pager has to chose a chunck of inactive pages to be released. The ones that have been written must be paged out, because the processes may need them later. The read only pages (like the ones storing the code) simply gets reassigned to the free memory. They are not caches, they are actual proper memory belonging to the processes.

Now we can argue if it is better to release first the read only pages avoiding writing to the swap area when possible or to free the oldest pages no matter if they require to be paged out or less. And you're right, that is out of the topic here. What is not clear is why I've got a page out of nearly 3GB when just writing in this same forum. Basically all my memory. Or why 8 GB are still not enough to run some simple applications. Sometimes. Because for 2 weeks I had no issues whatsover. And in Snow Leopard I could have an uptime of weeks without any page out. So the question is: what did change in Lion? Is it a bug? A wrong design decision? Lion became inefficient? Where does the memory go? That's what I'd like to know and I am researching it as much as I can. To keep reading that it is fine as it is, is not an answer.

BTW regarding the note. If "page outs" is the cumulative amount of data that Mac OS X has moved between RAM and disk space (so with cumulative I assume the total in time) and "swap used" is the amount of information copied to the swap file on the hard drive (again, I assume the actual amount of information) how comes that the second is bigger?? This is driving me crazy. Not in terms of solving an issue I have no power over it, just in terms of understanding.