Does OS X Mountain Lion have a built-in virtual memory limit (and how can I get rid of it)?

Mountain Lion has three memory management features:

1. Protected memory

2. Dynamic memory allocation

3. Secure memory allocation

Secure memory allocation is used to protect the system from malicious processes. But you are asking about the second, dynamci memory allocation.

The operating system automatically manages system memory for processes at their request. Though real memory is limited by hardware, the system dynamically allocates both real and virtual memory when needed. The only memory limitations in OS X are the size of installed RAM and the amount of free space you have available on your system volume.

The question then, becomes are you sure about the amount of free space you believe to be available? The ML dynamic memory management process should be using all free space on the mass storage, irrespective of the amount of RAM, that is, it is not a function or multiple of installed RAM.

After loading STATA, check the actual free space you have by going to Applications > Utilities > Activity Monitor and check out the storage to see what the system says is available.

What version of STATA do you have? Looking at their site in the support sections there seem to be a number of questions/comments on memory usage.

And to answer your first question there is no limit in VM. This article from Apple, though fairly technical, might explain it for you.

About the Virtual Memory System

There is a limit to the size of the virtual memory backing store, though I don't know how it's calculated. The error message means that the limit is being reached. You can't disable memory paging, and if you could the result would be that the system would crash.

I take this statment from the link I posted

Note: Unlike most UNIX-based operating systems, OS X does not use a preallocated disk partition for the backing store. Instead, it uses all of the available space on the machine’s boot partition.

to basically mean it's limited only by the disk space.

There is also this:

Both OS X and iOS include a fully-integrated virtual memory system that you cannot turn off; it is always on. Both systems also provide up to 4 gigabytes of addressable space per 32-bit process. In addition, OS X provides approximately 18 exabytes of addressable space for 64-bit processes. Even for computers that have 4 or more gigabytes of RAM available, the system rarely dedicates this much RAM to a single process.

To give processes access to their entire 4 gigabyte or 18 exabyte address space, OS X uses the hard disk to hold data that is not currently in use. As memory gets full, sections of memory that are not being used are written to disk to make room for data that is needed now. The portion of the disk that stores the unused data is known as the backing store because it provides the backup storage for main memory.

The article is not completely up to date. Before 10.8, I believe it was possible to fill up the boot volume with swap files, but that's no longer the case. That's why people now get messages about "no more space for application memory" when the volume still has plenty of free space. The change in the kernel isn't documented anywhere that I know of, so you'd have to dredge through the source code to find it.

ok thanks much

I think the problem stems from the design of that application. It probably tries to load the whole data set into memory, which is obviously not going to work for a data set of that size. Whether there is anything you can do about it, I don't know. You'd have to refer to the developer.

quaem wrote:

this feature is only a "problem" given Mountain Lion's undocumented limitation on VM size, which is the topic of this thread.

You have not proven any undocumented limitation on VM size. It is probably just a bug in STATA. Virtual memory is not a straightforward concept. it is the OS fighting with applications. Neither side is playing fair. Applications will typically ask for much more memory than they really need. In response, operating systems typically lie to them and only give them what they really use. It's a delicate dance and you've brought a rhinoceros to the ball.

People seem to use STATA because it is cheap and must faster because it works entirely in memory. If you are relying on VM, you've lost on point 2 anyway. Just use a different program designed to handle such large datasets.

Can I take it from your reply that you don't know how to modify this VM size limitation

Yes. I don't believe it's possible. There is certainly no documented way to change it that I know of.

In my opinion, the STATA developers are shirking their responsibility for memory management, which is their job, not the kernel's. They're dumping everything into memory and relying on the kernel to do the rest. Not only does that strategy result in poor performance, it sometimes results in complete failure, as you've discovered. What they could, and should, do is memory-map the data files and move pages in and out of memory efficiently.

The only thing you can do under the circumstances is to run the application on a Mac Pro with a ridiculous amount of RAM. I believe the current model can use up to 128 GB.

Not being familiar with the STATA application, I am by no means an expert on your particular scenario. That being said, it would not be the first time an advanced Unix application ran into memory problems on Mac OS X. Both IBM's Informix and Postgresql had similar issues in Mac OS X. You can adjust some shared kernel memory settings by following the instructions here:

http://www.spy-hill.com/myers/help/apple/SharedMemory.html

http://benscheirman.com/2011/04/increasing-shared-memory-for-postgres-on-os-x/

kern.sysv.shmmax must be set to EXACT multiples of 4096 or the setting may be ignored by OS X.

More info here:

http://www.postgresql.org/docs/9.1/static/kernel-resources.html

It's most likely the Shared Memory setting in the kernel as it's not set to a value conducive to servers but simpler workstations. You are dealing with a heck of a lot of data in STATA. Now the developers of STATA may not have provided you with an accurate error message to fully describe the problem. The error you see indicates you are out of application disk storage space which is simply not the case. The real problem could be it ran out of shared memory which is entirely possible with the conservative settings that Mac OS X employs.

Shared memory doesn't have anything to do with virtual memory but very well may be the root cause of your problem.

You are certain that you have the 64 bit version? Seems 12 comes in both flavors (32 and 64).

Love this bit from the FAQ

RAM

The most important consideration when buying a computer on which to run Stata is the amount of RAM (memory) you will need. You need at least 512 MB of RAM for Stata to run smoothly. Stata loads all of your data into RAM to perform its calculations. You must have enough physical RAM to load Stata and allocate enough memory to it to load and analyze your datasets.

Stata will be drastically slowed if the operating system has to use virtual memory to load your data or perform its calculations. One of the issues you have to consider when deciding how much RAM to purchase is the size of the datasets you will be working with. We recommend that your computer contain 50% more memory than the size of your largest dataset. Stata needs the extra room in memory to perform calculations, create temporary variables, etc., once the data have been loaded.

One last thing. This would seem to be something the Stata developers would like to here about. Hard to believe they are not aware of the situatuin if there was this big a change (and I emphasis if) in ML VM.

The error you see indicates you are out of application disk storage space which is simply not the case.

No, that's not what the error indicates, and the error isn't coming from the application, it's coming from the kernel. It means that the backing store is maxed out.