Is "Error: Too Many Open Files" caused by limited number of 'maxfiles'?

Question

Level 1

10 points

Is "Error: Too Many Open Files" caused by limited number of 'maxfiles'?

I notice that in the Terminal you can temporarily (until your next reboot) increase the maximum number of files capable of running on Snow Leopard from 256 (default) to 10,200. What's odd is that the system reports that 'maxfiles' has a maximum value of 'unlimited,' yet when I try to exceed 10200 files, even though it doesn't report any errors, it won't surpass this value. Is there a reason for that? Do you all notice the same problem on Snow Leopard? Is there a fix for this?

Example: launchctl limit maxfiles 10200 { WORKS 🙂 }
BUT
launchctl limit maxfiles 10300 { or above FAILS 😟 }

Black MacBook, Mac OS X (10.5.2), Intel Core 2 Duo 2.2 GHZ

Posted on Mar 22, 2010 1:29 AM

Reply

Answer 1

Jun T.

Level 4

2,185 points

Mar 22, 2010 5:13 AM in response to JFraser

'launchctl limit' just controls the resources used by launchd. Kernel level limit is controlled by sysctl command.


$ sysctl kern.maxfiles
kern.maxfiles: 12288

See 'man sysctl' for more detail.

Reply

Answer 2

JFraser Author

Level 1

10 points

Mar 23, 2010 12:41 AM in response to Jun T.

I was able to raise kern.maxfiles to 1,000,000 but, upon doing so, I can't also raise ulimit -n nor launchctl limit maxfiles beyond 10,200. Is there a reason for this?

Reply

Answer 3

Mar 23, 2010 6:56 AM in response to JFraser

There are 2 limits for the number of open files per processor.

The current limit, and the max limit. You can set both to "unlimited". But, unlimited
on unix and unix like systems, has a specific value. On OS X it is 10240. To increase the value beyond that there are 2 options. 1) file a bug with apple. 2) download the kernel source code, change the value and rebuild and install the kernel. Keep in mind though, just changing the value may not be all that is needed to make the change work.

xnu-1228.3.13

[./bsd/kern/kern resource.c|http://www.opensource.apple.com/source/xnu/xnu-1228.3.13/bsd/kern/ker nresource.c]


 117 /*
 118 * Limits on the number of open files per process, and the number
 119 * of child processes per process.
 120 *
 121 * Note: would be in kern/subr_param.c in FreeBSD.
 122 */
 123 int maxfilesperproc = OPEN_MAX; /* per-proc open files limit */

[./bsd/sys/syslimits.h|http://www.opensource.apple.com/source/xnu/xnu-1228.3.13/ bsd/sys/syslimits.h]


 88 #define OPEN_MAX 10240 /* max open files per process - todo, make a config option? */

Message was edited by: Nils C. Anderson

Reply

Answer 4

BobHarris

Level 9

53,147 points

Mar 23, 2010 7:13 AM in response to JFraser

Perhaps there is another way to get the job done without
opening so many concurrent files.

Can you describe the algorithm that needs this many files
to be opened. A few of the people hanging around here
have "Been There, Done That" before, and may be able to
suggest an alternate approach.

Reply

Answer 5

Jun T.

Level 4

2,185 points

Mar 23, 2010 8:03 AM in response to Nils C. Anderson

First, it seems I was wrong when I wrote 'sysctl' should be used instead of 'launchctl limit'; actually, both seem to have "similar" effects (at least "apparently"). For example


$ sysctl kern.maxfiles
12288
$ sysctl kern.maxfilesperproc
kern.maxfilesperproc: 10240
$ sudo launchctl limit maxfiles 13000
$ sysctl kern.maxfiles
13000
$ sysctl kern.maxfilesperproc
13000 # maxfilesperproc is also increased

Of course I didn't test whether more than 12288 open files can actually exist in the system, or a process can open more than 10240 files at the same time.

JFraster,
Did you run sysctl or launchctl with 'sudo' when increasing the limit?
Do you want to increase KERN_MAXFILES or KERN_MAXFILESPERPROC (or both)?

Nils,
sysctl(3) man pages show both KERN_MAXFILES and KERN_MAXFILESPERPROC as "changeable". Of course "changeable" may not necessarily mean "can be increased". But 'sysctl -w kern.maxfilesperproc' seems to work (or at least gives not error messages), although kern.maxfilesperproc does not appear in sysctl(8) man pages.


--- after reboot ---
$ sysctl kern.maxfiles
kern.maxfiles: 12288
$ sysctl kern.maxfilesperproc
kern.maxfilesperproc: 10240
$ sudo sysctl -w kern.maxfilesperproc=13000
kern.maxfilesperproc: 10240 -> 13000
$ sysctl kern.maxfilesperproc 
kern.maxfilesperproc: 13000
$ sysctl kern.maxfiles 
kern.maxfiles: 12288 # not affected. smaller than maxfilesperproc
$ sudo sysctl -w kern.maxfiles=14000
kern.maxfiles: 12288 -> 14000
$ sysctl kern.maxfiles 
kern.maxfiles: 14000
$ sysctl kern.maxfilesperproc 
kern.maxfilesperproc: 13000 # not affected, contrary to the case of 'launchctl limit'

As you see, maxfilesperproc can be set to larger than maxfiles; this is inconsistent, and I wonder whether the above commands are actually working, although there are no error messages.

Reply

Answer 6

JFraser Author

Level 1

10 points

Mar 23, 2010 10:23 PM in response to Jun T.

I was able to permanently raise maxfiles systemwide by using:

sudo pico /etc/sysctl.conf
kern.maxfiles=1000000 (arbitrary value)
kern.maxproc 2500

sudo pico /etc/launchd.conf
maxfiles 1000000
maxproc 2500

Then I rebooted the machine and the numbers stuck. It's been a day now with no red in vuze (i.e. "Too many open files" error) and hence the problem seems pretty much solved. I notice in the Activity Monitor, however that the number of Ports for Vuze alone is about 1800 (which I think represents the number of processes being consumed by Vuze), but if maxproc is limited to 2500, since I can't set the values any higher in those 2 files without them automatically reverting to a smaller size upon reboot, then doesn't that fact still make the whole process unstable? Is there any other way that you all know to increase maxproc beyond 2500 on OS X (non-server based)? I'm pretty sure that in server-based versions of OS X you're simply able to modify the /etc/r.server file, which doesn't exist in regular versions of the OS. So, how about without it?

Reply

Answer 7

Jun T.

Level 4

2,185 points

Mar 24, 2010 5:08 AM in response to JFraser

Ports for Vuze alone is about 1800 (which I think represents the number of processes being consumed by Vuze

The "Ports" reported by the Activity Monitor is the number of Mach ports used by the process. I don't know whether there is an upper limit for the number of Mach ports in the system (or per process). But anyway it has no relation with the number of processes.

If you are using the "All Processes" display mode (from the pop up menu) in the Activity Monitor, then the number of processes in the system is just the number of lines in the Activity Monitor; I guess it is far less than 2500. If you chose the "All Processes, Hierarchically" from the pop up, then if there are child processes of Vuze then they are shown when you click on the triangle to the left of the Vuze.

Reply