How to increase the limit of “maximum open files” in C?

acemtp wrote:
The default limit for the max open files on Mac OS X is 256 (ulimit -n) and my application needs about 400 file handlers.

Why?

I think etre rather meant, why does an application need 400 files opened +at the same time+. The default of 256 probably was defined as "let's take a large number that no-one ought to need".

acemtp wrote:
The problem is that we have a function 'setrlimit' that doesn't work on Mac OS X. The max limit was unlimited, so it should be possible to increase the current limit from 256 to at least 400 but it seems it's not possible... why?

I don't know but what about "not to put useless burden" on the OS ?
Ok, you are the boss ..
Did you read the man page ?
If I use Bwana (thanks to être for the link http://www.bruji.com/bwana/) and type man:setrlimit in Safari's navigation field I can read
" Only the super-user may raise the maximum limits. Other users may only
alter rlim_cur within the range from 0 to rlim_max or (irreversibly)
lower rlim_max."
Same if you type man setrlimit in the Xcode help menu for Man Pages or in the terminal..
So, maybe, you were not superuser when you tried to put machine at hard work for your purposes ...
Or maybe any other reason given in the man page.

Try this:

#include <stdio.h>
#include <stdlib.h>
#include <sys/resource.h>
 
int
main(void)
{
 struct rlimit rlp;
 FILE *fp[10000];
 int i, r;
 r = getrlimit(RLIMIT_NOFILE, &rlp);
 if (r == -1) {
 perror("getrlimit()");
 exit(1);
 }
 printf("before %d %d ", (int) rlp.rlim_cur, (int) rlp.rlim_max);
 rlp.rlim_cur = 10000;
 r = setrlimit(RLIMIT_NOFILE, &rlp);
 if (r == -1) {
 perror("setrlimit()");
 exit(1);
 }
 r = getrlimit(RLIMIT_NOFILE, &rlp);
 if (r == -1) {
 perror("setrlimit()");
 exit(1);
 }
 printf("after %d %d ", (int) rlp.rlim_cur, (int) rlp.rlim_max);
 for (i = 0; i < 10000; i++) {
 fp = fopen("a.out", "r");
 if (fp == NULL) {
 perror("fopen()");
 fprintf(stderr, "i == %d ", i);
 exit(1);
 }
 }
 exit(0);
}

You will notice that I

o Added <stdlib.h> to use exit().
o Added a check for an error return on all system calls
o Use perror() if a system call fails

When I run your program either as "me" or as root I see (correctly)

$ cc -Wall -o rlimit rlimit.c
$ touch a.out && ./rlimit
before 256 -1
after 10000 -1
fopen(): Too many open files
i == 9997

So a.out is opened 9997 times, which is what you expect as standard in (0), out (1) and error (2) are open by default.

As to why you want to have that many files open, having lots of network connections open in a server is not uncommon.

Regards,
Hans

{quote}
I bet that if you execute "ulimit -n" in a terminal, you'll have more than 256 (perhaps unlimited).
{quote}

Lucky we didn't enter in a wager 😉

$ ulimit -n
256

Maybe there is a difference between 10.5 and 10.6. I run 10.5.8. What do these two say on your system?

$ sysctl kern.maxfiles
kern.maxfiles: 12288
$ sysctl kern.maxfilesperproc
kern.maxfilesperproc: 10240

Regards,
Hans

{quote}
So it's really a new bug introduced with 10.6?
{quote}

Highly unlikely, I guess half the (Mac) world would keel over if such an obvious bug was introduced. Are you sure setrlimit(2) does not return -1 on your system?

How much memory do you have in your system, I have 4GiB. That could influence the number of files that can be open.

My 10.4 PowerBook gives the same numbers as 10.5, but does not want to open more than 253 files, both as a normal user and as root. It has a (mere) 512MiB.

An other way to find the cause is to look through the kernel source, see [http://www.opensource.apple.com/release/mac-os-x-1064/xnu-1504.7.4]. I'll give it a try this afternoon as I have a long and boring train ride ahead...

Regards,
Hans

hansz wrote:
As to why you want to have that many files open, having lots of network connections open in a server is not uncommon.

But a good server would fork enough processes to handle the load instead of trying to do it all in one app.

There is a good reason for this limit. The select() function, by default, is limited to this value. So, not only is our hapless original poster not using select(), the program is going to require admin authorization each and every time it runs.

There is a reason why increasing this limit is so hard - you aren't supposed to do it!

There is no reason to have more than 256 files open. Whatever you are doing, you are doing it wrong.

The setrlimit() function only applies to new processes. It does not affect the current process. You can either run "ulimit -n 10000" in the shell and then start your program or split your program into two segments:

#include <stdio.h>
#include <stdlib.h>
#include <sys/resource.h>
#include <unistd.h>
int main(void)
 {
 struct rlimit rlp;
 int r;
 r = getrlimit(RLIMIT_NOFILE, &rlp);
 if (r == -1)
 {
 perror("getrlimit()");
 exit(1);
 }
 
 printf("before %d %d ", (int) rlp.rlim_cur, (int) rlp.rlim_max);
 rlp.rlim_cur = 10000;
 
 /* I'm curious how anyone ever expected it to work or got it to work
 without setting the max, because it sure seems to be required. */ 
 rlp.rlim_max = 10000;
 
 r = setrlimit(RLIMIT_NOFILE, &rlp);
 if (r == -1)
 {
 perror("setrlimit()");
 exit(1);
 }
 r = getrlimit(RLIMIT_NOFILE, &rlp);
 if (r == -1)
 {
 perror("setrlimit()");
 exit(1);
 }
 
 printf("after %d %d ", (int) rlp.rlim_cur, (int) rlp.rlim_max);
 return execl("/tmp/f", "/tmp/f", NULL);
 }

And the source to the "f" program is:

#include <stdio.h>
 
int main(void)
 {
 FILE *fp[10000];
 int i;
 for(i = 0; i < 10000; i++)
 {
 fp = fopen("a.out", "r");
 if (fp == NULL)
 {
 perror("fopen()");
 fprintf(stderr, "i == %d ", i);
 
 return 1;
 }
 }
 return 0;
 }

[jdaniel@Pele:584] /tmp $ a.out
before 256 256
setrlimit(): Operation not permitted
[jdaniel@Pele:585] /tmp $ sudo a.out
before 256 256
after 10000 10000
fopen(): Too many open files
i == 9997

As you can see, it is convoluted and a real hassle. It isn't a bug in MacOS X 10.6 or any other version.

So, I repeat my question yet again. Why do you want to open more than 256 files? It is, of course, a rhetorical question. Don't do it. If you feel you need more than 256 open files, review your architecture and correct it.

FYI, if I run on my 10.6.4

Path:~/xcodeprojects/wall/build/debug$touch a.out
Path:~/xcodeprojects/wall/build/debug$./wall
before 256 -1
after 10000 -1
fopen(): Too many open files
i == 253

If I run under Xcode

run
[Switching to process 815]
Running…
before 9472 -1
after 10000 -1
fopen(): Too many open files
i == 9469
Debugger stopped.
Program exited with status value:1.(gdb) 

Seems to validate etresoft.