Re: file descriptors for squid using linux 2.2.1

Snippet from FAQ on Squid Home Page below:

========================================================================================================================

If you see the Too many open files error message, you are most likely running
out of file descriptors. This may be due to running Squid on an
operating system with a low filedescriptor limit. This limit is often
configurable in the kernel or with other system tuning tools. There are two ways
 to run
out of file descriptors: first, you can hit the per-process limit on file
descriptors. Second, you can hit the system limit on total file descriptors for
all
processes.

Linux

Have a look at filehandle.patch by Michael O'Reilly

If your kernel version is 2.0.35 or greater, you can read and write the maximum
number of file handles and/or inodes simply by accessing the special files:

        /proc/sys/kernel/file-max
        /proc/sys/kernel/inode-max

So, to increase your file descriptor limit:

        echo 3072 > /proc/sys/kernel/file-max

While this does increase the current number of file descriptors, Squid's
configure script probably won't figure out the new value unless you also update
the include files, specifically the value of OPEN_MAX in
/usr/include/linux/limits.h.

==========================================================================================================================

However, the above doesn't seem to be explicit enough to me. The next bit only
relates to a 2.0 Kernel , but I hope the argument below is sufficiently
generic.

In limits.h on a 2.0.36 box I have:

[root@thingy /root]# cat /usr/include/linux/limits.h
#ifndef _LINUX_LIMITS_H
#define _LINUX_LIMITS_H

#define NR_OPEN 256

#define NGROUPS_MAX 32 /* supplemental group IDs are available */
#define ARG_MAX 131072 /* # bytes of args + environ for exec() */
#define CHILD_MAX 999 /* no limit :-) */
#define OPEN_MAX 256 /* # open files a process may have */
#define LINK_MAX 127 /* # links a file may have */
#define MAX_CANON 255 /* size of the canonical input queue */
#define MAX_INPUT 255 /* size of the type-ahead buffer */
#define NAME_MAX 255 /* # chars in a file name */
#define PATH_MAX 1024 /* # chars in a path name */
#define PIPE_BUF 4096 /* # bytes in atomic write to a pipe */

#endif

[root@thingy kernel]# cat /proc/sys/kernel/file-max
1024

Meanwhile, in /usr/include/linux/posix_types.h

/*
 * This allows for 1024 file descriptors: if NR_OPEN is ever grown
 * beyond that you'll have to change this too. But 1024 fd's seem to be
 * enough even for such "real" unices like OSF/1, so hopefully this is
 * one limit that doesn't have to be changed [again].
 *
 * Note that POSIX wants the FD_CLEAR(fd,fdsetp) defines to be in
 * <sys/time.h> (and thus <linux/time.h>) - but this is a more logical
 * place for them. Solved by having dummy defines in <sys/time.h>.
 */

/*
 * Those macros may have been defined in <gnu/types.h>. But we always
 * use the ones here.
 */
#undef __NFDBITS
#define __NFDBITS (8 * sizeof(unsigned long))

#undef __FD_SETSIZE
#define __FD_SETSIZE 1024

=============================================================================================

I believe the combination of the above settings and include files means that the
 the total number of Open Files across the
WHOLE system - as given by /proc/sys/kernel/file-max - can be set to as much as
you like... but must not exceed the number set by FD_SETSIZE
which is used to set the size of a descriptor table used by select() and its
ilk.

The total number of FD's open by any given process is limited by OPEN_MAX, but
again, this must be less than FD_SETSIZE to avoid a single
process opening more FD's than select's descriptor table can handle.

So.

/proc/sys/kernel/file-max <= FD_SETSIZE == Total No. of File Descriptors
available across the system.

OPEN_MAX <= FD_SETSIZE == Max No. of open FD's any given
process may have open.

Since OPEN_MAX is in a header file, and hence is programmed in at compile time,
the squid process(es) can be configured to use more FD's than
other executables by adjusting limits.h at compile time.

I've probably got this all wrong, but at least this might provoke someone to
give the more explicitly accurate answer....

Ted Rule,

"Irfan Akber" <irfan@inet.com.pk> on 22/02/99 21:42:56

To: "Marlon Anthony Abao" <Marlon.Abao@Infodyne.Net>
cc: "squid-users" <squid-users@ircache.net> (bcc: Ted Rule/160GPS/Flextech/UK)

Subject: Re: file descriptors for squid using linux 2.2.1

Well squid sometimes cannot detect the actual limits. As in your earlier
email ulimit -a shows openfiles as 1024 that means squid should be able to
use 1024 open files. I dont know why it cant detect it. I guess a patch is
required then.

Irfan Akber

----------
> From: Marlon Anthony Abao <Marlon.Abao@Infodyne.Net>
> To: Irfan Akber <irfan@inet.com.pk>
> Subject: Re: file descriptors for squid using linux 2.2.1
> Date: Tuesday, February 23, 1999 6:22 AM
>
> won't this break things??
>
> if during compile, it allocates only 256, could i really force it to use
> the value in this tag?
>
>
> At 06:53 PM 2/22/99 -0000, you wrote:
> >Another thing that came to mind is max_open_disk_fds in squid.conf. Try
> >increasing this too.
> >
> >Irfan Akber
> >
> >----------
> >> From: Marlon Anthony Abao <Marlon.Abao@Infodyne.Net>
> >> To: Irfan Akber <irfan@inet.com.pk>
> >> Cc: squid-users <squid-users@ircache.net>
> >> Subject: Re: file descriptors for squid using linux 2.2.1
> >> Date: Tuesday, February 23, 1999 5:59 AM
> >>
> >> irfan,
> >>
> >> this is the new /proc/ filesystem in my box.
> >>
> >> cache:/proc/sys/fs# ls
> >> binfmt_misc/ dquot-max file-max inode-max inode-state
> >super-nr
> >> dentry-state dquot-nr file-nr inode-nr super-max
> >> cache:/proc/sys/fs# cat file-max
> >> 4096
> >> cache:/proc/sys/fs# cat inode-max
> >> 8192
> >> cache:/proc/sys/fs#
> >>
> >> and still i get a 256 value for fds on squid
> >>
> >> -marlon
> >>
> >> ps. anyone running squid on the new 2.2.x linux kernels?
> >>
> >>
> >>
> >> At 06:02 PM 2/22/99 -0000, Irfan Akber wrote:
> >> >I am not sure but it could be configurable using the /proc
filesystem.
> >Try
> >> >increasing the value of file-max.
> >> >
> >> >Irfan Akber
> >> >
> >> >----------
> >> >> From: Marlon Anthony Abao <Marlon.Abao@Infodyne.Net>
> >> >> To: squid-users@ircache.net
> >> >> Subject: file descriptors for squid using linux 2.2.1
> >> >> Date: Tuesday, February 23, 1999 4:02 AM
> >> >>
> >> >> guys,
> >> >> i just installed squid on a new box running linux kernel 2.2.1 and
> >squid
> >> >> complains that my kernel does not have enough file descriptors
(only
> >> >256).
> >> >>
> >> >> i thought that the new kernel has already lifted this limit without
> >need
> >> >> for anymore filehandle patches.
> >> >>
> >> >> is there any way to inform squid that this kernel could now handle
> >more
> >> >> than 256 fds at a time? or is there a special setting on the
kernel
> >to
> >> >> enable this?
> >> >>
> >> >> thanks.
> >> >>
> >> >> -marlon
> >> >>
> >> >
> >> >
> >
> >
Received on Mon Feb 22 1999 - 10:59:31 MST