Re: [squid-users] Squid and Squidguard using high disk IO

Hello Kaya,
first, don't forget to look at sysctl kern.maxfiles values.
Also improve daemon FD values in login.conf for squid. Don't forget each
connection is a FD (1 connection for the client, 1 for the transaction
to remote site, somes for the caching).

Also to improve performances of squidguard, i stored all blacklists DB
to a memory fs (mfs) this improve massively squidguard performance
I have wrote an article to improve squid perfs on OpenBSD:
http://www.unix-experience.fr/2013/monter-un-proxy-cache-performant-avec-squid-et-openbsd/

-- 
Best regards,
Loïc BLOT, 
UNIX systems, security and network engineer
http://www.unix-experience.fr
Le samedi 09 novembre 2013 à 19:39 +0000, Kaya Saman a écrit :
> Just found this is Squid cache log:
> 
> 2013/11/09 19:28:25 kid1| /var/squid/cache/04/7A: (24) Too many open files
> 2013/11/09 19:31:31 kid1| WARNING: All 20/20 redirector processes are busy.
> 2013/11/09 19:31:31 kid1| WARNING: 20 pending requests queued
> 2013/11/09 19:31:31 kid1| WARNING: Consider increasing the number of 
> redirector processes in your config file.
> 
> 
> The cache size is 2GB.... though that shouldn't affect performance as 
> far as I understand.
> 
> On 11/09/2013 05:23 PM, Eliezer Croitoru wrote:
> > Hey,
> >
> > Notes inside.
> >
> > On 11/09/2013 05:58 PM, Kaya Saman wrote:
> >>
> >> What can I do to improve performance with this?
> >>
> >>
> >> Is this line too high: url_rewrite_children  500
> > YES!!
> >
> >> or simply have a misconfigured something?
> >
> >
> >> I additionally have 'c-icap' running with squidclamav coupled to clamd
> >> in case that is of importance - not using the squidGuard line in the
> >> squidclamav.conf file!!!
> >>
> >> Basically how can I get the IO usage down and get the system to work 
> >> again?
> > For how many users exactly?
> > Just a note that I am not in a favor of any OS by default but I would 
> > feel better Using Linux.
> >
> >>
> >> - the logs don't indicate anything outside of 'starting squidGuard
> >> process' many times.
> > The basic assumption of using 500 child process is that you have 
> > atleast 100 CPUs.
> > SquidGuard was design for performance which is lots of urls per sec.
> > It can be tested just to clear the point out.
> > for example in a rate of 1500k requests per second you should not have 
> > a need in more then 40-50 children.
> > In practice it works a bit different speed since there is a speed 
> > limit on STDIN and STDOUT which slows down the speed of squid and 
> > squidguard communication blocking the whole squid instance(in a way).
> >
> > If you need basic url filtering you can use ICAP which has an option 
> > to run as a standalone service outside of squid settings and machine.
> >
> > I have written in the past a small ICAP service for the favor of 
> > requests manipulation and filtering.
> > I have never finished it in a level I was happy with but the basic 
> > code can be seen here:
> > https://github.com/elico/echelon
> >
> > I know for a fact that ICAP interface adds concurrency by the "nature" 
> > of it using TCP.
> >
> > This is not the place to ask about concurrency in squidguard which can 
> > allow the usage of square less processes(children) for more requests.
> >
> > In order to find the right number of children start with 40 and see if 
> > it fits you and then see what is the bottle neck in the whole setup.
> >
> > Eliezer
>