Re: [squid-users] 2.4.20-aa and LARGE Squid process -> SIGSEGV

On Sat, 2002-12-21 at 20:57, Ralf Hildebrandt wrote:
> Hi!
>
> Right now we're trying to implement a large scale Squid proxy on
> Debian/testing. We're using 2.4.20-aa and Squid-2.4.7-1.
>
> We're encountering sporadic crashes of the squid children (SIGSEGV,
> signal 11). We were investigating in several directions:

Which squid children? Squid only spawns 1 one squid process. If you are
using aufs, you may see many pids on linux, but that is due to a
'feature' of the linux thread support where each thread gets a different
pid.

> * the Kernel has highmem support enabled (we have 2GB physical RAM and
> 4 GB swap)

I've not used linux w/ highmem support - what changes does it make to
the kernel memory interface?
 
> * We recompiled Squid as Debian Package using gcc-3.2, since --
> according to the FAQ -- squid may crash with signal 11 with
> optimization enabled when using gcc-2.95.4 (Debian uses gcc-2.95.4,
> but still build squid using -O!)

-O is ok. The documented fault occurs in the internal dns protocol
support, due to an optimisation bug.
 
> * we closely observed dmesg, messages and syslog. No oddities were
> found. Squid simply crashes with signal 11.

What about cache.log?

> * we tried both ufs and aufs as cache filesystems, since the FAQ tells
> us the async I/O may have bugs. Yet, the crashes still occur.

diskd doesn't have any known bugs, and for any large squid install, ufs
will perform *very* poorly.
 
> To no avail -- Squid simply crashes from time to time. It's
> impossible to predict when.

Squid should trap the crash and log it to cache.log.
 
> Attached is the config we use (minus commentaries and empty lines --
> note that we precautiously changed cache_mem to 500 MB BUT never
> restarted; so we're still using cache_mem 700 MB).

> If we raise our current setting of "cache_mem 700 MB" in any way (e.g.
> to 1000 MB), the squid will eventually crash -- very often after about
> 1h. Note that we have 2GB of RAM.
>
> Questions:
>
> * What are we doing wrong?
> * Are other people using Squid with 2GB of RAM? How?

Yes, without doing anything special. Another user was talking recently
about use of squid on 4GB physical RAM machines.

> cache_mem 500 MB
> maximum_object_size 65536 KB
> cache_dir aufs /squid-cache0 16000 256 256
> cache_dir aufs /squid-cache1 16000 256 256

This configuration will attempt to use ~ 900Mb of RAM. That should be
fine.

As to your problem, may I suggest the following:
1) Use diskd. A nasty aufs bug was found recently that could impact you.
2) Check cache.log as your primary source for squid diagnostics.
3) Check squid memory use via cache manager for statistics.
4) Use squid 2.5-stable-cvs if you really want aufs support, the bug is
fixed there.

Step 2 should give you a handle on whats going on. If it doesn't, try
running squid with ufs cache dirs under gdb, and get a backtrace of the
failure.

Cheers,
Rob