Re: [squid-users] 2.4.20-aa and LARGE Squid process -> SIGSEGV

What you should do:

1. Use Squid-2.5.STABLE1 with suitable patches applied. With the
release of Squid-2.5.STABLE Squid-2.4 is no longer actively supported
by the developers.

2. When encountering a SIGSEGV/SIGFPE/SIGILL/SIGBUS the developers
need a stacktrace to be able to pinpoint the source of the error. See
the Squid FAQ on various methods of getting a stacktrace from a
crashing Squid. When such stacktrace is found and you are running the
current STABLE release (with appropriate patches) please register a
bugzilla entry for your problem.

3. As you have already noticed, try building Squid without
optimization enabled in the compiler (no -O option or -O0). But
please do this after extracting the stacktrace above..

4. After some time, please return to your bugzilla entry and update if
compiling without optimization did help or not.

Regards
Henrik

On Saturday 21 December 2002 10.57, Ralf Hildebrandt wrote:
> Hi!
>
> Right now we're trying to implement a large scale Squid proxy on
> Debian/testing. We're using 2.4.20-aa and Squid-2.4.7-1.
>
> We're encountering sporadic crashes of the squid children (SIGSEGV,
> signal 11). We were investigating in several directions:
>
> * the Kernel has highmem support enabled (we have 2GB physical RAM
> and 4 GB swap)
>
> * we tried different kernels (with or without highmem support)
>
> * we tried another squid version (Squid-2.4.6 from the "stable"
> distribution of Debian)
>
> * We recompiled Squid as Debian Package using gcc-3.2, since --
> according to the FAQ -- squid may crash with signal 11 with
> optimization enabled when using gcc-2.95.4 (Debian uses
> gcc-2.95.4, but still build squid using -O!)
>
> * we closely observed dmesg, messages and syslog. No oddities were
> found. Squid simply crashes with signal 11.
>
> * we tried both ufs and aufs as cache filesystems, since the FAQ
> tells us the async I/O may have bugs. Yet, the crashes still occur.
>
> * We use two identical machines -- the crash happens on both
> machines. Same CPU, disks, RAM, manufacturer, heck -- even the same
> series! Yet, the crashes still occur on both machines.
>
> To no avail -- Squid simply crashes from time to time. It's
> impossible to predict when.
>
> Attached is the config we use (minus commentaries and empty lines
> -- note that we precautiously changed cache_mem to 500 MB BUT never
> restarted; so we're still using cache_mem 700 MB).
>
> If we raise our current setting of "cache_mem 700 MB" in any way
> (e.g. to 1000 MB), the squid will eventually crash -- very often
> after about 1h. Note that we have 2GB of RAM.
>
> Questions:
>
> * What are we doing wrong?
> * Are other people using Squid with 2GB of RAM? How?
>
> # cat /proc/meminfo
> total: used: free: shared: buffers: cached:
> Mem: 2119065600 2013810688 105254912 0 21864448 1035046912
> Swap: 4293509120 2646016 4290863104
> MemTotal: 2069400 kB
> MemFree: 102788 kB
> MemShared: 0 kB
> Buffers: 21352 kB
> Cached: 1008304 kB
> SwapCached: 2484 kB
> Active: 688220 kB
> Inactive: 1003264 kB
> HighTotal: 1703872 kB
> HighFree: 91268 kB
> LowTotal: 365528 kB
> LowFree: 11520 kB
> SwapTotal: 4192880 kB
> SwapFree: 4190296 kB
> BigFree: 0 kB
>
> (Squid is "up" for about 1d now)
>
> # cat /proc/cpuinfo
> processor : 0
> vendor_id : GenuineIntel
> cpu family : 15
> model : 2
> model name : Intel(R) Xeon(TM) CPU 2.80GHz
> stepping : 7
> cpu MHz : 2785.629
> cache size : 512 KB
> fdiv_bug : no
> hlt_bug : no
> f00f_bug : no
> coma_bug : no
> fpu : yes
> fpu_exception : yes
> cpuid level : 2
> wp : yes
> flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr
> pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm
> bogomips : 5557.45
>
> # cat /proc/swaps
> Filename Type Size Used
> Priority /dev/sda6 partition 2096440 0
> -1 /dev/sda7 partition 2096440 0
> -2
>
> Yes, I guess we should remove or at least reduce the swap, since a
> swapping Squid process will suck incredibly.
>
> $ cat /proc/mounts
> rootfs / rootfs rw 0 0
> /dev/root / ext3 rw 0 0
> proc /proc proc rw 0 0
> devpts /dev/pts devpts rw 0 0
> /dev/sdb5 /squid-cache0 xfs rw 0 0
> /dev/sdc5 /squid-cache1 xfs rw 0 0
> /dev/sdd5 /squid-data xfs rw 0 0
> /dev/sda5 /boot ext3 rw 0 0
>
> Filesystem Size Used Avail Use% Mounted on
> /dev/sda8 13G 3.5G 8.6G 29% /
> /dev/sdb5 17G 3.7G 14G 22% /squid-cache0
> /dev/sdc5 17G 3.7G 14G 22% /squid-cache1
> /dev/sdd5 17G 505M 17G 3% /squid-data
> /dev/sda5 69M 9.0M 56M 14% /boot
>
> $ cat /proc/scsi/scsi
> Attached devices:
> Host: scsi0 Channel: 00 Id: 00 Lun: 00
> Vendor: SEAGATE Model: ST318406LC Rev: 8A03
> Type: Direct-Access ANSI SCSI revision: 03
> Host: scsi0 Channel: 00 Id: 01 Lun: 00
> Vendor: SEAGATE Model: ST318406LC Rev: 8A03
> Type: Direct-Access ANSI SCSI revision: 03
> Host: scsi0 Channel: 00 Id: 02 Lun: 00
> Vendor: SEAGATE Model: ST318406LC Rev: 8A03
> Type: Direct-Access ANSI SCSI revision: 03
> Host: scsi0 Channel: 00 Id: 03 Lun: 00
> Vendor: SEAGATE Model: ST318406LC Rev: 8A03
> Type: Direct-Access ANSI SCSI revision: 03
> Host: scsi0 Channel: 00 Id: 06 Lun: 00
> Vendor: PE/PV Model: 1x5 SCSI BP Rev: 0.34
> Type: Processor ANSI SCSI revision: 02
>
> from the menuconfig output:
>
> (4GB) High Memory Support
>
> This was based on the help in the kernel config:
>
> "If the machine has between 1 and 4 Gigabytes physical RAM, then
> answer "4GB" here."
>
> (3.5GB) User address space size
> [*] HIGHMEM I/O support
>
> dmesg says:
>
> Linux version 2.4.20aa1 (root@spidergirl) (gcc version 2.95.4
> 20011002 (Debian prerelease)) #1 Thu Dec 19 14:35:35 CET 2002
> BIOS-provided physical RAM map:
> BIOS-e820: 0000000000000000 - 00000000000a0000 (usable)
> BIOS-e820: 0000000000100000 - 000000007fff0000 (usable)
> BIOS-e820: 000000007fff0000 - 000000007fffec00 (ACPI data)
> BIOS-e820: 000000007fffec00 - 000000007ffff000 (reserved)
> BIOS-e820: 00000000fec00000 - 00000000fec10000 (reserved)
> BIOS-e820: 00000000fee00000 - 00000000fee10000 (reserved)
> BIOS-e820: 00000000fff80000 - 0000000100000000 (reserved)
> 1663MB HIGHMEM available.
> 384MB LOWMEM available.
> ...
> Calibrating delay loop... 5557.45 BogoMIPS
> Memory: 2069264k/2097088k available (1659k kernel code, 27440k
> reserved, 651k data, 136k init, 1703872k highmem)
> Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes)
> Inode cache hash table entries: 131072 (order: 8, 1048576 bytes)
> Mount-cache hash table entries: 32768 (order: 6, 262144 bytes)
> Buffer-cache hash table entries: 131072 (order: 7, 524288 bytes)
> Page-cache hash table entries: 524288 (order: 9, 2097152 bytes)
> CPU: L1 I cache: 0K, L1 D cache: 8K
> CPU: L2 cache: 512K
> Intel machine check architecture supported.
> Intel machine check reporting enabled on CPU#0.
> CPU: After generic, caps: bfebfbff 00000000 00000000 00000000
> CPU: Common caps: bfebfbff 00000000 00000000 00000000
> CPU: Intel(R) Xeon(TM) CPU 2.80GHz stepping 07
> ...
>
>
> $ top -u proxy
>
> top - 10:55:57 up 1 day, 19:54, 3 users, load average: 0.00,
> 0.00, 0.00 Tasks: 88 total, 2 running, 86 sleeping, 0
> stopped, 0 zombie Cpu(s): 0.3% user, 15.6% system, 0.0%
> nice, 84.1% idle Mem: 2069400k total, 1970396k used, 99004k
> free, 21768k buffers Swap: 4192880k total, 2584k used,
> 4190296k free, 1011120k cached
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
> Command 18064 proxy 15 0 625m 625m 1344 R 0.3 30.9
> 25:33.18 squid 18065 proxy 15 0 625m 625m 1344 S 0.0 30.9
> 0:00.00 squid 18066 proxy 15 0 625m 625m 1344 S 0.0 30.9
> 0:03.09 squid 18067 proxy 15 0 625m 625m 1344 S 0.0 30.9
> 0:03.05 squid 18068 proxy 15 0 625m 625m 1344 S 0.0 30.9
> 0:03.11 squid 18069 proxy 15 0 625m 625m 1344 S 0.0 30.9
> 0:02.87 squid 18070 proxy 15 0 625m 625m 1344 S 0.0 30.9
> 0:02.87 squid 18071 proxy 15 0 625m 625m 1344 S 0.0 30.9
> 0:02.96 squid 18072 proxy 15 0 625m 625m 1344 S 0.0 30.9
> 0:02.95 squid 18073 proxy 15 0 625m 625m 1344 S 0.0 30.9
> 0:02.73 squid 18074 proxy 15 0 625m 625m 1344 S 0.0 30.9
> 0:03.04 squid 18075 proxy 15 0 625m 625m 1344 S 0.0 30.9
> 0:02.73 squid 18076 proxy 15 0 625m 625m 1344 S 0.0 30.9
> 0:03.17 squid 18077 proxy 15 0 625m 625m 1344 S 0.0 30.9
> 0:03.14 squid 18078 proxy 15 0 625m 625m 1344 S 0.0 30.9
> 0:03.17 squid 18079 proxy 15 0 625m 625m 1344 S 0.0 30.9
> 0:02.85 squid 18080 proxy 15 0 625m 625m 1344 S 0.0 30.9
> 0:03.24 squid 18081 proxy 15 0 625m 625m 1344 S 0.0 30.9
> 0:02.78 squid 18082 proxy 15 0 625m 625m 1344 S 0.0 30.9
> 0:02.98 squid 18083 proxy 15 0 625m 625m 1344 S 0.0 30.9
> 0:03.17 squid 18084 proxy 15 0 625m 625m 1344 S 0.0 30.9
> 0:02.89 squid 18085 proxy 15 0 625m 625m 1344 S 0.0 30.9
> 0:02.93 squid 18086 proxy 15 0 625m 625m 1344 S 0.0 30.9
> 0:02.89 squid 18087 proxy 15 0 625m 625m 1344 S 0.0 30.9
> 0:02.67 squid 18088 proxy 15 0 625m 625m 1344 S 0.0 30.9
> 0:02.92 squid 18089 proxy 15 0 625m 625m 1344 S 0.0 30.9
> 0:03.07 squid 18090 proxy 15 0 625m 625m 1344 S 0.0 30.9
> 0:02.78 squid 18091 proxy 15 0 625m 625m 1344 S 0.0 30.9
> 0:02.85 squid 18092 proxy 15 0 625m 625m 1344 S 0.0 30.9
> 0:03.12 squid 18093 proxy 15 0 625m 625m 1344 S 0.0 30.9
> 0:03.04 squid 18094 proxy 15 0 625m 625m 1344 S 0.0 30.9
> 0:03.03 squid 18095 proxy 15 0 625m 625m 1344 S 0.0 30.9
> 0:02.81 squid 18096 proxy 15 0 625m 625m 1344 S 0.0 30.9
> 0:03.13 squid 18097 proxy 15 0 625m 625m 1344 S 0.0 30.9
> 0:02.74 squid
Received on Sat Dec 21 2002 - 05:04:58 MST