[squid-users] Finally, the first real coredump, was Re: [squid-users] Squid Error

What I got today:

# gdb /usr/local/squid/sbin/squid /squid-cache0/core-11\:07\:00
GNU gdb 5.3
Copyright 2002 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "sparc-sun-solaris2.8"...
Core was generated by /usr/local/squid/sbin/squid -NsY'.
Program terminated with signal 6, Aborted.
Reading symbols from /usr/lib/libcrypt_i.so.1...done.
Loaded symbols for /usr/lib/libcrypt_i.so.1
Reading symbols from /usr/lib/libpthread.so.1...done.
Loaded symbols for /usr/lib/libpthread.so.1
Reading symbols from /usr/lib/libm.so.1...done.
Loaded symbols for /usr/lib/libm.so.1
Reading symbols from /usr/lib/libresolv.so.2...done.
Loaded symbols for /usr/lib/libresolv.so.2
Reading symbols from /usr/lib/libsocket.so.1...done.
Loaded symbols for /usr/lib/libsocket.so.1
Reading symbols from /usr/lib/libnsl.so.1...done.
Loaded symbols for /usr/lib/libnsl.so.1
Reading symbols from /usr/lib/libc.so.1...done.
Loaded symbols for /usr/lib/libc.so.1
Reading symbols from /usr/lib/libgen.so.1...done.
Loaded symbols for /usr/lib/libgen.so.1
Reading symbols from /usr/lib/libdl.so.1...done.
Loaded symbols for /usr/lib/libdl.so.1
Reading symbols from /usr/lib/libmp.so.2...done.
Loaded symbols for /usr/lib/libmp.so.2
Reading symbols from /usr/platform/SUNW,Ultra-60/lib/libc_psr.so.1...done.
Loaded symbols for /usr/platform/SUNW,Ultra-60/lib/libc_psr.so.1
Reading symbols from /usr/lib/libthread.so.1...done.
Loaded symbols for /usr/lib/libthread.so.1
Reading symbols from /usr/lib/nss_files.so.1...done.
Loaded symbols for /usr/lib/nss_files.so.1
Reading symbols from /usr/lib/nss_dns.so.1...done.
Loaded symbols for /usr/lib/nss_dns.so.1
#0 0xff0b9bf0 in __sigprocmask () from /usr/lib/libthread.so.1

(gdb) bt
#0 0xff0b9bf0 in __sigprocmask () from /usr/lib/libthread.so.1
#1 0xff0ae628 in _resetsig () from /usr/lib/libthread.so.1
#2 0xff0add18 in _sigon () from /usr/lib/libthread.so.1
#3 0xff0b0e8c in _thrp_kill () from /usr/lib/libthread.so.1
#4 0xff149b10 in raise () from /usr/lib/libc.so.1
#5 0xff13512c in abort () from /usr/lib/libc.so.1
#6 0x000af224 in death (sig=-14983168) at tools.c:301

more corefiledebugging on request.

And in the cache log for that period:

...
2003/04/23 11:04:01| CACHEMGR: <unknown>@192.168.220.203 requesting 'info'
2003/04/23 11:05:01| CACHEMGR: <unknown>@192.168.220.203 requesting 'info'
2003/04/23 11:06:02| CACHEMGR: <unknown>@192.168.220.203 requesting 'info'
FATAL: Received Bus Error...dying.
2003/04/23 11:06:38| storeDirWriteCleanLogs: Starting...
2003/04/23 11:06:38| WARNING: Closing open FD 13
2003/04/23 11:06:38| WARNING: Closing open FD 14
2003/04/23 11:06:39| 65536 entries written so far.
2003/04/23 11:06:39| 131072 entries written so far.
2003/04/23 11:06:40| 196608 entries written so far.
2003/04/23 11:06:40| 262144 entries written so far.
2003/04/23 11:06:41| 327680 entries written so far.
2003/04/23 11:06:41| 393216 entries written so far.
2003/04/23 11:06:42| 458752 entries written so far.
2003/04/23 11:06:42| 524288 entries written so far.
2003/04/23 11:06:43| 589824 entries written so far.
2003/04/23 11:06:43| 655360 entries written so far.
2003/04/23 11:06:43| Finished. Wrote 660341 entries.
2003/04/23 11:06:43| Took 5.1 seconds (130146.9 entries/sec).
CPU Usage: 907.380 seconds = 626.250 user + 281.130 sys
Maximum Resident Size: 0 KB
Page faults with physical i/o: 12326
Memory usage for squid via mallinfo():
        total space in arena: 136496 KB
        Ordinary blocks: 136463 KB 9 blks
        Small blocks: 0 KB 0 blks
        Holding blocks: 7576 KB 5 blks
        Free Small blocks: 0 KB
        Free Ordinary blocks: 32 KB
        Total in use: 144039 KB 106%
        Total free: 32 KB 0%
2003/04/23 11:07:02| Starting Squid Cache version 2.5.STABLE2-20030422 for ...
2003/04/23 11:07:02| Process ID 28024
2003/04/23 11:07:02| With 4096 file descriptors available
2003/04/23 11:07:02| Performing DNS Tests...
2003/04/23 11:07:02| Successful DNS name lookup tests...
2003/04/23 11:07:02| DNS Socket created at 0.0.0.0, port 22916, FD 5
2003/04/23 11:07:02| Adding nameserver 193.175.66.33 from squid.conf
2003/04/23 11:07:02| Adding nameserver 193.175.70.246 from squid.conf
2003/04/23 11:07:02| User-Agent logging is disabled.
2003/04/23 11:07:02| Referer logging is disabled.

access log for that period:

Wed Apr 23 11:06:38 2003 0 192.168.220.203 UDP_MISS/000 74 ICP_QUERY http://forums.delphiforums.com/ICCdiscussion/messages - NONE/- -
Wed Apr 23 11:06:38 2003 0 192.168.220.204 UDP_MISS/000 68 ICP_QUERY http://in.yimg.com/i/in/adv/bharat/bm_25x25.gif - NONE/- -
Wed Apr 23 11:06:38 2003 0 192.168.220.203 UDP_MISS/000 104 ICP_QUERY http://ebay0.ipixmedia.com/abc/M28/_EBAY_e0c7da2bb0df75b2fab79e00971f5cee/i-5_B.JPG - NONE/- -
Wed Apr 23 11:06:38 2003 4 192.168.220.203 TCP_HIT/200 5324 GET http://images.gmx.net/images/bs/15113/GMX_Weltquiz_80x80_rot.gif - NONE/- image/gif
Wed Apr 23 11:06:38 2003 10 192.168.220.203 TCP_HIT/200 15929 GET http://images.gmx.net/images/bs/15095/ban_030211aveF1_296x99_15K.gif - NONE/- image/gif
Wed Apr 23 11:06:38 2003 121 141.42.121.108 TCP_MISS/200 1649 GET http://www.turist.vordingborg.dk/tysk/turistcenteret/turistcenter.htm - DIRECT/131.165.202.24 text/html
Wed Apr 23 11:06:38 2003 0 192.168.220.203 UDP_MISS/000 107 ICP_QUERY http://193.110.193.70/carconfig/images/fzg/20032/fabia/aussenfarben/admiral_blau_1.jpg - NONE/- -
Wed Apr 23 11:06:38 2003 84 141.42.121.108 TCP_MISS/200 1220 GET http://www.turist.vordingborg.dk/images/nyede/turistc.gif - DIRECT/131.165.202.24 image/gif
... the crash occurs ...
Wed Apr 23 11:07:03 2003 241 141.42.44.166 TCP_MISS/200 643 GET http://ad.doubleclick.net/adl/ebay.de.photo2/Digitalkameras;cat=;cat=;cat=;page=listing;sz=468x60;ord=1051088812959? - DIRECT/216.73.87.102 text/html

-- 
Ralf Hildebrandt (Im Auftrag des Referat V a)   Ralf.Hildebrandt@charite.de
Charite Campus Mitte                            Tel.  +49 (0)30-450 570-155
Referat V a - Kommunikationsnetze -             Fax.  +49 (0)30-450 570-916
AIM: ralfpostfix