Re: debugging high volume caches

What I do when core dumps is not possible is to run Squid under the
control of a debugger. That way you can get automated stack traces from
each restart, and it wont suffer that much in performance as you do from
full debug logging.

A full debug log is generally only useful when you can repeat a error in
a controlled environment. Stack traces is very much preffered to debug
logs on sporadic segmentation faults or assertion failures. If the stack
trace isn't engouth then we will get back to you with what debug output
or additional traces we'd like to se.

Attached is a modified RunCache script which uses gdb to get automated
stack traces on all failures (note: If you use async-io, then you need a
version of gdb supporting threads. The Redhat 6 version "GNU gdb
4.17.0.11 with Linux support" works fine for me)

--
Henrik Nordstrom
Spare time Squid hacker
tom minchin wrote:
> 
> Hi,
>         I've got two "crash and restart" type conditions in Squid 2.2S4 on
> Linux that I want to debug so I can send off the logs to squid-bugs. However,
> the volume on the caches is such that:
>         1) I generate about 4 gig of logs an hour
>         2) debug logging slows Squid down to a near halt

#!/bin/sh
# Modified RunCache script to get automated stacktraces on failures
#

prefix=/usr/local/squid
exec_prefix=${prefix}
logdir=${prefix}/logs
PATH=${exec_prefix}/bin:/bin:/usr/bin
export PATH

conf=""
if test "$1" ; then
        conf="-f $1"
        shift
fi

cat <<EOF >${prefix}/etc/stacktrace.gdb
handle all pass nostop noprint
handle SIGSEGV stop print
handle SIGBUS stop print
handle SIGABRT stop print
run -NDYd3 $conf
bt
c
bt
c
bt
c
bt
c
bt
c
bt
quit
EOF

failcount=0
while : ; do
        echo "Running: squid -NDYd3 $conf >> $logdir/squid.out 2>&1"
        echo "Startup: `date`" >> $logdir/squid.out
        start=`date '+%d%H%M%S'`
        gdb --batch -x ${prefix}/etc/stacktrace.gdb ${prefix}/bin/squid >> $logdir/squid.out 2>&1
        stop=`date '+%d%H%M%S'`
        t=`expr $stop - $start`
        if test 0 -le $t -a $t -lt 5 ; then
                failcount=`expr $failcount + 1`
        else
                failcount=0
        fi
        if test $failcount -gt 5 ; then
              echo "DebugCache: EXITING DUE TO REPEATED, FREQUENT FAILURES" >&2
                exit 1
        fi
        sleep 10
done
Received on Sat Aug 07 1999 - 14:30:48 MDT