Re: Squid Cache Statistics

You wrote:
> Is it possible to generate squid cache statistics using pwebstats?
> If yes, how? If not, what is used to generate web-based squid
> cache statistics?
>
> Any help will be very much appreciated.

Yes, it is possible. Below is a repost of a script that translates squid logs
into cern-style cache hit & miss logs for feeding into pwebstats. The next
version of pwebstats will handle squid logs internally.

--------------------------
/servers/http/squid/bin/squid2common.pl:

#!/usr/local/bin/perl
#
# Martin Gleeson, Daniel O'Callaghan, June 1996
#
# (c) Copyright, The University of Melbourne, 1996
#
#
$logfile = shift ( @ARGV );
$address_type=2;
$date_now=`date +"%I:%M %p, %A %B %e %Y"`;chop($date_now);
printf STDERR "convert-harvest-log.pl started on $logfile at $date_now.\n";
printf STDERR "==========================================================\n";

open(COUNT,"/bin/wc -l $logfile |");
while( <COUNT> ){ chop; ($line_count) = /^\s+(\d+)\s+\S+$/; }
close(COUNT);
printf STDERR "The logfile has $line_count entries.\n";
printf STDERR "Processing... [ \# = 500 log entries ]\n";

$counter=$linecount=0;

open(LOG_FILE,"$logfile");
open(PROXY_FILE,">> proxy.convert");
open(CACHE_FILE,">> cache.convert");

while(<LOG_FILE>){
        $linecount++;
        $counter++;
        if( $counter >= 500 )
        {
                $counter = 0;
                printf STDERR "\#";
        }
# split the input line into its various components

        $line=$_;
        ( $host, $rfc931, $user, $www_date, $request, $type, $size)
        = /^(\S+) (\S+) (\S+) \[(.+)\] \"(.*)\" (\S+) (\S+)\s/;

        # convert IP into hostname
        if( $hosts{$host} ) { $name = $hosts{$host};}
        else
        {
                if($host =~ /\d\d\.\d\d/)
                {
                        @address = split(/\./,$host);
                        $addpacked = pack('C4',@address);
                        ($name,$aliases,$addrtype,$length,@addrs) =
gethostbyaddr($addpacked,$address_type);
                }
                if ( $name eq "") { $name = $host };
                $name = "\L$name";
                $hosts{$host} = $name;
        }
        if( $type eq "TCP_DENIED" ){ # fetched from external source
                $line_new = "$hosts{$host} $rfc931 $user [$www_date]
\"$request HTTP/1.0\" 401 $size\n";
                print PROXY_FILE "$line_new";
                $sizeproxied += $size;
                $totalproxied += 1;
        }
        if( $type eq "TCP_MISS" || ($type eq "TCP_IFMODSINCE" && $size < 220)
                ||$type eq "TCP_EXPIRED" || $type eq "TCP_REFRESH"
                || $type eq "TCP_SWAPFAIL"){ # fetched from external source
                $line_new = "$hosts{$host} $rfc931 $user [$www_date]
\"$request HTTP/1.0\" 200 $size\n";
                print PROXY_FILE "$line_new";
                $sizeproxied += $size;
                $totalproxied += 1;
        }
        elsif( $type eq "TCP_HIT" || ($type eq "TCP_IFMODSINCE" && $size
>= 220) ){
            if( $size != 0 ) {
                        $line_new = "$hosts{$host} $rfc931 $user
[$www_date] \"$request HTTP/1.0\" 200 $size\n";
                print CACHE_FILE "$line_new";
                $sizecached += $size;
            $totalcached += 1;
           }
        }
}
close(LOG_FILE);
close(PROXY_FILE);
close(CACHE_FILE);

printf STDERR "\n%s lines processed, finding %s proxy requests and %s cache
requests.\n\n",

&commas($linecount),&commas($totalproxied),&commas($totalcached);
printf STDERR "Total sizes: Proxy %s bytes, Cache %s bytes\n\n",
                                  &commas($sizeproxied),&commas($sizecached);
if( ($totalproxied != 0 || $totalcached != 0) && ($sizeproxied!=0 ||
$sizecached!=0)){
printf STDERR "Hit rates: %s%% requests, %s%% bytes\n\n",
               &commas( ($totalcached/($totalproxied+$totalcached)) * 100),
               &commas( ($sizecached/($sizeproxied+$sizecached)) * 100);
}
$date_now=`date +"%I:%M %p, %A %B %e %Y"`;
chop($date_now);
printf STDERR "convert-harvest-log.pl finished on $logfile at $date_now.\n";
printf STDERR "==========================================================\n";

exit(0);

sub commas {
        local($_)=@_;
        $_ = sprintf "%ld", $_;
        1 while s/(.*\w)(\w\w\w)/$1,$2/;
        $_;
}
--------------------------

Cheers,
Marty.
-------------------------------------------------------------------------
Martin Gleeson Webmeister | http://www.unimelb.edu.au/%7Egleeson/
Information Technology Services | Email : gleeson@unimelb.edu.au
The University of Melbourne, Oz. | Opinions : Mine, all mine.
       "I hate quotations" -- Ralph Waldo Emerson; Journals (1843)
-------------------------------------------------------------------------
Received on Sun Jul 07 1996 - 20:11:09 MDT