RE: getting Squid logfiles into a database

Couldn't you set up a named pipe that Squid logs to and then have a
perl(ish) script that sits like a daemon and reads from that pipe? Then,
you could both record the entry to the DB (if desired) and to a log file.

When we've looked at real-time logging to a database, the thing that kept us
from doing it was the additional network overhead. Now, instead of having a
request from the client, the request to the server, the response from the
server, and the response to the client, we're also going to add the network
traffic to send the log entry to a database server. You could get around
this by putting the DB engine on the same box as the proxy, but then you've
got the additional disk, CPU, and network overhead associated with the
real-time ad-hoc queries.

Jon

> -----Original Message-----
> From: Chris Foote [SMTP:chris@senet.com.au]
> Sent: Tuesday, January 12, 1999 3:28 AM
> To: Gavin Cameron
> Cc: squid-users@ircache.net
> Subject: Re: getting Squid logfiles into a database
>
> On Tue, 12 Jan 1999, Gavin Cameron wrote:
>
> > It would be helpful, I think it would be, to be able to get your access
> > logs put into a database, and I think it could be done in much the same
> > way as authentication
> >
> > The directives could be something like
> > cache_access_log_program: name of the logging program
> > cache_access_log_program_children: how many logging programs to have
> > running
> > and it would accept native format messages (one per line).
> >
> > Does anyone have something like this running?
>
> Yep, we've got a Perl program which tails access.log files to do this -
> it ignores ICP queries and HTTP requests from neighbours (it has a simple
> neighbours config file), and spits the entries across to a central UDP
> logging box which works out where to classify them.
>
> It would be nice to have squid log to a pipe instead of tailing the
> log file, but it only gets rotated once a day, and very few records
> go missing.
>
> If it's of any use, let me know and I'll get permission from the script
> author to release it.
>
>
> Cheers,
>
> Chris Foote SE Net
> Technical Manager 222 Grote Street
> SE Network Access Adelaide SA 5000
> e-mail chris@senet.com.au Australia
> phone : (08) 8221 5221 PGP Public Key available from
> fax: (08) 8221 5220 http://www.senet.com.au/PGP
> support: (08) 8221 5792
Received on Tue Jan 12 1999 - 08:15:25 MST