Re: [squid-users] Syslog patch for 2.5STABLE9

On Tue, 01 Mar 2005 14:18:28 +0800, Martin Marji Cermak <mc1@trimedia.cz> wrote:
> Kevin wrote:
> > On Fri, 25 Feb 2005 12:54:30 -0600, Kevin <kkadow@gmail.com> wrote:
> >
> >>Has anybody put together a good patch for Squid (2.5.X) to record access
> >>information via syslog instead of writing to disk? It looks like I could simply
> >>modify logfilePrintf() in logfile.c?
> >
> >
> > While it's bad form to reply to one's own post, yes, it really is that simple,
> > I wrapped the logfilePrintf calls in access_log.c with if statements.
> >
> >>(P.S. Yes, I fully understand the various issues with and drawbacks of
> >>using "syslog" for access logs, particularly across a network.)
> >
> >
> > That said, here is a functional (beta) patch for sending access_log to syslog,
> > use at your own risk. To enable syslog logging, change cache_access_log
> > in squid.conf to read "cache_access_log syslog".
>
> Hello Kevin,
> yes, I also wanted to log access-log records via syslog to another
> machine (a log server in the same network) - I modified the code
> basically the same way you did.
>
> But I noticed that the access-log on the log server was not complete!
> During peaktime (when Squid served more than 150 requests/sec), there
> were lines of access-log lost. To be sure this was the case, I inserted
> a counter in every access-log line and really, there were gaps in the
> numbers in the access-log on the log server.
>
> I guess the reason was that syslog logging over the network uses udp and
> does not bother when it is overloaded.

I believe this behavior of syslogd is system dependent. The ultimate solution
could be to modify squid to either log to a tcp connection (has issues when the
log sink is not able to accept connections or is lagged) or to just spew the UDP
packets directly to the wire instead of making calls to syslog().

Additionally, there are issues with UDP syslog and network congestion.
I work around some of these problems by using a dedicated log interface to
a dedicated log switch and log host. I still lose around 0.0001% of
log messages,
but this is acceptable in my environment, most of the time I just use
the logs for
diagnostics and cache hit statistics.

> So I had to change the logging logic to the following one:
> log only error (HTTP status code >= 400) access-log records via syslog
> log all access-log locally, but rotate the access-log regularly(so I
> have the complete records for at least last 10 hours)
>
> Syslog on the log server holds all error access-logs for several days
> and if I am lucky and the error I am interested in occurred within last
> 10 hours, I can check the complete log on the Squid machine.

Makes sense.

I switched to network logging to take the load off of the disk subsystem,
so this option does not work well for me. OTOH, the servers where I do
syslog instead of local access_log only accept requests from a limited
number of "child caches", and those caches are logging to a local drive,
so I get some of the same benefits as your approach.

Kevin Kadow
Received on Tue Mar 01 2005 - 12:57:34 MST