Hello All,
I am running out of ideas and still want to resolve another
problem that has been occurring for some time.
Background info: There are 18 remote proxies running on
RedHat Linux with 2.2.14-5 kernel. Each of the 18 proxies
go to one parent proxy on HP-UX 11.0. All squid versions
are 2.4.STABLE4.
I modified a shutdown script (see enclosed) so that five
minutes is given for squid to shutdown. The shutdown
script runs a netstat and lsof before shutting down squid
and writes the processes to a log (see enclosed). The log
from the shutdown script tells me that squid has shutdown.
Squid is shutdown at approx. 9:00pm and restarts at 6:00am.
When squid was restarted the cache.lg says port 3128 was in
use can not bind. If squid shuts down successfully, can
the port still be in use? Is squid not really shutdown if
port 3128 is still in use?
How can I ensure that all connections are closed so that
proxy will start without the message "port in use"? Is
there anyway that squid can be configured to help out in
this problem?
Next, I looked at cache.lg (see enclosed) and saw that
"WARNING: Closing client xx.xx.xx.xx connection due to
lifetime timeout". Is this a clue to why port 3128 was
still in use? Does squid.conf have any parameters that can
be adjusted to help this particular problem? Suchas,
liftetime timeout?
Thank you for your time and any ideas/suggestions you may
have.
-pat
On 31 Dec 2001, at 14:38, Joe Cooper wrote:
> plendon@microelectronics.com wrote:
>
> > Hi All,
> >
> > I've had incidents with squid and have been
> > troubleshooting the problem for some time, maybe you can
> > help.
> >
> > Background info: There are 18 remote proxies running on
> > RedHat Linux with 2.2.14-5 kernel. Each of the 18 proxies
> > go to one parent proxy on HP-UX 11.0. All squid versions
> > are 2.4.STABLE4. All squids were compiled with --enable-
> > kill-parent-hack.
> >
> > Problem Description: Each night a main process is started
> > that remsh's to the 18 linux boxes and shuts down the
> > proxies. The proxies are shutdown so as not to interfere
> > with network traffic and the nightly processes. The
> > proxies are shutdown with /etc/rc.d/init.d/squid stop
> > (enclosed is startup/shutdown script). The shutdowns are
> > always successful. When squid is shutdown it will return
> > a "Y" for successful or a "N" for non-successful
> > shutdown. For the majority of times, a "Y" is returned.
> > A "N" has never returned. However, sometimes nothing is
> > returned (no "Y" and no "N") and then I am notified that
> > there might be a problem with a proxy shutting down. I
> > then log into the linux system and see that squid has
> > successfully shutdown.
> >
> > What is happening? Squid has shutdown, but has not
> > returned a "Y". Enclosed is the shutdown script.
> > Somewhere in the script is a 30 second wait to ensure
> > squid has shutdown.
> >
> > Could the squid shutdown be affected by a client
> > downloading or ftp'ing a large file when squid is being
> > shutdown?
> >
> > Do I need to adjust squid's configuration parameters? I'm
> > not sure what parameters I need to adjust to help
> > troubleshoot problem further.
>
> Sounds like the script is returning before it has a
> success/fail value (I hit this when I was writing dual Squid
> process handlers). Increase the 30 second timeout to 90
> seconds, or so--note that this will /not/ increase the time
> that the script takes to shutdown Squid, it merely tells the
> script how long to wait for a success/fail status from
> Squid.
> If this timeout runs out, you won't get a result and the
> script simply
> exits.
>
> --
> Joe Cooper <joe@swelltech.com>
> http://www.swelltech.com
> Web Caching Appliances and Support
>
2002/01/06 06:15:03| storeDirWriteCleanLogs: Starting...
2002/01/06 06:15:04| Finished. Wrote 12258 entries.
2002/01/06 06:15:04| Took 0.0 seconds (678211.8 entries/sec).
2002/01/06 06:15:04| logfileRotate: /usr/local/squid/logs/store.log
2002/01/06 06:15:04| logfileRotate: /usr/local/squid/logs/access.log
2002/01/06 06:15:04| helperOpenServers: Starting 5 'ncsa_auth' processes
2002/01/06 12:56:39| aclDecodeProxyAuth: Disallowing empty password,user is ''
2002/01/06 12:57:32| aclDecodeProxyAuth: Disallowing empty password,user is ''
2002/01/06 12:57:32| aclDecodeProxyAuth: Disallowing empty password,user is ''
2002/01/06 13:38:04| urlParse: Illegal character in hostname 'ad.linksyn%20ergy.com'
2002/01/06 13:54:21| urlParse: Illegal character in hostname 'ad.linksy%20nergy.com'
2002/01/06 14:13:30| aclDecodeProxyAuth: Disallowing empty password,user is ''
2002/01/06 14:52:52| aclDecodeProxyAuth: Disallowing empty password,user is 'comm131'
2002/01/06 15:04:59| aclDecodeProxyAuth: Disallowing empty password,user is 'comm131'
2002/01/06 15:25:37| aclDecodeProxyAuth: Disallowing empty password,user is ''
2002/01/06 15:25:37| aclDecodeProxyAuth: Disallowing empty password,user is ''
2002/01/06 15:26:49| aclDecodeProxyAuth: Disallowing empty password,user is ''
2002/01/06 15:26:49| aclDecodeProxyAuth: Disallowing empty password,user is ''
2002/01/06 16:23:16| aclDecodeProxyAuth: Disallowing empty password,user is ''
2002/01/06 16:23:16| aclDecodeProxyAuth: Disallowing empty password,user is ''
2002/01/06 16:23:16| aclDecodeProxyAuth: Disallowing empty password,user is ''
2002/01/06 16:23:28| aclDecodeProxyAuth: Disallowing empty password,user is ''
2002/01/06 16:23:28| aclDecodeProxyAuth: Disallowing empty password,user is ''
2002/01/06 16:23:28| aclDecodeProxyAuth: Disallowing empty password,user is ''
2002/01/06 16:55:22| aclDecodeProxyAuth: Disallowing empty password,user is ''
2002/01/06 17:00:13| aclDecodeProxyAuth: Disallowing empty password,user is ''
2002/01/06 17:00:13| aclDecodeProxyAuth: Disallowing empty password,user is ''
2002/01/06 17:00:13| aclDecodeProxyAuth: Disallowing empty password,user is ''
2002/01/06 17:09:57| aclDecodeProxyAuth: Disallowing empty password,user is 'sarahsmile'
2002/01/06 17:10:11| aclDecodeProxyAuth: Disallowing empty password,user is 'sarahsmile'
2002/01/06 17:50:58| Killing RunCache, pid 2054
2002/01/06 17:51:00| Preparing for shutdown after 13839 requests
2002/01/06 17:51:00| Waiting 30 seconds for active connections to finish
2002/01/06 17:51:00| FD 16 Closing HTTP connection
2002/01/06 17:51:00| FD 17 Closing HTTP connection
2002/01/06 17:51:31| Shutting down...
2002/01/06 17:51:31| FD 18 Closing ICP connection
2002/01/06 17:51:31| FD 19 Closing SNMP socket
2002/01/06 17:51:31| WARNING: Closing client 10.101.11.50 connection due to lifetime timeout
2002/01/06 17:51:31| http://www.powershot.com/powershot2/G2/pdf/G2_bro.pdf
2002/01/06 17:51:31| WARNING: Closing client 10.101.11.50 connection due to lifetime timeout
2002/01/06 17:51:31| http://www.powershot.com/powershot2/G2/pdf/G2_bro.pdf
2002/01/06 17:51:31| Closing unlinkd pipe on FD 14
2002/01/06 17:51:31| storeDirWriteCleanLogs: Starting...
2002/01/06 17:51:31| Finished. Wrote 11026 entries.
2002/01/06 17:51:31| Took 0.0 seconds (655139.6 entries/sec).
CPU Usage: 36.610 seconds = 20.960 user + 15.650 sys
Maximum Resident Size: 0 KB
Page faults with physical i/o: 2926
Memory usage for squid via mallinfo():
total space in arena: 14011 KB
Ordinary blocks: 13951 KB 364 blks
Small blocks: 0 KB 0 blks
Holding blocks: 176 KB 1 blks
Free Small blocks: 0 KB
Free Ordinary blocks: 60 KB
Total in use: 14127 KB 101%
Total free: 60 KB 0%
2002/01/06 17:51:31| Squid Cache (Version 2.4.STABLE2): Exiting normally.
2002/01/07 06:00:00| Starting Squid Cache version 2.4.STABLE2 for i686-pc-linux-gnu...
2002/01/07 06:00:00| Process ID 4912
2002/01/07 06:00:00| With 1024 file descriptors available
2002/01/07 06:00:00| Performing DNS Tests...
2002/01/07 06:00:00| Successful DNS name lookup tests...
2002/01/07 06:00:00| DNS Socket created on FD 4
2002/01/07 06:00:00| Adding nameserver 10.10.1.100 from /etc/resolv.conf
2002/01/07 06:00:00| helperOpenServers: Starting 5 'ncsa_auth' processes
2002/01/07 06:00:00| User-Agent logging is disabled.
2002/01/07 06:00:00| Unlinkd pipe opened on FD 14
2002/01/07 06:00:00| Swap maxSize 102400 KB, estimated 7876 objects
2002/01/07 06:00:00| Target number of buckets: 393
2002/01/07 06:00:00| Using 8192 Store buckets
2002/01/07 06:00:00| Max Mem size: 8192 KB
2002/01/07 06:00:00| Max Swap size: 102400 KB
2002/01/07 06:00:00| Rebuilding storage in /usr/local/squid/cache (CLEAN)
2002/01/07 06:00:00| Using Least Load store dir selection
2002/01/07 06:00:00| Set Current Directory to /usr/local/squid/cache
2002/01/07 06:00:00| Loaded Icons.
2002/01/07 06:00:01| commBind: Cannot bind socket FD 16 to *:3128: (98) Address already in use
2002/01/07 06:00:01| Accepting HTTP connections at 0.0.0.0, port 8080, FD 16.
2002/01/07 06:00:01| Accepting ICP messages at 0.0.0.0, port 3130, FD 17.
2002/01/07 06:00:01| Accepting SNMP messages on port 3401, FD 18.
2002/01/07 06:00:01| WCCP Disabled.
2002/01/07 06:00:01| Configuring Parent 10.10.1.100/3128/3130
2002/01/07 06:00:01| Ready to serve requests.
2002/01/07 06:00:01| Store rebuilding is 37.1% complete
2002/01/07 06:00:01| Done reading /usr/local/squid/cache swaplog (11026 entries)
2002/01/07 06:00:01| Finished rebuilding storage from disk.
2002/01/07 06:00:01| 11026 Entries scanned
2002/01/07 06:00:01| 0 Invalid entries.
2002/01/07 06:00:01| 0 With invalid flags.
2002/01/07 06:00:01| 11026 Objects loaded.
2002/01/07 06:00:01| 0 Objects expired.
2002/01/07 06:00:01| 0 Objects cancelled.
2002/01/07 06:00:01| 0 Duplicate URLs purged.
2002/01/07 06:00:01| 0 Swapfile clashes avoided.
2002/01/07 06:00:01| Took 0.8 seconds (13855.5 objects/sec).
2002/01/07 06:00:01| Beginning Validation Procedure
2002/01/07 06:00:01| Completed Validation Procedure
2002/01/07 06:00:01| Validated 11026 Entries
2002/01/07 06:00:01| store_swap_size = 92148k
2002/01/07 06:00:01| storeLateRelease: released 0 objects
#!/bin/sh
#
# Starts and stops squid proxy/cache server .
#
# See how we were called.
PIDLOG=/usr/local/squid/logs/pid.log
PIDFILE=/usr/local/squid/logs/squid.pid
SQUID_TIMEOUT=300
case "$1" in
start_msg)
echo "Starting squid server"
;;
stop_msg)
echo "Stopping squid server"
;;
start)
echo "Starting squid server"
/usr/local/squid/bin/squid
;;
stop)
# Log some info
echo "`date`" > $PIDLOG
echo "Squid processes are:" >> $PIDLOG
ps -ef|grep squid|awk '{print $0}' >> $PIDLOG
echo "netstat listing:" >> $PIDLOG
netstat -a |grep 3128 >> $PIDLOG
echo "lsof listing:" >> $PIDLOG
/usr/sbin/lsof -i :3128|awk '{print $0}' >> $PIDLOG
echo "Checking to see if squid is running..." >> $PIDLOG
/usr/local/squid/bin/squid -k check >/dev/null 2>&1
RETVAL=$?
echo $RETVAL >> $PIDLOG
if [ $RETVAL -eq 0 ] ; then
echo "Stopping squid server" >> $PIDLOG
/usr/local/squid/bin/squid -k shutdown
RETVAL=$?
echo $RETVAL >> $PIDLOG
if [ $RETVAL -ne 0 ] ; then
echo "Unable to shutdown squid" >> $PIDLOG
exit 1
else
echo "Now sleeping for five minutes to give squid some time to shutdown..." >> $PIDLOG
sleep $SQUID_TIMEOUT
fi
else
echo "Squid already not running, or other problem?" >> $PIDLOG
exit 0
fi
echo "Checking to see if squid is still running." >> $PIDLOG
/usr/local/squid/bin/squid -k check >/dev/null 2>&1
RETVAL=$?
echo $RETVAL >> $PIDLOG
if [ $RETVAL -eq 0 ] ; then
echo "Squid is still running..." >> $PIDLOG
SQUID_PIDS=`ps -ef|grep squid|awk '{if ($8 != "grep" && $9 != "squid" && $10 != "stop") print $2}'`
echo "What are the remaining processes?" >> $PIDLOG
echo $SQUID_PIDS >> $PIDLOG
echo "Going to kill remaining processes" >> $PIDLOG
2>&- kill -15 $SQUID_PIDS
sleep 100
else
echo "Squid is not running" >> $PIDLOG
exit 0
fi
echo "Checking again to see if squid is still running." >> $PIDLOG
/usr/local/squid/bin/squid -k check >/dev/null 2>&1
RETVAL=$?
echo $RETVAL >> $PIDLOG
if [ $RETVAL -eq 0 ] ; then
echo "Squid is still running ... " >> $PIDLOG
exit 1
else
echo "Squid is not running" >> $PIDLOG
exit 0
fi
;;
restart)
echo "Re-read squid conf file"
/usr/local/squid/bin/squid -k reconfigure
;;
*)
echo "Usage: /sbin/init.d/squid {start|stop|restart}"
exit 1
esac
exit
Sun Jan 6 17:50:59 CST 2002
Squid processes are:
root 2054 1 0 06:00 ? 00:00:00 /usr/local/squid/bin/squid
squid 2056 2054 0 06:00 ? 00:00:36 (squid)
squid 2062 2056 0 06:00 ? 00:00:00 (unlinkd)
squid 2120 2056 0 06:15 ? 00:00:00 (ncsa_auth) /usr/local/squid/etc
squid 2121 2056 0 06:15 ? 00:00:00 (ncsa_auth) /usr/local/squid/etc
squid 2122 2056 0 06:15 ? 00:00:00 (ncsa_auth) /usr/local/squid/etc
squid 2124 2056 0 06:15 ? 00:00:00 (ncsa_auth) /usr/local/squid/etc
squid 2125 2056 0 06:15 ? 00:00:00 (ncsa_auth) /usr/local/squid/etc
squid 3307 3306 0 17:50 ? 00:00:00 sh /usr/local/squid/bin/down -c
squid 3308 3307 0 17:50 ? 00:00:00 sh /usr/local/squid/bin/down -c
squid 3309 3308 0 17:50 ? 00:00:00 sh /etc/rc.d/init.d/squid stop
squid 3312 3309 0 17:50 ? 00:00:00 ps -ef
squid 3313 3309 0 17:50 ? 00:00:00 grep squid
squid 3314 3309 0 17:50 ? 00:00:00 awk {print $0}
netstat listing:
tcp 0 38603 dllinux2:3128 dlpeapc.microcente:2861 ESTABLISHED
tcp 0 38567 dllinux2:3128 dlpeapc.microcente:2859 ESTABLISHED
tcp 0 0 *:3128 *:* LISTEN
lsof listing:
Checking to see if squid is running...
0
Stopping squid server
0
Now sleeping for five minutes to give squid some time to shutdown...
Checking to see if squid is still running.
1
Squid is not running
Received on Mon Jan 07 2002 - 16:49:14 MST
This archive was generated by hypermail pre-2.1.9 : Tue Dec 09 2003 - 17:05:40 MST