I've re-compiled with --enable-dlmalloc and it didn't make a difference. I'm still leaking memory. The cachemgr lists 427621 KB in use, but only 260926 KB accounted for. Here's the cachemgr output:
Squid Object Cache: Version 2.5.STABLE4-20030929
Start Time: Wed, 01 Oct 2003 03:58:41 GMT
Current Time: Thu, 02 Oct 2003 15:07:46 GMT
Connection information for squid:
Number of clients accessing cache: 0
Number of HTTP requests received: 3500368
Number of ICP messages received: 378876
Number of ICP messages sent: 379091
Number of queued ICP replies: 0
Request failure ratio: 0.00
Average HTTP requests per minute since start: 1659.7
Average ICP messages per minute since start: 359.4
Select loop called: 17631204 times, 7.177 ms avg
Cache information for squid:
Request Hit Ratios: 5min: 68.9%, 60min: 67.2%
Byte Hit Ratios: 5min: 9.5%, 60min: 19.1%
Request Memory Hit Ratios: 5min: 6.0%, 60min: 7.5%
Request Disk Hit Ratios: 5min: 12.9%, 60min: 13.8%
Storage Swap size: 31896476 KB
Storage Mem size: 98292 KB
Mean Object Size: 14.27 KB
Requests given to unlinkd: 0
Median Service Times (seconds) 5 min 60 min:
HTTP Requests (All): 0.01847 0.02592
Cache Misses: 0.17711 0.17711
Cache Hits: 0.01309 0.01648
Near Hits: 0.10281 0.10857
Not-Modified Replies: 0.01387 0.01648
DNS Lookups: 0.00464 0.00669
ICP Queries: 0.00541 0.00575
Resource usage for squid:
UP Time: 126544.845 seconds
CPU Time: 24888.760 seconds
CPU Usage: 19.67%
CPU Usage, 5 minute avg: 64.31%
CPU Usage, 60 minute avg: 54.81%
Process Data Segment Size via sbrk(): 427621 KB
Maximum Resident Size: 0 KB
Page faults with physical i/o: 837
Memory usage for squid via mallinfo():
Total space in arena: 427621 KB
Ordinary blocks: 405719 KB 102690 blks
Small blocks: 0 KB 0 blks
Holding blocks: 17336 KB 9 blks
Free Small blocks: 0 KB
Free Ordinary blocks: 21901 KB
Total in use: 423055 KB 95%
Total free: 21901 KB 5%
Total size: 444957 KB
Memory accounted for:
Total accounted: 260926 KB
memPoolAlloc calls: 427293143
memPoolFree calls: 422443907
File descriptor usage for squid:
Maximum number of file descriptors: 4096
Largest file desc currently in use: 997
Number of file desc currently in use: 877
Files queued for open: 0
Available number of file descriptors: 3219
Reserved number of file descriptors: 100
Store Disk files open: 3
Internal Data Structures:
2241473 StoreEntries
18942 StoreEntries with MemObjects
18926 Hot Object Cache Items
2234980 on-disk objects
-----Original Message-----
From: Mike Mitchell
Sent: Monday, September 29, 2003 6:23 PM
To: squid-users@squid-cache.org
Subject: Squid memory leak?
I've been chasing what looks like a memory leak for quite some time. I've seen it all through the Squid 2.X releases, and it still is present in 2.5STABLE4. I've seen it on machines running HP-UX 10.20 and on Red Hat Linux. Currently I'm running on three identical Dell 2650 servers, each with dual 1.8 GHz processors, 2 GB of RAM, and two 20 GB cache partitions. They are all running Red Hat ES 2.1, kernel 2.4.9-e.25smp.
Squid is built with these configuration options:
Squid Cache: Version 2.5.STABLE4
configure options: --prefix=/local/proxy/squid --enable-cache-digests
--enable-underscores --with-pthreads --enable-storeio=aufs,ufs
--enable-removal-policies=heap --enable-gnuregex
With a full cache, Squid memory utilization starts at about 250 MB. It then grows at about 300 MB per week. The graph at
ftp://ftp.sas.com/pub/nam/squid-mem.png
shows how the memory grows.
We see about 80 requests per second during business hours. The graph
ftp://ftp.sas.com/pub/nam/squid-reqs.png
shows our requests per second.
We use DNS round-robin to load balance requests between the three systems.
Each squid process forwards most requests on to a Trend Interscan VirusWall process listening on port 8080 on the same server. It also sends the URLs through the "adzap" redirector. ACLs are used to decide which URLs are sent through the redirector and also which URLs are sent to the virus scanner.
Here's some squid information output from the cachemgr.cgi: ===========================================================
Squid Object Cache: Version 2.5.STABLE4
Start Time: Sat, 20 Sep 2003 19:29:16 GMT
Current Time: Mon, 29 Sep 2003 21:14:57 GMT
Connection information for squid:
Number of clients accessing cache: 0
Number of HTTP requests received: 17264989
Number of ICP messages received: 1961488
Number of ICP messages sent: 1961585
Number of queued ICP replies: 0
Request failure ratio: 0.00
Average HTTP requests per minute since start: 1321.4
Average ICP messages per minute since start: 300.3
Select loop called: 90171815 times, 8.694 ms avg
Cache information for squid:
Request Hit Ratios: 5min: 62.3%, 60min: 65.9%
Byte Hit Ratios: 5min: 15.8%, 60min: 21.1%
Request Memory Hit Ratios: 5min: 4.5%, 60min: 4.2%
Request Disk Hit Ratios: 5min: 18.4%, 60min: 12.5%
Storage Swap size: 31876696 KB
Storage Mem size: 98348 KB
Mean Object Size: 14.58 KB
Requests given to unlinkd: 0
Median Service Times (seconds) 5 min 60 min:
HTTP Requests (All): 0.02317 0.02317
Cache Misses: 0.16775 0.16775
Cache Hits: 0.01309 0.01469
Near Hits: 0.09736 0.10857
Not-Modified Replies: 0.01387 0.01469
DNS Lookups: 0.00464 0.00464
ICP Queries: 0.00316 0.00541
Resource usage for squid:
UP Time: 783941.680 seconds
CPU Time: 118154.720 seconds
CPU Usage: 15.07%
CPU Usage, 5 minute avg: 65.24%
CPU Usage, 60 minute avg: 60.68%
Process Data Segment Size via sbrk(): 556455 KB
Maximum Resident Size: 0 KB
Page faults with physical i/o: 33606
Memory usage for squid via mallinfo():
Total space in arena: 556455 KB
Ordinary blocks: 491460 KB 532257 blks
Small blocks: 0 KB 0 blks
Holding blocks: 19384 KB 12 blks
Free Small blocks: 0 KB
Free Ordinary blocks: 64995 KB
Total in use: 510844 KB 89%
Total free: 64995 KB 11%
Total size: 575839 KB
Memory accounted for:
Total accounted: 263830 KB
memPoolAlloc calls: 2122989169
memPoolFree calls: 2118393830
File descriptor usage for squid:
Maximum number of file descriptors: 4096
Largest file desc currently in use: 877
Number of file desc currently in use: 465
Files queued for open: 0
Available number of file descriptors: 3631
Reserved number of file descriptors: 100
Store Disk files open: 4
Internal Data Structures:
2199786 StoreEntries
13566 StoreEntries with MemObjects
13530 Hot Object Cache Items
2186456 on-disk objects ===================================================
What is most interesting from this is that mallinfo says 556455 KB is in the arena, but squid can only account for 263830 KB of it.
Here's the interesting bits of our configuration: ===================================================
http_port 80 8000
cache_peer 127.0.0.1 parent 8080 7 no-query no-digest default cache_peer inetgw02.unx.sas.com sibling 80 3130 proxy-only cache_peer inetgw03.unx.sas.com sibling 80 3130 proxy-only cache_peer inetgw02.unx.sas.com parent 8080 7 no-query no-digest round-robin cache_peer inetgw03.unx.sas.com parent 8080 7 no-query no-digest round-robin cache_mem 96 MB cache_swap_low 95 cache_swap_high 97 maximum_object_size 700 MB maximum_object_size_in_memory 32 KB cache_replacement_policy heap LFUDA memory_replacement_policy heap GDSF cache_dir aufs /cache/cache1/squid 16384 53 254 cache_dir aufs /cache/cache2/squid 16384 53 254 cache_store_log none emulate_httpd_log on acl all src 0.0.0.0/0.0.0.0 acl noAds myport 80 acl Reads method GET HEAD acl CIDR_A src 10.0.0.0/255.0.0.0 acl virus_proto proto HTTP acl novirus-url urlpath_regex -i \.gif(\?.*)?$ \.jpg(\?.*)?$ \.png(\?.*)?$ ident_lookup_access allow CIDR_A ident_lookup_access deny all memory_pools on log_icp_queries off client_db off
always_direct allow novirus-url always_direct allow !virus_proto never_direct allow all strip_query_terms off redirect_program /opt/adzap/wrapzap redirect_children 10 redirector_access deny !Reads redirector_access allow noAds redirector_bypass on ===================================================
Here's some messages that are showing up in the cache.log file.
===================================================
2003/09/29 16:38:26| sslReadServer: FD 578: read failure: (104) Connection reset by peer 2003/09/29 16:39:00| sslReadServer: FD 574: read failure: (104) Connection reset by peer 2003/09/29 16:41:44| sslReadServer: FD 560: read failure: (104) Connection reset by peer 2003/09/29 16:42:14| sslReadServer: FD 485: read failure: (104) Connection reset by peer 2003/09/29 16:42:54| urlParse: Illegal character in hostname 'inetgw01.unx.sas.com:80inetgw01.unx.sas.com'
2003/09/29 16:42:55| urlParse: Illegal character in hostname 'inetgw01.unx.sas.com:80inetgw01.unx.sas.com'
2003/09/29 16:42:59| urlParse: Illegal character in hostname 'inetgw01.unx.sas.com:80inetgw01.unx.sas.com'
2003/09/29 16:43:03| urlParse: Illegal character in hostname 'inetgw01.unx.sas.com:80inetgw01.unx.sas.com'
2003/09/29 16:43:07| urlParse: Illegal character in hostname 'inetgw01.unx.sas.com:80inetgw01.unx.sas.com'
2003/09/29 16:43:08| urlParse: Illegal character in hostname 'inetgw01.unx.sas.com:80inetgw01.unx.sas.com'
2003/09/29 16:43:17| urlParse: Illegal character in hostname 'inetgw01.unx.sas.com:80inetgw01.unx.sas.com'
2003/09/29 16:43:21| urlParse: Illegal character in hostname 'inetgw01.unx.sas.com:80inetgw01.unx.sas.com'
2003/09/29 16:43:31| urlParse: Illegal character in hostname 'inetgw01.unx.sas.com:80inetgw01.unx.sas.com'
2003/09/29 16:43:32| urlParse: Illegal character in hostname 'inetgw01.unx.sas.com:80inetgw01.unx.sas.com'
2003/09/29 16:43:33| urlParse: Illegal character in hostname 'inetgw01.unx.sas.com:80inetgw01.unx.sas.com'
2003/09/29 16:43:34| urlParse: Illegal character in hostname 'inetgw01.unx.sas.com:80inetgw01.unx.sas.com'
2003/09/29 16:43:46| urlParse: Illegal character in hostname 'inetgw01.unx.sas.com:80inetgw01.unx.sas.com'
2003/09/29 16:43:47| urlParse: Illegal character in hostname 'inetgw01.unx.sas.com:80inetgw01.unx.sas.com'
2003/09/29 16:44:00| urlParse: Illegal character in hostname 'inetgw01.unx.sas.com:80inetgw01.unx.sas.com'
2003/09/29 16:44:00| urlParse: Illegal character in hostname 'inetgw01.unx.sas.com:80inetgw01.unx.sas.com'
===================================================
Does anyone have an idea of why I'm consuming so much memory? Am I just running into memory fragmentation issues? Should I be linking with a different version of malloc? Would the configuration option '--enable-dlmalloc' help? I've worked around this problem by having a cron job check how large the squid process is. If it grows too large I restart it. Currently it gets restarted twice a month.
------------------------------------------------------------------------------
Mike Mitchell
Mike.Mitchell@sas.com
(919) 677-8000 X16793
Received on Thu Oct 02 2003 - 09:18:51 MDT
This archive was generated by hypermail pre-2.1.9 : Tue Dec 09 2003 - 17:20:16 MST
