Re: [squid-users] High CPU usage when cache full

Hi,

I don't have any scientific metrics regarding mem cache versus disk
and cache, apart from metrics collected by keynote.com which show a
slight inprovment in overall site speed which may or may not be
related.. I had been using both memory and disk until fairly recently
but feel I should have enough RAM to cache frequently accessed files
in Memory only and avoid any potential disk buffer issues during
flashmob type events where our traffic can potentially increase 10
fold in a matter of minutes.

Will try LRU on one of the machines next time I do a restart.

squidclient output below.

HTTP/1.0 200 OK
Server: squid
Date: Tue, 27 Nov 2007 16:33:36 GMT
Content-Type: text/plain
Expires: Tue, 27 Nov 2007 16:33:36 GMT
Last-Modified: Tue, 27 Nov 2007 16:33:36 GMT
X-Cache: MISS from www.xxxxxxx.xxx
X-Cache-Lookup: MISS from www.xxxxxxx.xxx:80
Via: 1.0 www.xxxxxxxx.xxx:80 (squid)
Connection: close

Squid Object Cache: Version 2.6.STABLE6
Start Time: Mon, 26 Nov 2007 11:16:17 GMT
Current Time: Tue, 27 Nov 2007 16:33:36 GMT
Connection information for squid:
        Number of clients accessing cache: 43833
        Number of HTTP requests received: 14361302
        Number of ICP messages received: 0
        Number of ICP messages sent: 0
        Number of queued ICP replies: 0
        Request failure ratio: 0.00
        Average HTTP requests per minute since start: 8172.3
        Average ICP messages per minute since start: 0.0
        Select loop called: 175178637 times, 0.602 ms avg
Cache information for squid:
        Request Hit Ratios: 5min: 87.5%, 60min: 87.2%
        Byte Hit Ratios: 5min: 48.2%, 60min: 53.3%
        Request Memory Hit Ratios: 5min: 44.7%, 60min: 45.0%
        Request Disk Hit Ratios: 5min: 0.1%, 60min: 0.2%
        Storage Swap size: 0 KB
        Storage Mem size: 6181676 KB
        Mean Object Size: 0.00 KB
        Requests given to unlinkd: 0
Median Service Times (seconds) 5 min 60 min:
        HTTP Requests (All): 0.00091 0.00179
        Cache Misses: 0.00286 0.00286
        Cache Hits: 0.00179 0.00179
        Near Hits: 0.00562 0.00678
        Not-Modified Replies: 0.00091 0.00091
        DNS Lookups: 0.00000 0.03223
        ICP Queries: 0.00000 0.00000
Resource usage for squid:
        UP Time: 105439.283 seconds
        CPU Time: 5072.935 seconds
        CPU Usage: 4.81%
        CPU Usage, 5 minute avg: 8.48%
        CPU Usage, 60 minute avg: 7.95%
        Process Data Segment Size via sbrk(): 8582112 KB
        Maximum Resident Size: 0 KB
        Page faults with physical i/o: 0
Memory usage for squid via mallinfo():
        Total space in arena: 193504 KB
        Ordinary blocks: 193413 KB 14 blks
        Small blocks: 0 KB 0 blks
        Holding blocks: 17104 KB 3 blks
        Free Small blocks: 0 KB
        Free Ordinary blocks: 90 KB
        Total in use: 210517 KB 100%
        Total free: 90 KB 0%
        Total size: 210608 KB
Memory accounted for:
        Total accounted: 7735377 KB
        memPoolAlloc calls: 1284461973
        memPoolFree calls: 1253790650
File descriptor usage for squid:
        Maximum number of file descriptors: 16384
        Largest file desc currently in use: 4287
        Number of file desc currently in use: 4080
        Files queued for open: 0
        Available number of file descriptors: 12304
        Reserved number of file descriptors: 100
        Store Disk files open: 0
        IO loop method: epoll
Internal Data Structures:
        1138371 StoreEntries
        1138371 StoreEntries with MemObjects
        1138143 Hot Object Cache Items
             0 on-disk objects

J

On Nov 26, 2007 2:55 PM, Tek Bahadur Limbu <teklimbu@wlink.com.np> wrote:
> Hi John,
>
> John Moylan wrote:
> > Hi,
> >
> > I have three memory only caches set up 7GB of memory each (the
> > machines have 12GB of physical memory each). Throughput is fairly high
> > and this setup works well in reducing the number of requests for
> > smaller files from my backend storage with lower latency that a disk
> > and mem. solution.
>
> Do you have statistics regarding fetching from memory and disk? How much
> is the performance increment when using memory cache only?
>
>
> However, the cache's on of the machines fill up
> > every 2-3 days and Squid's CPU usage subsequently goes up to 100%
> > (These are all dual SMP machines and system load average remains
> > around 0.7). FD's, the number of connections and swap are all fine
> > when the CPU goes up so the culprit is more than likely to be cache
> > replacement.
> >
> > I am using heap GDSF as the policy. The maximum size in memory is set
> > to 96 KB.
>
> Have you tried the LFUDA or the default LRU memory replacement policies?
>
> I am using squid-2.6.STABLE6-4.el5 on Linux 2.6.
>
> Try upgrading to the latest version of squid.
>
> http://www.squid-cache.org/Versions/v2/2.6/squid-2.6.STABLE16.tar.gz
>
> It probably contains some improvements over version 2.6.6.
>
> >
> > Is there anything I can do to improve expensive cache replacement
> > apart from stopping and starting Squid every day?
>
> By the way, which Linux distro are you using?
>
> Can you post the output of "squidclient mgr:info" or the relevant parts
> of your squid.conf?
>
> Thanking you...
>
>
> >
> > J
> >
> >
> >
>
>
> --
>
> With best regards and good wishes,
>
> Yours sincerely,
>
> Tek Bahadur Limbu
>
> System Administrator
>
> (TAG/TDG Group)
> Jwl Systems Department
>
> Worldlink Communications Pvt. Ltd.
>
> Jawalakhel, Nepal
>
> http://www.wlink.com.np
>
> http://teklimbu.wordpress.com
>
Received on Tue Nov 27 2007 - 09:44:51 MST