Re: [squid-users] cache hit 100%

UK SquidUser (AXA-TECH-UK) wrote:
> Hi...
> we are running squid 2.6 stable 17 that have been built within the last
> 6 months.
> The problem we are experiencing is that one of our servers has hit 100%
> on the proxycache partition. It is configured in its own filesystem, it
> shares with nothing... it has a partition of 36Gb and squid is
> configured in squid.conf to use 28Gb. Once it has got to 28Gb we see the
> following messages in cache.log (some entries have been deleted,
> hopefully I have the relevant ones below)
>
> Rebuilding storage in /proxycache (DIRTY)
> Store rebuilding is 0.1% complete
> 2009/01/06 12:36:11| diskHandleWrite: FD 536: disk write error: (28) No
> space left on device
> 2009/01/06 12:36:11| storeAufsWriteDone: got failure (-6)
> 2009/01/06 12:36:11| storeSwapOutFileClosed: dirno 0, swapfile 00001392,
> errflag=-6
> (28) No space left on device
> 2009/01/06 12:36:29| WARNING: newer swaplog entry for dirno 0, fileno
> 000015D8
> 2009/01/06 12:44:13| Store rebuilding is 92.2% complete
> 2009/01/06 12:44:15| WARNING: Disk space over limit: 13552 KB > 13512 KB
> 2009/01/06 12:44:26| WARNING: Disk space over limit: 13552 KB > 13512 KB
>

Huh. Here Squid is saying that the disk usage is ~13MB.

> FATAL: xcalloc: Unable to allocate 1 blocks of 28 bytes!
> Squid Cache (Version 2.6.STABLE17): Terminated abnormally.
> CPU Usage: 531.367 seconds = 482.302 user + 49.066 sys
> Maximum Resident Size: 0 KB
> Page faults with physical i/o: 0
> Memory usage for squid via mallinfo():
> total space in arena: -307440 KB
>

I don't think that's supposed to be negative...

> Ordinary blocks: -308814 KB 87176 blks
> Small blocks: 0 KB 0 blks
> Holding blocks: 25248 KB 6 blks
> Free Small blocks: 0 KB
> Free Ordinary blocks: 1373 KB
> Total in use: -283566 KB 100%
> Total free: 1373 KB 0%
> 2009/01/06 12:44:27| Not currently OK to rewrite swap log.
> 2009/01/06 12:44:27| storeDirWriteCleanLogs: Operation aborted.
> 2009/01/06 12:44:38| Store rebuilding is 0.0% complete
>
> And this process repeats continually. We have taken it out of service,
> but previously to that it did look like it was processing requests in
> the cache.. ie TCP_HIT's were appearing in the log. But surely what is
> appearing in the cache log is incorrect.
>
> Squid is configured with the default cache_swap_low and cache_swap_high
> and cache_mem 1024 MB, maximum_object_size 16384 KB
> maximum_object_size_in_memory 64 KB
> cache_replacement_policy heap GDSF
> memory_replacement_policy heap GDSF
> cache_dir aufs /proxycache 28000 256 256
>
> Can anyone advise why the proxycache hits 100% and it appears to have
> problems when it does?
>

My first guess would be either swap.state or filesystem corruption.
Find and remove the swap.state files (if you didn't specify a location
in squid.conf, it should reside in /proxycache) and see if that fixes
it. If that doesn't do it, reformat /proxycache, run squid -z to
rebuild the directory structure and start it up.

> Thanks, K.
>
> Kev Shurmer
> Network Analyst - TS Data Networks
> AXA Technology Services
> kev.shurmer_at_axa-tech.com
> Tel. : +44 1 253 68 4652 - Mob. : +44 7974 83 0090

Chris
Received on Thu Jan 08 2009 - 20:40:27 MST