Re: [squid-users] using squid for mirroring freesound.iua.upf.edu

Bram de Jong wrote:
> Hello all,
>
>
> over at freesound ( http://freesound.iua.upf.edu ) we're slowly but
> surely going over 3TB/month and we thought we would use squid for
> "mirroring" the content. Freesound hosts a lot of rather large
> (2MB->600MB) audio files. The whole Freesound colection of files is
> about 260G large.
>
> We set up 2 test-servers elsewhere with this config file :
> http://pastebin.com/f3f34e0c2 (search for FREESOUND_TODO to see the
> changed values ).
>
> What we have noticed is that the upload/download ratio of these
> test-machines is very close to 1 and isn't really lowering. In the log
> files we see a lot of TCP_HIT, SWAP_OUT and RELEASE. One of the test
> servers was running with 70GB of cache (of which only 27GB used!).
> When the cache was augmented to 90GB this morning (for testing), the
> ratio suddenly jumped to something MUCH better.

Thats swap.log right? it ONLY shows whats going on in the cache itself.
That says that some things are being cached.

>
> We're a bit confused, and it would be nice if someone could have a
> look at our config files to see if we have missed something obvious...

Start with enabling cache_log and access_log files.

You can set "debug_options ALL,0" to restrict the cache.log file to only
  log serious problems. But I highly recommend having that log going,
either to a file or to syslog if you prefer that way.

access_log - will show you what requests are going through squid and
where they are being sourced from; cache, or parent, or the memory cache.

cache_store_log is not entirely helpful unless you are doing low-level
disk debugging. It can safely be set to "none".

"http_access allow our_sites"

This line appears to be the only access line you have configured. By
itself it runs the risk of causing squid to lookup in DNS where the
"freesound.iua.upf.edu" domain can be found and pass the request there.
It's likely that the domain points at the squid box, so round things go.
But with cache.log turned off you will never be notified of these things.

I'd recommend adding "never_direct allow all" to force all requests to
the explicity located peer proxy.

Probably with a cache_peer_domain line to indicate which domains the
peer should be providing.

Other mentionables:

cache_effective_user - acceptable if your squid has been built with a
run-time user you don't want. The defaults are generally best on
packages versions though as teh maintainer has configured the OS properly.

cache_effective_group - I'd highly recommend setting the user/group
properly at the OS level and leaving group alone in squid.conf.

Amos

-- 
Please use Squid 2.6.STABLE19 or 3.0.STABLE4