[squid-users] Peering caches (squid and 3rd parties) - How to from Guillermo Javier Nardoni

From: Guillermo Javier Nardoni - Grupo GERYON <guillermo_at_geryon.com.ar>
Date: Tue, 11 Jun 2013 17:24:19 -0300

Hello everyone,

We have this situation and we tried a lot of configurations without success.

• 1000 Customers
• 4 Caches BOX running Squid 2.7 on Debian Squeeze • Caches are full-meshed
to each other • Every Squid is running in transparent mode (http_port 3128
transparent) • Every Squid is running HAARPCACHE on localhost at port 8080
(HAARPCACHE is a Thundercache 3.1 fork wich Works PERFECT for caching sites
like youtube with lots of HITS) .
• Every Squid is connected to Internet through RB1 • RB2 (Mikrotik RouterOS)
is doing round-robin selection on every squid redirecting all trafic to port
80 to internet to port 3128 on squid

cat /etc/haarp/haarp.lst
root_at_cpe-58-1-26-172:/etc/haarp# cat /etc/haarp/haarp.lst
http.*\.4shared\.com.*(\.exe|\.iso|\.torrent|\.zip|\.rar|\.pdf|\.doc|\.tar|\
.mp3|\.mp4|\.avi|\.wmv)
http.*\.avast\.com.*(\.def|\.vpu|\.vpaa|\.stamp)
http.*(\.avg\.com|\.grisoft\.com|\.grisoft\.cz).*(\.bin|\.exe)
http.*(\.avgate\.com|\.avgate\.net|\.freeav\.net|\.freeav\.com).*(\.gz)
http.*\.bitgravity\.com.*(\.flv\.mp4)
http.*\.etrustdownloads\.ca\.com.*(\.tar|\.zip|\.exe|\.pkg)
http.*flashvideo\.globo\.com.*(\.mp4|\.flv)
http.{1,4}vsh\.r7\.com\/.*(\.mp4)$
74\.125\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[
0-9][0-9]?)
#http.*\.googlevideo\.com.*videoplayback
#http.*fpatch\.grandchase\.com\.br.*(\.kom|\.mkom|\.mp3)
http.*(\.kaspersky-labs\.com|\.geo\.kaspersky\.com|kasperskyusa\.com).*(\.av
c|\.kdc|\.klz|\.bz2|\.dat|\.dif)
#http.*\.mccont\.com.*\.flv
http.*\.metacafe\.com.*\.flv
http.{1,4}media\w*\.justin.tv\/archives\/(\w|\/|-)*\.flv(\?.*|$)
http.{1,4}\w*juegos\w*\.juegosdiarios\.com\/(\w|\/|-)*\.swf$
http.{1,4}\w*\.juegosjuegos\.com\/games(\w|\/|-)*\.swf$
##http.*(\.windowsupdate\.com|(\.microsoft\.com)).*(\.cab|\.exe|\.iso|\.zip|
\.psf)
http.*(\.windowsupdate\.com|(update|download|dlservice|windowsupdate)\.micro
soft\.com)\/.*(\.cab|\.exe|\.iso|\.zip|\.psf|\.txt|\.crt)$
http.*\.pornotube\.com.*\.flv
http.*\.terra\.com.*\.flv
#http.*uol\.com\.br.*\.flv
http.*\.viddler\.com.*\.flv
#http.*\.video\.msn\.com.*\.flv
http.*(porn|img).*\.xvideos\.com\/videos\/(thumbs\/)?.*(\.jpg|\.flv\?.*|\.mp
4\?.*)$
http.*\.youtube\.com.*videoplayback\?
http.*\.ziddu\.com.*(\.exe|\.iso|\.torrent|\.zip|\.rar|\.pdf|\.doc|\.tar|\.m
p3|\.mp4|\.avi|\.wmv)
http.*edgecastcdn\.net/.*(\.mp4|\.flv)
http.*adobe\.com/.*(\.cab|\.aup|\.exe|\.msi|\.upd|\.msp)
http.*\.eset\.com.*\.nup
http.*\.nai\.com.*(\.zip|\.tar|\.exe|\.gem)
http.*\.pop6\.com.*(\.flv)
http.*\.symantecliveupdate\.com.*(\.zip|\.exe)
#http.*\.xpg\.com\.br.*
http.{1,4}\w*\.ytimg\.com.*(hqdefault(\.jpg|\.mp4)$|M[0-9]+\.jpg\?sigh=)
http.{1,4}\w*google(\.\w|\w)*\.doubleclick\.net\/pagead\/ads\?.*
http.*img[0-9]\.submanga\.com\/(hd)?pages\/.*(\.jpg|\.webp)
http.*(profile|s?photos|video).{0,5}\.ak\.fbcdn\.net\/.*(\.mp4\?.*|\_[a-z]\.
jpg$|\.mp4$|\_[a-z]\.png$)
#http.*(profile|s?photos|video).{0,5}\.ak\.fbcdn\.net\/.*(\.mp4\?.*|\_n\.jpg
$|\.mp4$|\_n\.png$)
http.*\.video\.pornhub\.\w*\.com\/videos\/.*\.flv\?.*
http.*\.(publicvideo|publicphoto)\.xtube\.com\/(videowall\/)?videos?\/.*(\.f
lv\?.*|\_Thumb\.flv$)
http.*public\.tube8\.com\/.*\.mp4.*
http.*videos\..*\.redtubefiles\.com\/.*\.flv
(205\.196\.|199\.91\.)[0-9]{2,3}\.[0-9]{1,3}\/.*
#http.*\.rapidshare\.com\/cgi-bin\/.*\.cgi\?.*sub=download
http.*\.vimeo.com\/.*\.mp4(\?.*)?$
http.*images\.orkut\.com\/orkut\/photos\/.*\.jpg$
http.{1,4}(\w|\/|\.|-)*media\.tumblr\.com\/(\w|\/|-|\.)*tumblr(\w|\/|-)*(\.p
ng|\.jpg)$
#http.{1,7}speedtest(\w|-)*(\.|\w)+\/speedtest\/(random.*\.jpg|latency\.txt)
\?.*
#http.{1,10}testdevelocidad.{1,5}\/speedtest\/(random.*\.jpg|latency\.txt)\?
.*
#http.{1,7}(\.|[a-z]|[0-9]|-)+(\/\w+)?(\/speedtest)+\/(random[0-9]+x[0-9]+\.
jpg|latency\.txt)

As you can well see!, youtube and many others sites is cachings its content
through HAARPCACHE and not by squid itself. BTW It Works GREAT.

Configuration on every squid.conf at /etc/squid

Proxy1:
IP: 192.168.1.1

cache_peer�� 192.168.2.1�� sibling�� 3128�� 3130��
proxy-only cache_peer�� 192.168.3.1�� sibling�� 3128��
3130�� proxy-only cache_peer�� 192.168.4.1�� sibling��
3128�� 3130�� proxy-only

acl haarp_lst url_regex -i "/etc/haarp/haarp.lst"
cache deny haarp_lst
cache_peer 127.0.0.1 parent 8080 0 proxy-only no-digest dead_peer_timeout 2
seconds cache_peer_access 127.0.0.1 allow haarp_lst cache_peer_access
127.0.0.1 deny all

Proxy2:
IP: 192.168.2.1

cache_peer�� 192.168.1.1�� sibling�� 3128�� 3130��
proxy-only cache_peer�� 192.168.3.1�� sibling�� 3128��
3130�� proxy-only cache_peer�� 192.168.4.1�� sibling��
3128�� 3130�� proxy-only

Proxy3:
IP: 192.168.3.1

cache_peer�� 192.168.2.1�� sibling�� 3128�� 3130��
proxy-only cache_peer�� 192.168.1.1�� sibling�� 3128��
3130�� proxy-only cache_peer�� 192.168.4.1�� sibling��
3128�� 3130�� proxy-only

Proxy4:
IP: 192.168.4.1

cache_peer�� 192.168.2.1�� sibling�� 3128�� 3130��
proxy-only cache_peer�� 192.168.3.1�� sibling�� 3128��
3130�� proxy-only cache_peer�� 192.168.1.1�� sibling��
3128�� 3130�� proxy-only

Everything “Works” fine when you browse sites but those who MUST go through
HAARPCACHE PEER don’t.
Let’s picture that.

Client 1 ask for http://www.youtube.com/watch?v=juqyzgnbspY . RB2, through
its round-robin selection, redirect this petition to Proxy1

Proxy1 accept this connection and according to “acl haarp_lst”, it goes to
cache_peer 127.0.0.1.

Proxy1 -> Peer 127.0.0.1:
� Is http://www.youtube.com/watch?v=juqyzgnbspY on local cache?.
o If� YES: RETURN HIT with the FILE
o If NO: RETURN MISS, download the file from Internet, save it to disk and
serve the file.

Client 80 ask for http://www.youtube.com/watch?v=juqyzgnbspY . RB2, through
its round-robin selection, redirect this petition to Proxy3

Proxy3 accept this connection and according to “acl haarp_lst”, it goes to
cache_peer 127.0.0.1.

Proxy3 -> Peer 127.0.0.1:
� Is http://www.youtube.com/watch?v=juqyzgnbspY on local cache?.
o If� YES: RETURN HIT with the FILE
o If NO: RETURN MISS, download the file from Internet, save it to disk and
serve the file.

As you can see, the same file is downloaded twice (at least) if the petition
is not redirected to the same cache box.
How can I achieve the goal to ask every cache and if the file is cached on
any sibling or parent it shouldn’t be downloaded from internet but the cache
itself.

Note 1: I can run HAARPCACHE on 0.0.0.0/0 if this is a solution.

The schematic bellow shows how are connected clients, caches and routers.
http://picpaste.com/njTPeEBb.jpg

Please forgive my errors writing in english.
Received on Tue Jun 11 2013 - 20:29:33 MDT

This archive was generated by hypermail 2.2.0 : Wed Jun 12 2013 - 12:00:17 MDT