Re: [squid-users] Squid->DG->Squid

On 28/07/11 19:27, Andrew Rogers wrote:
> Hi Amos
>
> Thanks for the detailed reply, I will study your comments accordingly
> and will make changes to my squid.conf where necessary thankyou.
>
> On 27 July 2011 07:15, Amos Jeffries<squid3_at_treenet.co.nz> wrote:
>>
>>
>> This is the sandwich configuration looping on itself. You haev several
>> choices:
>>
>> * configure two instances of Squid
>> * configure client->DG->Squid
>> * configure client->Squid->DG
>> * disable "via off" and cross your fingers there is never any actual
>> infinite loop. You will need a Squid built with HTTP violations enabled to
>> do that. Loop protection is REQUIRED by the HTTP standards.
>
> I have decided to with a Squid1->DG->Squid2 approach now, and have
> already been changing my config files etc accordingly.
>
>> This only solves the loop issue though. Cache MISS is separate...
>
> After some initial testing the loop issue has disappeared, and iam
> getting quite a few more what looks like cache hits now of
> "TCP_REFRESH_UNMODIFIED" not which are showing like:-
>
> 1311836509.795 162 localhost TCP_REFRESH_UNMODIFIED/304 553 GET
> http://i2.cdnds.net/11/30/P/gaming_sidpirates1.jpg -
> DIRECT/213.244.185.38 -
> 1311836509.795 163 mycomp.tg.local TCP_MISS/304 691 GET
> http://i2.cdnds.net/11/30/P/gaming_sidpirates1.jpg [email protected]
> FIRST_UP_PARENT/127.0.0.1
>
> Which iam assumeing it has had a successfull cache hit from Squid2?

Looks that way. The particular example was a revalidation request
though. If they are both logging to the one file first line is squid2,
second line squid1?

> With you saying Cache MISS is seperate, will using the 2 seperate
> Squid instances automatically have a better hit rate by the looks
> already from here?

It will resolve cases such as ones where a cache HIT prevents the
request going through DG controls of whether the client is allowed that
object.

>
>> On the whole, since you have requests that skip DG, and some get cached you
>> are not in a position to only have caching on the second loop. Things
>> entering the first loop WILL be found in the cache even if you wanted them
>> to go through DG first.
>
> Why do some request's skip DG and go direct from the first Squid
> instance? I know I have asked whitelistsites to always go direct
> which is not a problem, but it is other sites whcih initially go
> through Squid->DG but them pages etc which the sites load up seem to
> go direct in some circumstances, why is this? and can I configure so
> all traffic is forced through DG then to Squid2 instance? If so how
> can I achieve this, and what are the knock on effects if any?

Some requests are best served that way rather than going through a
hierarchy. Such as CONNECT requests which are explicit requests to do that.
  nonhierarchichal_direct and hierarchy_stoplist control whether these
types of requests are required to go through the peer (DG) or allowed to
go direct.

The effect is a reduction of speed on those requests.

>
> Would something along the line of work:-
>
> http_reply_access allow localhost
> http_reply_access deny blockedsites
> http_reply_access allow all
> always_direct allow localhost
> always_direct allow whitelistsites
> always_direct deny all
>
> I would want all traffic to go from Squid1->DG and then all traffic
> from DG->Squid2 which would then pass all the traffic out direct as
> expected?
> What can of config along this line would I need in squid2.conf?

http_reply_access is *way* too late to be doing anything like
destination selection. The request has already left squid via some path
and the reply is coming back.

always_direct forces things to go direct. It does not prevent.

never_direct along with the hierarchical controls mentioned above is
what you want to use.

>
> so far this is my changed squid1.conf file:-
>
> ####### /etc/squid3/squid.conf Configuration File #######
> ####### cache manager
> cache_mgr squid_at_mydomain.co.uk
> ####### kerberos authentication
> auth_param negotiate program /usr/lib/squid3/squid_kerb_auth -d -s
> HTTP/squid.my.local
> auth_param negotiate children 10
> auth_param negotiate keep_alive on
> ####### provide access via ldap for clients not authenticated via kerberos
> auth_param basic program /usr/lib/squid3/squid_ldap_auth -R \
> -b "dc=my,dc=local" \
> -D squid_at_my.local \
> -w "password" \
> -f sAMAccountName=%s \
> -h dc.my.local
> auth_param basic children 10
> auth_param basic realm Internet Proxy
> auth_param basic credentialsttl 1 minute
> ####### ldap authorizations
> # restricted proxy access logged
> external_acl_type internet_users %LOGIN /usr/lib/squid3/squid_ldap_group -R -K \
> -b "dc=my,dc=local" \
> -D squid_at_my.local \
> -w "password" \
> -f "(&(objectclass=person)(sAMAccountName=%v)(memberof=cn=Internet
> Users,ou=Internet Groups,dc=my,dc=local))" \
> -h dc.my.local
> # full proxy access no logging
> external_acl_type internet_users_full_nolog %LOGIN
> /usr/lib/squid3/squid_ldap_group -R -K \
> -b "dc=my,dc=local" \
> -D squid_at_my.local \
> -w "password" \
> -f "(&(objectclass=person)(sAMAccountName=%v)(memberof=cn=Internet
> Users Full NoLog,ou=Internet Groups,dc=my,dc=local))" \
> -h dc.my.local
> # full proxy access logged
> external_acl_type internet_users_full_log %LOGIN
> /usr/lib/squid3/squid_ldap_group -R -K \
> -b "dc=my,dc=local" \
> -D squid_at_my.local \
> -w "password" \
> -f "(&(objectclass=person)(sAMAccountName=%v)(memberof=cn=Internet
> Users Full Log,ou=Internet Groups,dc=my,dc=local))" \
> -h dc.my.local
> ####### acl for proxy auth and ldap authorizations
> acl auth proxy_auth REQUIRED
> # format "acl, aclname, acltype, acltypename, activedirectorygroup"
> acl RestrictedAccessLog external internet_users Internet\ Users
> acl FullAccessNoLog external internet_users_full_nolog Internet\
> Users\ Full\ NoLog
> acl FullAccessLog external internet_users_full_log Internet\ Users\ Full\ Log
> acl whitelistsites url_regex -i "/etc/squid3/whitelistsites.txt"
> acl blockedsites url_regex -i "/etc/squid3/blockedsites.txt"
>
> ##Public DMZ Filtering, do not need to authenticate create acl type
> acl DmzPublicWifiSubnet src 10.0.0.0/24
>
> ####### squid defaults
> acl manager proto cache_object
> acl localhost src 127.0.0.1/32 ::1
> acl to_localhost dst 127.0.0.0/8 0.0.0.0/32 ::1
> acl SSL_ports port 443
> acl Safe_ports port 80-81 # http
> acl Safe_ports port 21 # ftp
> acl Safe_ports port 443 # https
> acl Safe_ports port 70 # gopher
> acl Safe_ports port 210 # wais
> acl Safe_ports port 1025-65535 # unregistered ports
> acl Safe_ports port 280 # http-mgmt
> acl Safe_ports port 488 # gss-http
> acl Safe_ports port 591 # filemaker
> acl Safe_ports port 777 # multiling http
> acl CONNECT method CONNECT
> http_access allow manager localhost
> http_access deny manager
> http_access deny !Safe_ports
> http_access deny CONNECT !SSL_ports
> http_access allow localhost
> ####### enforce auth: order of rules is important for authorization levels
> no_cache deny whitelistsites

Remember "no_cache" == "cache"

For no caching you want:
  cache deny all

> http_access allow whitelistsites
>
> ## Allow DMZ Public Wifi Access out without authorisation
> http_access allow DmzPublicWifiSubnet
>
> http_access allow FullAccessNoLog auth
> http_access allow FullAccessLog auth
> http_access deny blockedsites
> http_access allow RestrictedAccessLog auth
>
> ####### logging
> # don't log FullAccessNoLog
> access_log /var/log/squid3/access-auth.log squid !FullAccessNoLog
>
> ####### squid defaults
> http_access deny all
>
> #Log Connecting Client DNS Names instead on IP Names.
> log_fqdn on
>
> http_port 8080
> ##Push Traffic Through DansGuradian for Content Filtering
> cache_peer 127.0.0.1 parent 8081 0 no-query proxy-only no-delay
> no-netdb-exchange no-digest connect-timeout=15 login=PASS
> cache_peer_Access 127.0.0.1 deny localhost
> ##Push Public DMZ Wifi Subnet through DG
> cache_peer_access 127.0.0.1 allow DmzPublicWifiSubnet
>
> cache_peer_access 127.0.0.1 allow RestrictedAccessLog
> cache_peer_access 127.0.0.1 deny all
>
> pid_filename /var/run/squid3-auth.pid
>
> visible_hostname mysquid-auth.my.local
>
> hierarchy_stoplist cgi-bin ?

any URL with "?" or "cgi-bin" in it will go DIRECT from this Squid.

Remove "hierarchy_stoplist".

Add these:
  nonhierarchical_direct off

to force everything going to DG to ONLY go that way also add these:

  never_direct allow DmzPublicWifiSubnet
  never_direct allow RestrictedAccessLog

> coredump_dir /var/spool/squid3
> refresh_pattern ^ftp: 1440 20% 10080
> refresh_pattern ^gopher: 1440 0% 1440
> refresh_pattern -i (/cgi-bin/|\?) 0 0% 0
> refresh_pattern . 0 20% 4320
>
> squid2.conf:-
> ####### /etc/squid3/squid.conf Configuration File #######
> ####### cache manager
> cache_mgr squid_at_mycache.co.uk
> ####### kerberos authentication
> auth_param basic program /usr/lib/squid3/squid_ldap_auth -R \
> -b "dc=my,dc=local" \
> -D squid_at_my.local \
> -w "password" \
> -f sAMAccountName=%s \
> -h dc.my.local
> auth_param basic children 10
> auth_param basic realm Internet Proxy
> auth_param basic credentialsttl 1 minute
> ####### ldap authorizations
> # restricted proxy access logged
> external_acl_type internet_users %LOGIN /usr/lib/squid3/squid_ldap_group -R -K \
> -b "dc=my,dc=local" \
> -D squid_at_my.local \
> -w "password" \
> -f "(&(objectclass=person)(sAMAccountName=%v)(memberof=cn=Internet
> Users,ou=Internet Groups,dc=my,dc=local))" \
> -h dc.my.local
> # full proxy access no logging
> external_acl_type internet_users_full_nolog %LOGIN
> /usr/lib/squid3/squid_ldap_group -R -K \
> -b "dc=my,dc=local" \
> -D squid_at_my.local \
> -w "password" \
> -f "(&(objectclass=person)(sAMAccountName=%v)(memberof=cn=Internet
> Users Full NoLog,ou=Internet Groups,dc=my,dc=local))" \
> -h dc.my.local
> # full proxy access logged
> external_acl_type internet_users_full_log %LOGIN
> /usr/lib/squid3/squid_ldap_group -R -K \
> -b "dc=my,dc=local" \
> -D squid_at_my.local \
> -w "password" \
> -f "(&(objectclass=person)(sAMAccountName=%v)(memberof=cn=Internet
> Users Full Log,ou=Internet Groups,dc=my,dc=local))" \
> -h dc.my.local
> ####### acl for proxy auth and ldap authorizations
> acl auth proxy_auth REQUIRED
> # format "acl, aclname, acltype, acltypename, activedirectorygroup"
> acl RestrictedAccessLog external internet_users Internet\ Users
> acl FullAccessNoLog external internet_users_full_nolog Internet\
> Users\ Full\ NoLog
> acl FullAccessLog external internet_users_full_log Internet\ Users\ Full\ Log
> acl whitelistsites url_regex -i "/etc/squid3/whitelistsites.txt"
> acl blockedsites url_regex -i "/etc/squid3/blockedsites.txt"
>
> ##Public DMZ Filtering, do not need to authenticate create acl type
> acl DmzPublicWifiSubnet src 10.0.0.0/24
>
> ####### squid defaults
> acl manager proto cache_object
> acl localhost src 127.0.0.1/32 ::1
> acl to_localhost dst 127.0.0.0/8 0.0.0.0/32 ::1
> acl SSL_ports port 443
> acl Safe_ports port 80-81 # http
> acl Safe_ports port 21 # ftp
> acl Safe_ports port 443 # https
> acl Safe_ports port 70 # gopher
> acl Safe_ports port 210 # wais
> acl Safe_ports port 1025-65535 # unregistered ports
> acl Safe_ports port 280 # http-mgmt
> acl Safe_ports port 488 # gss-http
> acl Safe_ports port 591 # filemaker
> acl Safe_ports port 777 # multiling http
> acl CONNECT method CONNECT
> http_access allow manager localhost
> http_access deny manager
> http_access deny !Safe_ports
> http_access deny CONNECT !SSL_ports
> http_access allow localhost
> ####### enforce auth: order of rules is important for authorization levels
> no_cache deny whitelistsites
> http_access allow whitelistsites
>
> ## Allow DMZ Public Wifi Access out without authorisation
> http_access allow DmzPublicWifiSubnet
>
> http_access allow FullAccessNoLog auth
> http_access allow FullAccessLog auth
> http_access deny blockedsites
> http_access allow RestrictedAccessLog auth
>
> ####### logging
> # don't log FullAccessNoLog
> access_log /var/log/squid3/access-proxy.log squid !FullAccessNoLog
>
> ####### squid defaults
> http_access deny all
>
> #Log Connecting Client DNS Names instead on IP Names.
> log_fqdn on
>
> http_port 127.0.0.1:3128
>
> pid_filename /var/run/squid3-proxy.pid
>
> visible_hostname mysquid-proxy.my.local
>
> hierarchy_stoplist cgi-bin ?
> coredump_dir /var/spool/squid3
> refresh_pattern ^ftp: 1440 20% 10080
> refresh_pattern ^gopher: 1440 0% 1440
> refresh_pattern -i (/cgi-bin/|\?) 0 0% 0
> refresh_pattern . 0 20% 4320
>
> End of squid2.conf...
>
> I know I have got duplicate acl's which don't need to be in
> squid2.conf, i will remove these shortly, but just need to make sure
> iam removing the correct lines which are no longer needed.
> How can I make squid1 not cache anything, as I want all the cacheing
> to be done at squid2?

see comments above inline with the configs.

Amos

-- 
Please be using
   Current Stable Squid 2.7.STABLE9 or 3.1.14
   Beta testers wanted for 3.2.0.10