Re: [squid-users] slow internet browsing. from Muhammad Yousuf Khan on 2012-04-27 (squid-users)

From: Muhammad Yousuf Khan <sirtcp_at_gmail.com>
Date: Fri, 27 Apr 2012 13:09:10 +0500

i think the delay was due to the 10mb of domain list. it seems that
things are back to track now. however for further restriction ill look
into other solutions as suggested in this thread.

Thanks alot

On Wed, Apr 25, 2012 at 9:13 PM, Muhammad Yousuf Khan <sirtcp_at_gmail.com> wrote:
> Thanks , i learn some thing new from you all. however ill update the
> results in few days as i am monitoring the stuff as how things are
> going.
>
> Thanks,
>
> On Wed, Apr 25, 2012 at 7:38 AM, Amos Jeffries <squid3_at_treenet.co.nz> wrote:
>> On 25/04/2012 3:34 a.m., Eliezer Croitoru wrote:
>>>
>>> On 24/04/2012 18:14, Muhammad Yousuf Khan wrote:
>>>>
>>>> ok i trim down config file ï¿½to this as you suggested of blocking
>>>> whitelist to local net.. let see how things work tommorw. ill update.
>>>> but block list is like 10MB big do you think it could be the
>>>> problem.as every query has to be matched with 10 MB database.
>>>>
>>>> ?
>>>
>>> in any case a dstdomain of 10MB is a very bad idea from what i know.
>>> one thing about dstdomain is that squid must validate the request dns
>>> records and it will take more bandwidth on dns queries.
>>
>>
>> Only if comparing a raw-IP to a domain name. If the raw-IP is on teh tested
>> URL it is faster as the DNS result gets re-used for all tests. The common
>> case though is straight domain-vs-domain comparisons.
>>
>> Amos
>>
>>
>>> if you still dont have local dns server for cahing only this is the time
>>> to add it.
>>>
>>> i think that 10MB of domains can be optimized into some basic DST DOMAINS
>>> REGEX and some blacklist DSTDOMS REGEX.
>>>
>>> i think that some db application for this kind of amount of dstdoms can
>>> much more effective.
>>> you can also use squidguard for that.
>>>
>>> if you can share some (1MB) of the dstdoms of the whole list i might be
>>> able to try to optimize it in a way.
>>>
>>>
>>> Regards,
>>> Eliezer
>>>
>>>>
>>>>
>>>>
>>>> #-------------Allow All ACL-------------
>>>> acl aci_lan src 10.51.100.0/24
>>>> acl aci_general src 10.51.100.0/24
>>>>
>>>> #---------------------Assurety ï¿½Whitelist---------------
>>>> acl aci_whitelist ï¿½dstdomain "/blocklist/aci_list/whitelist"
>>>> http_access allow aci_whitelist aci_general
>>>>
>>>> #----------TimeDomainBlock
>>>> acl aci_dest dstdomain "/blocklist/aci_list/time_block_domains"
>>>>
>>>> #--General Timing------------ Normal Days Working hours--------------
>>>> acl aci_working_hours time MTWH 10:04-13:04
>>>> acl aci_working_hours time MTWH 14:04-18:04
>>>> #--General Timing-------------Friday------------------------
>>>> acl aci_working_hours time F 10:04-13:04
>>>> acl aci_working_hours time F 15:04-18:04
>>>>
>>>> http_access deny ï¿½aci_dest aci_working_hours aci_general
>>>>
>>>>
>>>> On Tue, Apr 24, 2012 at 1:11 PM, Eliezer Croitoru<eliezer_at_ngtech.co.il>
>>>> ï¿½wrote:
>>>>>
>>>>> are you taking about the delay pools rules?
>>>>> also if it's a proxy that is open to the internet i would limit the
>>>>> access
>>>>> to port 3128 to only lan.
>>>>> your http_access rules are allowing anyone to use the proxy for the
>>>>> whitelist.
>>>>>
>>>>> Regards,
>>>>> Eliezer
>>>>>
>>>>>
>>>>>
>>>>> On 24/04/2012 09:06, Muhammad Yousuf Khan wrote:
>>>>>>
>>>>>>
>>>>>> ok i just disabled all the rules and it works for me now ill test
>>>>>> which rule is making a problem and let you know also.
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>> On Mon, Apr 23, 2012 at 11:20 PM, Muhammad Yousuf
>>>>>> Khan<sirtcp_at_gmail.com>
>>>>>> ï¿½wrote:
>>>>>>>
>>>>>>>
>>>>>>> here is the log for bbc.co.uk . first and last msg of log
>>>>>>>
>>>>>>> so you can see the time delay.
>>>>>>>
>>>>>>> 335205033.183 ï¿½ ï¿½841 10.51.100.240 TCP_MISS/200 24506 GET
>>>>>>> http://www.bbc.co.uk/ - DIRECT/212.58.244.66 text/html
>>>>>>> 1335205057.936 ï¿½ ï¿½328 10.51.100.240 TCP_REFRESH_HIT/304 435 GET
>>>>>>>
>>>>>>> http://static.bbci.co.uk/wwhomepage-3.5/1.0.41/img/broadcast-sprite.png
>>>>>>> - DIRECT/80.239.148.70 image/png
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Apr 23, 2012 at 11:12 PM, Muhammad Yousuf
>>>>>>> Khan<sirtcp_at_gmail.com>
>>>>>>> ï¿½wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> Here you go with my squid.conf
>>>>>>>>
>>>>>>>> acl all src all
>>>>>>>> acl manager proto cache_object
>>>>>>>> acl localhost src 127.0.0.1/32
>>>>>>>> acl to_localhost dst 127.0.0.0/8
>>>>>>>> acl localnet src 10.0.0.0/8 ï¿½ ï¿½ # RFC1918 possible internal network
>>>>>>>> acl localnet src 172.16.0.0/12 ï¿½# RFC1918 possible internal network
>>>>>>>> acl localnet src 192.168.0.0/16 # RFC1918 possible internal network
>>>>>>>> acl SSL_ports port 443 ï¿½ ï¿½ ï¿½ ï¿½ ï¿½# https
>>>>>>>> acl SSL_ports port 563 ï¿½ ï¿½ ï¿½ ï¿½ ï¿½# snews
>>>>>>>> acl SSL_ports port 873 ï¿½ ï¿½ ï¿½ ï¿½ ï¿½# rsync
>>>>>>>> acl Safe_ports port 80 ï¿½ ï¿½ ï¿½ ï¿½ ï¿½# http
>>>>>>>> acl Safe_ports port 21 ï¿½ ï¿½ ï¿½ ï¿½ ï¿½# ftp
>>>>>>>> acl Safe_ports port 443 ï¿½ ï¿½ ï¿½ ï¿½ # https
>>>>>>>> acl Safe_ports port 70 ï¿½ ï¿½ ï¿½ ï¿½ ï¿½# gopher
>>>>>>>> acl Safe_ports port 210 ï¿½ ï¿½ ï¿½ ï¿½ # wais
>>>>>>>> acl Safe_ports port 1025-65535 ï¿½# unregistered ports
>>>>>>>> acl Safe_ports port 280 ï¿½ ï¿½ ï¿½ ï¿½ # http-mgmt
>>>>>>>> acl Safe_ports port 488 ï¿½ ï¿½ ï¿½ ï¿½ # gss-http
>>>>>>>> acl Safe_ports port 591 ï¿½ ï¿½ ï¿½ ï¿½ # filemaker
>>>>>>>> acl Safe_ports port 777 ï¿½ ï¿½ ï¿½ ï¿½ # multiling http
>>>>>>>> acl Safe_ports port 631 ï¿½ ï¿½ ï¿½ ï¿½ # cups
>>>>>>>> acl Safe_ports port 873 ï¿½ ï¿½ ï¿½ ï¿½ # rsync
>>>>>>>> acl Safe_ports port 901 ï¿½ ï¿½ ï¿½ ï¿½ # SWAT
>>>>>>>> acl purge method PURGE
>>>>>>>> acl CONNECT method CONNECT
>>>>>>>>
>>>>>>>> # sqstat
>>>>>>>> acl manager proto cache_object
>>>>>>>> acl webserver src 10.51.100.206/255.255.255.255
>>>>>>>> http_access allow manager webserver
>>>>>>>> http_access deny manager
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> # Skype
>>>>>>>> acl numeric_IPs dstdom_regex
>>>>>>>>
>>>>>>>>
>>>>>>>> ^(([0-9]+\.[0-9]+\.[0-9]+\.[0-9]+)|(\[([0-9af]+)?:([0-9af:]+)?:([0-9af]+)?\])):443
>>>>>>>> acl Skype_UA browser ^skype
>>>>>>>> acl validUserAgent browser \S+
>>>>>>>>
>>>>>>>> # for cheetah only
>>>>>>>>
>>>>>>>> #acl usman src 10.51.100.107
>>>>>>>> #delay_pools 1
>>>>>>>> #delay_class 1 1
>>>>>>>> #delay_parameters 1 22000/22000
>>>>>>>> #delay_access 1 allow usman
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> #-------------Allow All ACL-------------
>>>>>>>> acl aci_lan src 10.51.100.0/24
>>>>>>>> acl aci_general src 10.51.100.0/24
>>>>>>>>
>>>>>>>>
>>>>>>>> #----My ip
>>>>>>>> acl my_ip src 10.51.100.240
>>>>>>>> http_access allow my_ip
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> # Testing delay pool
>>>>>>>> delay_pools 1
>>>>>>>> delay_class 1 1
>>>>>>>> delay_parameters 1 22000/10240000
>>>>>>>> delay_access 1 allow aci_general
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> #---------------------Assurety ï¿½Whitelist---------------
>>>>>>>> acl aci_whitelist ï¿½dstdomain "/blocklist/aci_list/whitelist"
>>>>>>>> http_access allow aci_whitelist
>>>>>>>>
>>>>>>>> #--Senior Allow Domainlist------------------------------
>>>>>>>> acl aci_seniors dstdomain "/blocklist/aci_list/whitelist_seniors"
>>>>>>>> #---------------------------------------------------------#See
>>>>>>>> implimentation in ACI implimentation section
>>>>>>>>
>>>>>>>> #--------------------Assurety ï¿½Hard_Block--------------
>>>>>>>> acl aci_hard_block dstdomain "/blocklist/aci_list/hard_block_domains"
>>>>>>>> http_access deny aci_hard_block
>>>>>>>>
>>>>>>>> #--------------------Hard_Block EXE and E.T.C---------------------
>>>>>>>> #acl mime_block_hard rep_mime_type -i
>>>>>>>> "/blocklist/aci_list/hard_mime_block"
>>>>>>>> #http_reply_access deny mime_block_hard
>>>>>>>>
>>>>>>>>
>>>>>>>> #--General------Streaming Block------------------------------
>>>>>>>> acl mime_block rep_mime_type -i "/blocklist/aci_list/time_mime_block"
>>>>>>>>
>>>>>>>> #--General Domainlist------------------------------
>>>>>>>> acl aci_dest dstdomain "/blocklist/aci_list/time_block_domains"
>>>>>>>>
>>>>>>>> #--Seniors MAC list ï¿½mouting------------------------------
>>>>>>>> acl aci_mac_seniors arp "/blocklist/aci_list/mac_list_seniors"
>>>>>>>>
>>>>>>>> #--General Timing------------ Normal Days Working hours--------------
>>>>>>>> acl aci_working_hours time MTWH 10:04-13:04
>>>>>>>> acl aci_working_hours time MTWH 14:04-18:04
>>>>>>>> #--General Timing-------------Friday------------------------
>>>>>>>> acl aci_working_hours time F 10:04-13:04
>>>>>>>> acl aci_working_hours time F 15:04-18:04
>>>>>>>>
>>>>>>>> #--General/Seniors-------------Implimentation------------------
>>>>>>>> http_access allow aci_seniors aci_mac_seniors
>>>>>>>> http_access deny ï¿½aci_dest aci_working_hours aci_general
>>>>>>>> http_reply_access deny mime_block aci_working_hours aci_general
>>>>>>>> !my_ip
>>>>>>>>
>>>>>>>> #skype deny
>>>>>>>> http_access deny numeric_IPS aci_working_hours
>>>>>>>> http_access deny Skype_UA aci_working_hours
>>>>>>>> http_access deny !validUserAgent aci_working_hours
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> #Error Directory by Ykhan
>>>>>>>> error_directory /usr/share/squid/errors/en-us/
>>>>>>>> #------------------------TheEnd----------------------
>>>>>>>> http_access allow aci_lan
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> http_access allow manager localhost
>>>>>>>> http_access deny manager
>>>>>>>> http_access allow purge localhost
>>>>>>>> http_access deny purge
>>>>>>>> http_access deny !Safe_ports
>>>>>>>> http_access deny CONNECT !SSL_ports
>>>>>>>> http_access allow localhost
>>>>>>>> http_access deny all
>>>>>>>> icp_access allow localnet
>>>>>>>> icp_access deny all
>>>>>>>> http_port 3128
>>>>>>>> hierarchy_stoplist cgi-bin ?
>>>>>>>> access_log /var/log/squid/access.log squid
>>>>>>>> refresh_pattern ^ftp: ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ 1440 ï¿½ ï¿½20% ï¿½ ï¿½ 10080
>>>>>>>> refresh_pattern ^gopher: ï¿½ ï¿½ ï¿½ ï¿½1440 ï¿½ ï¿½0% ï¿½ ï¿½ ï¿½1440
>>>>>>>> refresh_pattern -i (/cgi-bin/|\?) 0 ï¿½ ï¿½ 0% ï¿½ ï¿½ ï¿½0
>>>>>>>> refresh_pattern (Release|Package(.gz)*)$ ï¿½ ï¿½ ï¿½ ï¿½0 ï¿½ ï¿½ ï¿½ 20% ï¿½ ï¿½ 2880
>>>>>>>> refresh_pattern . ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ 0 ï¿½ ï¿½ ï¿½ 20% ï¿½ ï¿½ 4320
>>>>>>>> acl shoutcast rep_header X-HTTP09-First-Line ^ICY\s[0-9]
>>>>>>>> upgrade_http0.9 deny shoutcast
>>>>>>>> acl apache rep_header Server ^Apache
>>>>>>>> broken_vary_encoding allow apache
>>>>>>>> extension_methods REPORT MERGE MKACTIVITY CHECKOUT
>>>>>>>> hosts_file /etc/hosts
>>>>>>>> coredump_dir /var/spool/squid
>>>>>>>>
>>>>>>>> ##ykhan squid redirection to squidguard
>>>>>>>>
>>>>>>>> #redirect_program /usr/bin/squidGuard
>>>>>>>> #url_rewrite_program /usr/bin/squidGuard
>>>>>>>> #url_rewrite_children 5
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Apr 23, 2012 at 8:42 PM, Eliezer
>>>>>>>> Croitoru<eliezer_at_ngtech.co.il>
>>>>>>>> ï¿½wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 23/04/2012 18:38, Muhammad Yousuf Khan wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> well i have been experiencing slow Internet browsing. not very slow
>>>>>>>>>> but comparatively slower then IPCOP firewall. i can not understand
>>>>>>>>>> how
>>>>>>>>>> come i diagnose the issue.
>>>>>>>>>> i mean. i increase the RAM , i checked the DNS every thing is fine
>>>>>>>>>> but
>>>>>>>>>> my browser stuck at "connecting" ones it start download it do it
>>>>>>>>>> fast
>>>>>>>>>> but then stop for something then start. i am not getting the clear
>>>>>>>>>> picture. can anyone help
>>>>>>>>>>
>>>>>>>>>> i am suing debian 6.0.4 ï¿½with 2.7 stable squid.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>>
>>>>>>>>>> MYK
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> what is your exact problem? slow downloads?
>>>>>>>>> what is your squid setup?transparent ?regular forward proxy?
>>>>>>>>> what browser are you using?
>>>>>>>>> do you have some squid logs? or squid.conf?
>>>>>>>>> what dns server are you using?
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> Eliezer
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Eliezer Croitoru
>>>>>>>>> https://www1.ngtech.co.il
>>>>>>>>> IT consulting for Nonprofit organizations
>>>>>>>>> eliezer<at> ï¿½ ï¿½ngtech.co.il
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Eliezer Croitoru
>>>>> https://www1.ngtech.co.il
>>>>> IT consulting for Nonprofit organizations
>>>>> eliezer<at> ï¿½ngtech.co.il
>>>
>>>
>>>
>>
Received on Fri Apr 27 2012 - 08:09:18 MDT

This archive was generated by hypermail 2.2.0 : Sun Apr 29 2012 - 12:00:04 MDT