RE: [squid-users] Streaming asx files through squid

>
>
>> -----Original Message-----
>> From: Amos Jeffries [mailto:squid3@treenet.co.nz]
>> Sent: 16 January 2008 21:12
>> To: Tris Hoar
>> Cc: squid-users@squid-cache.org
>> Subject: Re: [squid-users] Streaming asx files through squid
>>
>> > Hi List,
>> >
>> > We are fairly new to using squid and are in the process of migrating
>> 400
>> > odd
>> > proxy servers over to it. This is going fairly smoothly at the moment.
>> > However some of the customers have reported that
>> http://www.heartfm.co.uk
>> > is
>> > no longer streaming radio for them.
>> >
>> > The server that the audio streams from firstly sends this HTML file to
>> the
>> > client
>> >
>> > <asx version = "3.0">
>> > <title>Heart West Midlands 100.7 Live Stream</title>
>> > <entry>
>> >
>> > <title>LIVE :
>> > Heart1007</title>
>> >
>> > <ref
>> > href="http://wm-global.london.as34763.net/Heart1007" />
>> >
>> > </entry>
>> >
>> > </asx>
>> >
>> >
>> > Then http://wm-global.london.as34763.net/Heart1007
>> >
>> >
>> >
>> > Points to
>> >
>> >
>> >
>> > [Reference]
>> >
>> > Ref1=http://wm-global.london.as34763.net/Heart1007?MSWMExt=.asf
>> >
>> > Ref2=http://192.168.97.221:80/Heart1007?MSWMExt=.asf
>> >
>> >
>> >
>> > If we use http://wm-global.london.as34763.net/Heart1007?MSWMExt=.asf
>> > directly in Media player 11 this works through squid
>> >
>> >
>> >
>> > Our setup is a local squid server at each site > to DNS round robin of
>> > parent servers in the core > Internet
>> >
>> > The squid servers are built on Xeon 3.0 GHz servers with 2GB ram and
>> > ~500GB
>> > disk space of which 400GB is cache. The OS is REHL 5.1 and running
>> Squid
>> > 2.6.STABLE6 (we are using this version due to redhat offering support
>> for
>> > it)
>>
>> For that size disk cache the RAM is very low. A rule of thumb for
>> general
>> principles is to dedicate 10MB-RAM per 1 GB-disk just to hold the cache
>> index. If your primary business is streaming media you can get away with
>> less (very-large files need less index per disk byte).
>
> The customers are Schools and many of them have subscriptions to services
> like Espresso and Knowledgebox. These services are web based and have a
> lot
> of video content on them. The plan is to pre populate most of the cache
> space with the video content using an application called cachestream. Thus
> the low memory should not be a problem.
>
>>
>> >
>> > Using the parents directly or the old proxy server's this streams fine
>> > though windows media player 11.
>> >
>> >
>> >
>> > Our squid config is
>> >
>> > ------------------------------------------------------------------------
>> ----
>> >
>> > http_port 80
>>
>> Is this meant to be a website accelerator or customer proxy?
>>
>> Accelerators need accel or vhost and could do with defaultsite= in their
>> http_port.
>>
>
> The servers are intended to function as a standard web proxy's, not as
> accelerators.

In which case port 80 is not the best choice to open. Particularly in
light of the unrestrictive ACLs below.

>
>>
>> >
>> > hierarchy_stoplist cgi-bin ?
>> >
>> > acl QUERY urlpath_regex cgi-bin \?
>> >
>> > cache deny QUERY
>> >
>>
>> Are you sure you don't want to cache anything with '?' in the URI?
>> The dev team are now recommending killing this and using a
>> refresh_pattern
>> instead to allow caching of properly working dynamic pages/files.
>>
>
> We were just using the example included with the default squid.conf for
> this. Do you have an example of a refresh_pattern we could try?

We are now recomending a default set of refresh_patterns:

 refresh_pattern ^ftp: 1440 20% 10080
 refresh_pattern ^gopher: 1440 0% 1440
 refresh_pattern \? 0 0% 0
 refresh_pattern cgi-bin 0 0% 0
 refresh_pattern . 0 20% 4320

Instead of these:
 acl QUERY urlpath_regex cgi-bin \?
 cache deny QUERY

>
>>
>> >
>> > acl apache rep_header Server ^Apache
>> >
>> > broken_vary_encoding allow apache
>> >
>> >
>> > cache_replacement_policy heap LRU
>> >
>> > memory_replacement_policy heap GDSF
>> >
>> > cache_dir aufs /var/spool/squid 400000 64 256
>> >
>> > access_log /var/log/squid/access.log
>> >
>> > access_log none
>> >
>>
>> This second access_log line is not correct. 'none' is not a proper
>> filename for the log, did you mean 'cache_log none'?.
>> I would recommend having a cache_log with 'debug_options ALL,0' so that
>> you get a log of even just the critical errors squid has.
>>
>
> Opps! We have a duplicated line
> access_log /var/log/squid/access.log <- should not have been there
> We monitor this from our core servers rather than the edge devices. Though
> while testing I will have the log on.
>
>> >
>> > refresh_pattern ^ftp: 1440 20% 10080
>> >
>> > refresh_pattern ^gopher: 1440 0% 1440
>> >
>> > refresh_pattern . 0 20% 4320
>> >
>> >
>> >
>> >
>> >
>> > acl all src 0.0.0.0/0.0.0.0
>> >
>> > acl manager proto cache_object
>> >
>> > acl localhost src 127.0.0.1/255.255.255.255
>> >
>> > acl to_localhost dst 127.0.0.0/8
>> >
>> > acl SSL_ports port 443 563 8443
>> >
>> > acl Safe_ports port 80 81 21 443 563 70 210 280 554 488 591 777 889
>> > 1025-65535
>> >
>> > acl CONNECT method CONNECT
>> >
>> > acl Proxy_bypass dstdomain .misportal.net .bgfl.org
>> >
>> >
>> >
>> >
>> >
>> > #Access Control Rules
>> >
>> > http_access allow manager localhost
>> >
>> > http_access deny manager
>> >
>> > http_access deny !Safe_ports
>> >
>> > http_access deny CONNECT !SSL_ports
>> >
>> > http_access allow localhost
>> >
>> > http_access allow all
>>
>> Oh yay! An open web proxy on a high-speed server.
>>
>> If this is actually a site accelerator note the comment after http_port
>> above, and you will need some ACL restricting the destination sites by
>> name or rDNS. Probably listing some cache_peer's as preferred
>> information
>> sources.
>>
>
> The proxy sits out at the customer's site which connects back to our
> servers
> over a 10MB MPLS connection. This connection is firewalled from the
> internet
> and restricted with ACL's on the routers at each site so that only people
> at
> that site or at our office can connect to the server. Every one at the
> site
> MUST use the proxy for internet access however we do not wish to use a
> transparent proxy. Rather than having a custom config for each site we
> allow
> any one to connect to the server as only people at that site will be able
> to.
>
>> >
>> > http_reply_access allow all
>> >
>> > icp_access allow all
>> >
>> >
>> >
>> > always_direct allow Proxy_bypass
>> >
>> > never_direct allow all
>> >
>> >
>> >
>> > nonhierarchical_direct off
>> >
>> >
>> >
>> > coredump_dir /var/spool/squid
>> >
>> > visible_hostname eduproxy.bgfl.org
>> >
>> >
>> >
>> > cache_effective_user squid
>> >
>> > cache_effective_group squid
>> >
>> > cachemgr_passwd xxxxxx info stats/objects
>> >
>> >
>> >
>> > ------------------------------------------------------------------------
>> ----
>> >
>> >
>> >
>> > If you can shed any light on how to resolve this that would be great!
>> >
>>
>> Firstly check that the clients software is even using the proxy.
>> This should show up as entries in access.log for the stream URI.
>>
>
> This is showing up in the access log fine.
> TCP_MISS/200 6310 GET http://meta.as34763.net/content/9.asx -
> FIRST_UP_PARENT/parentproxy.bgfl.org video/x-ms-asf
>
>> Then enable a cache_log and see if there are any serious problems
>> happening in squid about that time.
>
> Nothing showing up in the cache_log that seams wrong
>
>>
>> Then check the request headers going to/from squid for one of the
>> failing
>> requests. Also, does it work with any other software? and what does that
>> show?
>>
>
> Looking at TCPDump I think the problem is due to the streaming server
> (which
> we do not control) trying to talk in HTTP1.1 and Squid not understanding
> it.
> Unless we can find a better solution what we are planning to do is set the
> core servers to just rewrite the url from
> http://meta.as34763.net/content/9.asx
> Into
> http://wm-global.london.as34763.net/Heart1007?MSWMExt=.asf
>
>
>> If that does not lead you to a solution, we will need to know the squid
>> release (including the stable part) and what you have managed to find
>> out
>> from trying the above.
>>
>>
>
> 2.6.Stable6 - this is the RedHat maintained release.

Aha, older release + broken streaming server. Try it with a later squid on
general principles. We have been seeing more and more of these problems
with broken HTTP/1.1 servers and are adding compatibility hacks with
nearly every release now.

Amos
Received on Thu Jan 17 2008 - 15:49:39 MST