Re: SYN floods?

Jon Drukman wrote:
>
> Recently I have been getting a bunch of complaints about my Squid machine,
> which I am using strictly in http-accelerator mode. (We are running
> SQUID2.2-STABLE2 on FreeBSD as img.gamespot.com to accelerate images for
> www.gamespot.com.)
>
> First, the complaints were that we were flooding a bunch of identd
> requests. I rebuilt squid without the identd code. Now I am getting
> complaints about SYN floods. I can't make it happen and since I only get a
> complaint once every few days I'm assuming it's a transient problem. Does
> anybody know anything about this?

Well, a SYN flood occurs any time you start making TCP connections to a
host faster than they can respond to them. I don't actually know offhand
how many it takes for a SYN flood to be reported, but it's not unusual
to see them on busy webservers, in my experience.

Taking a WAG (wild-assed-guess) at what's going on, I might guess that
you have a bunch of images expiring from the accelerator's cache at
about the same time, (Gamespot has quite a few small to mid-size
images), and at some point a request is going to cause revalidation on a
bunch of those at once.

If you're of a mind to, you could attach a cheapish box, run tcpdump
into a logfile, and - when someone reports a SYN flood (I'm assuming
this is happening inside your network) - marry the time up with the
tcpdump logs and just take a peek at what's going on. Probably it's just
a short burst of revalidations, and not much to worry about.

> Question 2: Since I'm running in accelerator-only mode, do I need the
> dnsservers at all? If not, how can I disable them?

Not my department. I'll leave someone else to answer this.

> Question 3: I always get the "WARNING! Your cache is running out of
> filedescriptors" message even though I have allocated 16000+ descriptors to
> the process. Is that not enough?

Well, now, squid's file-descriptor usage (and indeed, that of most
proxies) is a function of number-of-requests-per-second and
average-connection-length. Now, usually you'll need - say - two
descriptors per request. One to the client, and one to the server or
cache-file. (This is a slight simplification)...so..if you have a
request rate of 40/sec and the average connection duration is 30
seconds, then you'll be using an average of 30*40*2 (2400) descriptors
at any given moment. Short-term network delays that increase latency
between you and the client, or you and the server that push up the
connection duration will drastically increase that number. For example,
if the average connection duration goes up 15 seconds (to 45) then
you'll wind up using 45*40*2 descriptors (3600). Quite a jump.

If you're going out through a proxy, a stall in the packet-trains
upstream (due to congestion or route-flapping) or congestion or loss on
the client-side can do appalling things to the number of
file-descriptors you have in-use. And worse...once you pass a certain
mark, the problem becomes pathological, and simply managing the number
of descriptors becomes time-consuming enough that you _contribute_ to
connection-duration times, thus exacerbating the problem.

Things to do:
Check squid's startup messages in the cache-log. It'll tell you how many
descriptors it _thinks_ it has available. That number may not equal the
number you _think_ it can use. If they don't match, try reconfiguring
and recompiling again. If that fails, call for help.

Do some analysis: What is your average connection duration? How many
requests per second? Write a little script and plot yourself some
data-points. Compare the graph of these two with requests-per-second vs
(connection duration divided by object size).

Consider using the cache-manager interface to check the FD statistics
periodically to find out what they're doing.

Lower your persistent-connection timeouts, if you really are using that
many.

(BTW, I (and probably others) often use gamespot.com as a test-site
(it's certainly on our testing checklists, to see whether some of our
gimmicks work properly with layered pages, and black backgrounds) as
well as actually using the site more-or-less recreationally. I rather
hope we're not contributing to any problems you might be having)
>
> Question 4: When I start squid, I get the following series of messages:

You appear to be missing an ACL definition:

# Match all origin addresses
acl all src 0/0

> 1999/06/24 17:46:31| realconf line 18: ident_lookup_access deny all
> 1999/06/24 17:46:31| aclParseAccessLine: ACL name 'all' not found.
> 1999/06/24 17:46:31| realconf line 18: ident_lookup_access deny all
> 1999/06/24 17:46:31| aclParseAccessLine: Access line contains no ACL's,
> skipping
> 1999/06/24 17:46:31| realconf line 19: http_access deny all
> 1999/06/24 17:46:31| aclParseAccessLine: ACL name 'all' not found.
> 1999/06/24 17:46:31| realconf line 19: http_access deny all
> 1999/06/24 17:46:31| aclParseAccessLine: Access line contains no ACL's,
> skipping
> 1999/06/24 17:46:31| realconf line 20: icp_access deny all
> 1999/06/24 17:46:31| aclParseAccessLine: ACL name 'all' not found.
> 1999/06/24 17:46:31| realconf line 20: icp_access deny all
> 1999/06/24 17:46:31| aclParseAccessLine: Access line contains no ACL's,
> skipping
>
> These are troubling because there are only 16 lines in the configuration file!
>
> Here is the complete configuration file:
>
> http_port 80
> icp_port 0
> cache_mem 8 MB
> cache_dir /data1/squid/cache 500 16 256
> cache_access_log /dev/null
> cache_log /usr/local/squid/logs/squid-cache.log
> cache_store_log none
> emulate_httpd_log on
> pid_filename /usr/local/squid/logs/squid.pid
> cache_mgr jsd
> cache_effective_user nobody
> cache_effective_group nobody
> visible_hostname www.gamespot.com
> httpd_accel_host www.west.gamespot.com
> httpd_accel_port 80
> append_domain .gamespot.com

Consider:
* Disabling emulate_http_log. Your logs are going into the bit-bucket,
so diddling with the format is probably a waste of time. I don't
guarantee that it uses much more in the way of cpu-cycles to do the
httpd format, but converting the timestamp's got to be a microscopic
waste, and that all adds up. Especially if nobody ever sees it.

D
Received on Thu Jun 24 1999 - 16:57:21 MDT