Re: [squid-users] Opinions sought on best storage type for FreeBSD

Michel Santos wrote:
> Tek Bahadur Limbu disse na ultima mensagem:
>> diskd indeed seems to fail under load especially when approaching
>> 200/300 requests per second.
>
> are you sure this numbers are correct? where do you get them from?

Hi Michel,

I am getting these numbers from one of my busy proxy server. At peak
times, I get anywhere from 150-200 requests per second. However to cross
the 300 mark, it only happens when 1 or 2 of my other proxy servers go
down and then our load balancer redirects web requests to whichever
proxy server is up and functioning.

>
>> It causes Squid to crash and restart automatically. Though, the side
>> effects are not noticed to the causal user, it prevents the cache from
>> stabilizing in the first place.
>
>
>
> in first place diskd does not cause automatic restart ;) that is RunCache
> who does it and I also do not believe that diskd cause squid to crash

Ok I should have said my Squid process crashes and RunCache restarts it
automatically. Since I am using diskd for most of my FreeBSD proxies, I
suspect that it might be the diskd storage which might be causing this
problem, especially while serving about 200 requests per second. Most of
my proxy servers serving less than 100 requests per second don't have
this problem.

Again I stress that it's not noticeable if the Squid process gets
restarted. So I only come to know of it after sometime while checking
it's graphs and other stats.

I guess that I may have to really commit my time and resources to find
out if other factors could be causing this to happen.

Haven't you faced any automatic restart of your Squid process. Does that
mean that your Squid process uptime is months?

>
>
> if the crash really happens then there is something wrong on your machine
>
> if the problem is the load and your computer can not handle the load then
> it first gets slow or you get out of memory and then squid may crash but
> you better should look what is really wrong there before blaming the fs
> type you use

I don't think that there is anything wrong with my machines and their
configurations. But I also don't want to say that they are 100% perfect
because then why would people develop newer versions of softwares. They
are machines which we need to fine tune correctly and it is always an
ongoing process which will never stop.

They have been in production for years and each of their average uptime
is about 120 days. As far as the load is concerned, my CPU usage never
goes above 30-40% but sometimes my memory usage crosses 80% of it's
capacity though.

By the way, do you have some optimal settings which can be applied to
diskd? Below are some values I use:

options SHMSEG=128
options SHMMNI=256
options SHMMAX=50331648 # max shared memory segment size (bytes)
options SHMALL=16384 # max amount of shared memory (pages)
options MSGMNB=16384 # max # of bytes in a queue
options MSGMNI=48 # number of message queue identifiers
options MSGSEG=768 # number of message segments
options MSGSSZ=64 # size of a message segment
options MSGTQL=4096 # max messages in system

Correct me where necessary.

>
>> If I opt to use aufs, will the following compilations work?
>>
>> '--enable-async-io' '--with-pthreads'
>>
>
> with-pthreads is not necessary
>
> but certainly this switch is kind of strange for freebsd since you need to
> remap the process-threads to kernel-threads in order to get it right
> (faster), both thread implementations should work well then with kqueue
> which also is correctly detected by configure when available
>

Can you explain in a layman point of view regarding remapping the
process-threads to kernel-threads?

Why don't you write an article on this topic so that people would know
how to fine tune their FreeBSD OS if they opt to use the aufs storage
schema.

I would certainly appreciate it.

Thanking you...

>
> Michel
> ...
>
>
>
>
> ****************************************************
> Datacenter Matik http://datacenter.matik.com.br
> E-Mail e Data Hosting Service para Profissionais.
> ****************************************************
>
>
>
>

-- 
With best regards and good wishes,
Yours sincerely,
Tek Bahadur Limbu
(TAG/TDG Group)
Jwl Systems Department
Worldlink Communications Pvt. Ltd.
Jawalakhel, Nepal
http://www.wlink.com.np