Re: HUGE size of squid binary after compile

Henrik Nordstrom wrote:
>
> John Line (as web server manager) wrote:
>
> > Are you sure about that ...? I've never seen Solaris 2 report fragments (the
> > only thing I recall reporting fragments is fsck...), though by default space
> > is reported in 512-byte blocks rather than KB (seems to be a SysV versus BSD
> > distinction) - "df -k" will get KB. The free space percentages should be OK
> > though, giving percent of non-reserved space free.
>
> I have been told that Solaris 2.6 df reports free diskspace as the sum of
> free fragments presented as 512-byte blocks or KB depending on which
> options you use.
>
> The problem is then that the filesystem uses blocks of 4KB or 8KB divided
> into fragments, and only the last portion of a file may be located in a
> block fragment. If your filesystem allocation policy is "optimize for
> time" and you have lots of small files then you may end up in a situation
> where you have lots of free fragments, but no really free filesystem
> blocks. It is then impossible to store files requiring a full filesystem
> block or more (or is the limit is one fragment?).

Ah, I misunderstood - thought you were saying that df actually displayed the
number of fragments. I hadn't heard about the problem you describe here, but
if true I suppose it could be described as both "reasonable" (it is the
amount of free space, after all) and "misleading" (since it is not
necessarily all usable).
 
> If you change allocation policy to "optimize for space" then the last
> portion of a file is always located in a fragment, and the filesystem
> tries to maintain as many free blocks as possible. If the policy is
> "optimize for time" then it never uses fragments of a partially used
> block if there is a free block available.

True, *but* if you optimise for space so it uses best-match fragments
(i.e. multiple of the fragment size chunk that will hold the new data)
it is not only slower to find and allocate a fragment, but it every time the
file is extended beyond the existing allocation the final fragment (if less
than a full block) will have to be copied to the new, slightly larger,
fragment. For something like squid with *very* busy log files, I'd imagine
that growing another 512 or 1024 bytes happens *very* frequently. And since
(if Solaris 2's ufs implementation matches that described in the ("The")
4.3BSD book) the fragment used at the end of a file when it is first written
will always be best-match even with optimisation for time, it should only be
files that are extended subsequently (e.g. logs) that actually risk
using/wasting full blocks - and that's likely to be a very small number of
files.

It's not clear, therefore (unless I've missed something, always possible)
why fragmentation should be a problem (in the absence of bugs...) with
something like Squid cache directories. Unless the log files are on the same
partition, the metadata file might well be the only file that was being
extended after initial creation. While the cached documents will often end
with a fragment, it seems like that should always be a best-match fragment
from the free list, not causing gratuitous fragmentation. *If* Solaris
behaves as described in the 4.3BSD book.

That aside, since Solaris (and BSD) will switch automatically from time to
space optimisation (when the free list contains too many fragments),
superficially at least it ought to "just work". I've certainly seen console
messages about time/space changes with both news spools and Squid caches on
Solaris 2.x (when the partition was getting very full), so it definitely can
switch unaided (and indeed, I'm not sure if a manual setting fixes it
permanently or serves solely to start it off in the specified state).

I can believe it may hit problems in spite of that, but wonder if it may
(for example) happen primarily or only with the 2.6FCS bug that results in
the effective free space reserve being zero, which might well make a mess of
the decision about when to switch optimisation mode.

                                John Line

-- 
University of Cambridge WWW manager account (usually John Line)
Send general WWW-related enquiries to webmaster@ucs.cam.ac.uk