RE: [squid-users] Band New Squid Log Analyzer!

Hi,

What AWK version do you use?
You can tray nawk and gawk also...

Regards

-----Original Message-----
From: Li Wei [mailto:liw@jftt.fujitsu.com]
Sent: Friday, September 12, 2003 05:34
To: Yuri N. Fominov
Subject: Re: [squid-users] Band New Squid Log Analyzer!

hi,

sorry. I got it.
quickly running this script, but I faced problem listed following:
------------------------------------------------------------------------

----
orca/liw$awk -f scalar.awk access-log.20030911
awk: syntax error near line 98
awk: illegal statement near line 98
awk: syntax error near line 99
awk: illegal statement near line 99
awk: syntax error near line 100
awk: illegal statement near line 100
awk: syntax error near line 101
awk: illegal statement near line 101
awk: syntax error near line 102
awk: illegal statement near line 102
awk: syntax error near line 351
awk: illegal statement near line 351
awk: syntax error near line 352
awk: illegal statement near line 352
awk: syntax error near line 361
awk: illegal statement near line 361
awk: syntax error near line 362
awk: illegal statement near line 362
awk: syntax error near line 363
awk: illegal statement near line 363
awk: syntax error near line 364
awk: illegal statement near line 364
awk: syntax error near line 371
awk: illegal statement near line 371
awk: syntax error near line 372
awk: illegal statement near line 372
awk: syntax error near line 373
awk: illegal statement near line 373
awk: syntax error near line 374
awk: illegal statement near line 374
awk: syntax error near line 375
awk: illegal statement near line 375
awk: syntax error near line 376
awk: illegal statement near line 376
awk: syntax error near line 377
awk: illegal statement near line 377
awk: syntax error near line 378
awk: illegal statement near line 378
awk: syntax error near line 379
awk: illegal statement near line 379
awk: syntax error near line 387
awk: bailing out near line 387
------------------------------------------------------------------------
-----
I guess it caused by the lack of reconfiguration.
can you help me?
----- Original Message ----- 
From: "Yuri N. Fominov" <YuriF@risk.az>
To: "Squid-users" <squid-users@squid-cache.org>
Sent: Friday, September 12, 2003 12:24 AM
Subject: RE: [squid-users] Band New Squid Log Analyzer!
> 
> Here is a sample report for your reference:
> 
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> ~~
> ~~~~~~~
> ~~~~  SCALAR (Squid Cache Advanced Log Analyzer & Reporter), Version
> 0.91  ~~~~
> ~~~~
> ~~~~
> ~~~~   (C) 2003 by Yuri N. Fominov, YuriF@risk.az,
http://scalar.risk.az
> ~~~~
> ~~~~  SCALAR has no warranty and it is completely free, so you are
> welcome ~~~~
> ~~~~  to re-distribute this pretty useful piece of software. Good
> Luck2All ~~~~
>
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> ~~~~~~~
> 
> 
> 
> 
> ~~~ Analysis Headlines 
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> 
>   Log Start Time [02-10-2003 04:00:07]
>     Log End Time [13-10-2003 01:01:16]
>   Lines Analyzed     1.371M
> 
>   Analysis Took: 128 sec.  Average: 10707 Lines/sec
> 
>    Unique Hosts: 626
> 
>      In Traffic:    4.208 GB
>     Out Traffic:    5.094 GB
>   ------------------------------------
>   Saved Traffic:  907.252 MB   17.39 %
> 
> 
> ~~~ Requests By Hours 
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> 
> 132.1K
> ==
> 125.5K
> ==
> 118.9K
> ==
> 112.3K
> ==
> 105.7K                                                        ==
> ==
>  99.1K                                                        ==
> ==
>  92.5K                                                        ==
> ==
>  85.9K                                                        ==
> ==
>  79.3K                                                        ==    ==
> ==
>  72.7K                                                        == == ==
> == ==
>  66.1K                                                        == == ==
> == ==
>  59.5K                                         ==             == == ==
> == ==
>  52.9K                                         ==             == == ==
> == ==
>  46.2K                                         ==          == == == ==
> == ==
>  39.6K                                         == == == == == == == ==
> == ==
>  33.0K     ==                            == == == == == == == == == ==
> == ==
>  26.4K  == ==                      == == == == == == == == == == == ==
> == ==
>  19.8K  == == ==                   == == == == == == == == == == == ==
> == ==
>  13.2K  == == == ==                == == == == == == == == == == == ==
> == ==
>   6.6K  == == == == ==          == == == == == == == == == == == == ==
> == == 
> ______________________________________________________________________
> __
> _______
>   Hours: 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20
> 21 22 23
> 
> Maximum Requests: 132.150K/hour,    2.202K/min,   36.7/sec.
> Average Requests:  40.970K/hour,      683 /min,   11.4/sec.
> Minimum Requests:       0 /hour,        0 /min,    0.0/sec.
> 
> 
> ~~~ Traffic by Hours 
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> 
> 560.6MB                                                       ==
> 532.6MB                                                       ==
> 504.5MB                                                       ==
> 476.5MB                                                       ==
> 448.5MB                                                       ==
> 420.4MB                                                       ==
> 392.4MB                                                       ==
> 364.4MB                                                       ==
> ==
> 336.4MB                                                       ==
> ==
> 308.3MB                                                       ==    ==
> ==
> 280.3MB                                                       == == ==
> ==
> 252.3MB                                                       == == ==
> ==
> 224.2MB                                                       == == ==
> == ==
> 196.2MB                                              ==    == == == ==
> == ==
> 168.2MB                                              ==    == == == ==
> == ==
> 140.1MB                                  ==    == == == == == == == ==
> == ==
> 112.1MB                            == == == == == == == == == == == ==
> == ==
>  84.1MB    ==                      == == == == == == == == == == == ==
> == ==
>  56.1MB == == ==                   == == == == == == == == == == == ==
> == ==
>  28.0MB == == == == == ==          == == == == == == == == == == == ==
> == == 
> ______________________________________________________________________
> __
> _______
>   Hours: 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20
> 21 22 23
> 
> Maximum Xfers: 560.591 MB/hour,    9.343 MB/min, 159.457 KB/sec [1276
> kbps]
> Average Xfers: 148.851 MB/hour,    2.481 MB/min,  42.340 KB/sec [ 339
> kbps]
> Minimum Xfers:       0  B/hour,        0  B/min,       0  B/sec [   0
> kbps]
> 
> 
> 
> ~~~ File Extensions Report 
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> 
> -------|------ R E Q U E S T S  --------|----------- T R A F F I C
> -----------|
>  Ext.  |  total | misses |  hits  |hit% |   total  |  misses  |   hits
> |hit%|
> -------|--------|--------|--------|-----|----------|----------|-------
> -------|--------|--------|--------|-----|----------|----------|--
> -|----|
> gif     417.258K 118.583K 298.675K  72%  629.652 MB 303.157 MB 326.495
> MB  52%
> <QUERY> 355.245K 354.751K     494    0%    1.054 GB   1.053 GB   1.112
> MB   0%
> jpg     109.525K  61.756K  47.769K  44%  723.888 MB 508.744 MB 215.144
> MB  30%
> /        47.375K  36.040K  11.335K  24%  814.252 MB 727.019 MB  87.233
> MB  11%
> [other]  19.944K  16.722K   3.222K  16%  880.673 MB 805.038 MB  75.635
> MB   9%
> js       19.202K   6.194K  13.008K  68%   38.306 MB  15.891 MB  22.416
> MB  59%
> css      19.007K   4.863K  14.144K  74%   23.750 MB   9.125 MB  14.624
> MB  62%
> html     17.821K  13.700K   4.121K  23%  130.971 MB 115.499 MB  15.472
> MB  12%
> htm      14.731K   7.480K   7.251K  49%   68.843 MB  48.772 MB  20.071
> MB  29%
> php      13.787K  13.289K     498    4%   37.826 MB  31.791 MB   6.035
> MB  16%
> pl        9.636K   9.636K       0    0%   31.349 MB  31.349 MB       0
> B   0%
> exe       9.123K   8.948K     175    2%  295.223 MB 284.060 MB  11.163
> MB   4%
> swf       5.011K   1.722K   3.289K  66%  102.082 MB  43.100 MB  58.982
> MB  58%
> zip       4.088K   3.997K      91    2%  200.740 MB 196.248 MB   4.492
> MB   2%
> <LONG.>   3.191K   3.037K     154    5%    4.651 MB   4.553 MB 100.545
> KB   2%
> asp       3.094K   3.092K       2    0%   24.827 MB  24.740 MB  89.520
> KB   0%
> shtml     2.408K   1.111K   1.297K  54%   32.453 MB  23.902 MB   8.550
> MB  26%
> dll       2.235K   2.200K      35    2%    1.879 MB   1.058 MB 840.496
> KB  44%
> png       2.145K     770    1.375K  64%    6.044 MB   2.965 MB   3.079
> MB  51%
> cab       1.839K   1.504K     335   18%   69.790 MB  34.048 MB  35.742
> MB  51% 
> ================|========|========|=====|==========|==========|=======
> ==
> =|====|
> TOTALS:   1.077M 669.395K 407.270K  38%    5.075 GB   4.189 GB 907.252
> MB  17%
> 
> 
> -----------------------|-- R E Q U E S T S ---|-------- T R A F F I C
> --------|
>  Content Type          |  total |  hits  |hit%|  total   |   hits
> |hit%|%VOL|
> -----------------------|--------|--------|----|----------|----------|-
> -----------------------|--------|--------|----|----------|----------|-
> -----------------------|--------|--------|----|----------|----------|-
> -|----|
> Images/Graphics & Flash 533.939K 351.108K  66%   1.427 GB 603.700 MB
> 41%  28%
> <QUERY>                 355.245K     494    0%   1.054 GB   1.112 MB
> 0%  21%
> Web Pages                80.484K  27.313K  34% 350.018 MB  64.840 MB
> 19%   7%
> / -any content possible  47.375K  11.335K  24% 814.252 MB  87.233 MB 
> 11%  16%
> [other]                  25.370K   3.411K  13% 887.203 MB  76.554 MB
> 9%  17%
> Java & VBS Scripts       19.202K  13.008K  68%  38.306 MB  22.416 MB
> 59%   1%
> Archives & Executables   13.211K     266    2% 495.962 MB  15.655 MB
> 3%  10%
> Software Updtaes          1.839K     335   18%  69.790 MB  35.742 MB
> 51%   1%
> ======================================================================
> ==
> ======
> 
> 
> ~~~ Objects Size Report 
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> 
> ---------|------ R E Q U E S T S  -------|----------- T R A F F I C
> -----------
>   SIZE   |  total | misses |  hits  |hit%|   total  |  misses  |
hits
> |hit%
> ---------|--------|--------|--------|----|----------|----------|------
> ---------|--------|--------|--------|----|----------|----------|--
> --|----
>   0-0.1KB  17.580K  16.998K     582    3%  25.126 KB  10.641 KB
14.485
> KB  58%
> 0.1-1.0KB 636.376K 337.853K 298.523K  47% 255.872 MB 163.672 MB  
> 92.200 MB  36%
>   1-5  KB 260.617K 193.254K  67.363K  26% 591.831 MB 431.422 MB 
> 160.409 MB  27%
>   5-10 KB  73.782K  52.249K  21.533K  29% 516.538 MB 368.189 MB 
> 148.349 MB  29%
>  10-50 KB  78.482K  60.374K  18.108K  23%   1.496 GB   1.183 GB
321.196
> MB  21%
>  50-100KB   6.629K   5.788K     841   13% 441.292 MB 388.636 MB
52.656
> MB  12%
> 100-500KB   2.570K   2.293K     277   11% 472.769 MB 414.595 MB
58.174
> MB  12%
> 0.5-1.0MB     341      320       21    6% 239.358 MB 226.247 MB
13.111
> MB   5%
>   1-5  MB     268      246       22    8% 529.863 MB 468.722 MB
61.141
> MB  12%
>   5-10 MB      13       13        0    0%  85.475 MB  85.475 MB
0
> B   0%
>  10-50 MB       5        5        0    0%  90.252 MB  90.252 MB
0
> B   0%
>  50-100MB       0        0        0    0%       0  B       0  B
0
> B   0%
>    >100MB       2        2        0    0% 440.657 MB 440.657 MB
0
> B   0%
> ======================================================================
> ==
> =======
> 
> 
> ~~~ Squid Result Codes 
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> 
> |      Result Code       | Requests  |  Traffic   |
> |------------------------|-----------|------------|
>              TCP_IMS_HIT   190.349K      60.396 MB
>                  TCP_HIT    95.187K     535.010 MB
>          TCP_REFRESH_HIT    68.172K     223.466 MB
>              TCP_MEM_HIT    51.147K      85.047 MB
>         TCP_NEGATIVE_HIT     2.415K       3.332 MB
>              TOTAL HITS:   407.270K     907.252 MB
> 
> |------------------------|-----------|------------|
>                 TCP_MISS   651.604K       4.106 GB
>         TCP_REFRESH_MISS    10.033K      52.649 MB
>  TCP_CLIENT_REFRESH_MISS     6.595K      28.116 MB
>        TCP_SWAPFAIL_MISS     1.163K       3.501 MB
> ==================================================
>            TOTAL MISSES:   669.395K       4.189 GB
> 
>           TOTAL HITS VOLUME: 37.83 %       17.46 %
> 
> |------------------------|-----------|------------|
>               TCP_DENIED       109       49.332 KB
>                  UDP_HIT    18.371K       1.180 MB
>                 UDP_MISS   275.321K      19.112 MB
>                     NONE        65       50.930 KB
>              UDP_INVALID        24       23.501 KB
> ==================================================
> 
> 
> ~~~ HTTP Status Codes 
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> 
>  ## | Description/Name        | Requests  |  Traffic   |  Group  |
> ----|-------------------------|-----------|------------|---------|
> 200  OK                          669.548K      3.775 GB  OK
> 000  UDP Traffic                 310.744K     20.315 MB  UDP
> 304  Not Modified                270.559K     59.994 MB  OK
> 302  Moved Temporarily            66.366K     36.985 MB  OTHER
> 206  Parital Content              29.051K    723.151 MB  OK
> 404  Not Found                    14.937K     29.027 MB  ERROR
> 504  Gataway Timeout               2.675K      1.197 MB  TIMEOUT
> 301  Moved Permanetly              1.993K      1.059 MB  OTHER
> 503  Service Unavailable           1.120K    528.900 KB  ERROR
> 403  Forbidden                     1.039K    873.904 KB  DENY
> 401  Unauthorised                    660     608.599 KB  DENY
> 502  Bad Gateway                     436       1.416 MB  ERROR
> 500  Internal Server Error           391       1.311 MB  ERROR
> 204  No Content                      290      81.331 KB  OTHER
> 207  Multi Status                    248       1.479 MB  OK
> 400  Bad Request                     166      86.908 KB  ERROR
> 600  Squid Header Parse Err.         145     472.820 MB  ERROR
> 416  Request Range Not Satif.         95      60.488 KB  ERROR
> 408  Request Timeout                  26       8.443 KB  TIMEOUT
> 303  See Other                        21      12.162 KB  OTHER
> 501  Not Implemented                  12      28.893 KB  OTHER
> 201  Created                          10       5.278 KB  OK
> 411  Length Requred                    8       4.065 KB  ERROR
> 413  Request Entity Too Large          8       3.703 KB  ERROR
> 300  Multiple Choices                  3       2.057 KB  OK
> 019                                    2       1.643 KB
> 560                                    1      14.365 KB
> 405  Method Not Allowed                1         756  B  ERROR
> ==============================|===========|============|========|
>                       TOTALS:      1.371M      5.094 GB
> 
> 
>  Group         | Requests     %Vol      |     Traffic     %Vol  |
> ---------------|------------------------|-----------------------|
> OK               969.419K    70.73%            4.541 GB  89.14%
> UDP              310.744K    22.67%           20.315 MB   0.39%
> OTHER             68.682K     5.01%           38.164 MB   0.73%
> ERROR             17.307K     1.26%          505.243 MB   9.69%
> TIMEOUT            2.701K     0.20%            1.206 MB   0.02%
> DENY               1.699K     0.12%            1.448 MB   0.03%
> ===============================================================
> 
> 
> ~~~ Request Methods 
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> 
>  Method        | Requests     %Vol      |    Traffic      %Vol  |
> ---------------|------------------------|-----------------------|
> GET                1.038M    75.76%            4.960 GB  97.36%
> ICP_QUERY        293.716K    21.43%           20.315 MB   0.39%
> POST              35.273K     2.57%          115.097 MB   2.21%
> HEAD               1.578K     0.12%          492.291 KB   0.01%
> PROPFIND           1.477K     0.11%            1.595 MB   0.03%
> CONNECT              100      0.01%           45.215 KB   0.00%
> OPTIONS               49      0.00%          122.443 KB   0.00%
> NONE                  31      0.00%           14.462 KB   0.00%
> PROPPATCH             30      0.00%           20.119 KB   0.00%
> MOVE                  10      0.00%            5.516 KB   0.00%
> BMOVE                  9      0.00%           10.204 KB   0.00%
> SUBSCRIBE              3      0.00%              912  B   0.00%
> ===============================================================
> 
> 
> ~~~ Protocols & Ports (Requests) 
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> 
>  Port | Protocol  | Total    %VOL | Misses Miss% |  Hits   Hit% |
Deny
> Deny%|
> ------|-----------|---------------|--------------|--------------|-----
> ------|-----------|---------------|--------------|--------------|--
> ------|
> [ALL]  HTTP          1.077M 99.99%  669.369K  62%  407.270K  38%
9
> 0%
> 6667   IRC              71   0.01%        0    0%        0    0%
71
> 100%
> [ALL]  FTP              36   0.00%       26   72%        0    0%
10
> 28%
> 25     SMTP             13   0.00%        0    0%        0    0%
13
> 100%
> 110    POP3              3   0.00%        0    0%        0    0%
3
> 100%
> 119                      2   0.00%        0    0%        0    0%
2
> 100%
> 1337                     1   0.00%        0    0%        0    0%
1
> 100% 
> ==================|===============|==============|==============|=====
> ==
> ======|
>           TOTALS:    1.077M   100%  669.395K  62%  407.270K  38%
109
> 0%
> 
> 
> ~~~ Protocols & Ports (Traffic) 
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> 
>  Port | Protocol  |   Total    %VOL | Misses   Miss%|  Hits    Hit% |
> Deny
> ------|-----------|-----------------|---------------|---------------|-
> ------|-----------|-----------------|---------------|---------------|-
> ------|-----------|-----------------|---------------|---------------|-
> -------
> [ALL]  HTTP          5.050 GB 99.51%   4.164 GB  82% 907.252 MB  18%
> 4.117 KB
> [ALL]  FTP          25.199 MB  0.48%  25.195 MB 100%       0  B   0%
> 4.529 KB
> 6667   IRC          32.050 KB  0.00%       0  B   0%       0  B   0%
> 32.050 KB
> 25     SMTP          5.940 KB  0.00%       0  B   0%       0  B   0%
> 5.940 KB
> 110    POP3          1.348 KB  0.00%       0  B   0%       0  B   0%
> 1.348 KB
> 119                    924  B  0.00%       0  B   0%       0  B   0%
> 924  B
> 1337                   456  B  0.00%       0  B   0%       0  B   0%
> 456  B 
> ==================|=================|===============|===============|=
> ==
> =======
>           TOTALS:    5.075 GB   100%   4.189 GB  83% 907.252 MB  17%
> 49.332 KB
> 
> 
> ~~~ Hierarchy (Requests) 
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> 
> Peer/Direct    |  Total   %VOL| Misses Miss% |  Hits   Hit% |  Deny
> |Timeouts|
>
---------------|--------------|--------------|--------------|--------|--
> ------|
> DIRECT          729.013K   68% 321.634K  44%  407.270K  56%      109
> 0
> 62.212.224.25   329.677K   31% 228.203K  69%  101.474K  31%        0
> 0
> 62.212.224.26    15.292K    1%   9.124K  60%    6.168K  40%        0
> 0
> 62.212.224.28     2.792K    0%       0    0%    2.792K 100%        0
> 0
> 
> 
> ~~~ Hierarchy (Traffic) 
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> 
> Peer/Direct    |  Total    %VOL| Misses    Miss% |    Hits    Hit%
> |Deny&T-out|
> ---------------|---------------|-----------------|-----------------|--
> ---------------|---------------|-----------------|-----------------|--
> ------|
> DIRECT            3.095 GB  61%   2.208 GB   71%   907.252 MB  29%
> 50.516K
> 62.212.224.26     1.023 GB  20% 544.206 MB   52%   503.569 MB  48%
> 0
> 62.212.224.25     0.952 GB  19% 672.922 MB   69%   301.722 MB  31%
> 0
> 62.212.224.28     5.355 MB   0%       0  B    0%     5.352 MB 100%
> 0
> 
> 
> ~~~ All Done. Enjoy Your Report.  /  SCALAR (C) 2003, 
> http://scalar.risk.az ~~~
> 
> 
> -----Original Message-----
> From: Yuri N. Fominov
> Sent: Thursday, September 11, 2003 21:07
> To: Squid-users
> Subject: [squid-users] Band New Squid Log Analyzer!
> Importance: High
> 
> 
> 
> Dear All,
> 
> I have developed SCALAR (Squid Cache Advanced Log Analyzer & 
> Reporter). It has many of very useful reports, which definitely will 
> help you to understand your statistics and tune Squid up for maximum 
> cachability and performance.
> 
> SCALAR is written under AWK.
> You can run it 'awk -f scalar.awk access.log' or make scalar.awk 
> executable and run 'scalar.awk access.log'. By editing scalar.awk 
> script you will be able to configure its reporting and other options.
> 
> You can download SCALAR from 
> http://www.risk.az/scalar/scalar091/scalar.awk.
> Try it. You feedbacks & suggestions will be highly appreciated.
> 
> Regards,
> 
> Yuri Fominov
> 
> 
>