| Author: | Shohei Tanaka(@xcir) |
|---|---|
| Date: | 2017-11-08 |
| Version: | 52.20 |
| Support Varnish Version: | 4.0.x, 4.1.x, 5.0.x, 5.1.x, 5.2.x |
| Manual section: | 1 |
See this link. https://github.com/xcir/varnishHostStat/tree/varnish30
[varnish-version].[library-version]
50.17 is v17 for Varnish5.0.x
Display to the Statistics for each domain or url-pattern
This script use to high cpu power. Is half cpu usage as compared with the varnishd in my environment.
2014-02-03 23:13:00 - 2014-02-03 23:13:05 (interval:5) Host | Mbps | rps | hit | time/req | (H)time/req | (M)time/req | KB/req | 2xx/s | 3xx/s | 4xx/s | 5xx/s | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| #alldata | 0.000031 | 0.800000 | 0.000000 | 0.001085 | 0.000000 | 0.001085 | 0.004883 | 0.800000 | 0.000000 | 0.000000 | 0.000000 | example.net | 0.000015 | 0.400000 | 0.000000 | 0.000933 | 0.000000 | 0.000933 | 0.004883 | 0.400000 | 0.000000 | 0.000000 | 0.000000 | hoge.example.net | 0.000015 | 0.400000 | 0.000000 | 0.001237 | 0.000000 | 0.001237 | 0.004883 | 0.400000 | 0.000000 | 0.000000 | 0.000000 |
Group by url-pattern (varnishhoststat.py -i 5 -F "example.net@^/img/" -F "example.net@^/css/" -F "example.net")
2014-02-03 23:14:19 - 2014-02-03 23:14:23 (interval:5) Host | Mbps | rps | hit | time/req | (H)time/req | (M)time/req | KB/req | 2xx/s | 3xx/s | 4xx/s | 5xx/s | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| #alldata | 0.001808 | 1.800000 | 44.444444 | 0.000583 | 0.000119 | 0.000954 | 0.128581 | 1.000000 | 0.000000 | 0.800000 | 0.000000 | [F1]example.net@^/img/ | 0.000885 | 0.400000 | 100.000000 | 0.000110 | 0.000110 | 0.000000 | 0.283203 | 0.000000 | 0.000000 | 0.400000 | 0.000000 | [F2]example.net@^/css/ | 0.000885 | 0.400000 | 100.000000 | 0.000127 | 0.000127 | 0.000000 | 0.283203 | 0.000000 | 0.000000 | 0.400000 | 0.000000 | [F3]example.net | 0.000038 | 1.000000 | 0.000000 | 0.000954 | 0.000000 | 0.000954 | 0.004883 | 1.000000 | 0.000000 | 0.000000 | 0.000000 |
2014-02-03 23:26:48 - 2014-02-03 23:26:57 (interval:10) Host | req | fetch | fetch_time | no_fetch_time | totallen | 2xx | 3xx | 4xx | 5xx | -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| #alldata | 23 | 14 | 0.012728 | 0.000753 | 2680 | 14 | 0 | 9 | 0 | example.net | 18 | 9 | 0.008177 | 0.000753 | 2655 | 9 | 0 | 9 | 0 | hoge.example.net | 5 | 5 | 0.004551 | 0.000000 | 25 | 5 | 0 | 0 | 0 |
{"hoge.example.net": {"avg_fsize": 0.0048828125, "rps": 0.5, "avg_not_fetch_time": 0.0, "hit": 0.0, "avg_fetch_time": 0.00086789131164550777, "fetch_time": 0.0043394565582275391, "2xx": 5, "avg_2xx": 0.5, "mbps": 1.9073486328125e-05, "req": 5, "5xx": 0, "avg_3xx": 0.0, "no_fetch_time": 0, "totallen": 25, "4xx": 0, "3xx": 0, "avg_time": 0.00086789131164550777, "avg_5xx": 0.0, "fetch": 5, "avg_4xx": 0.0}, "#alldata": {"avg_fsize": 0.1162109375, "rps": 2.5, "avg_not_fetch_time": 8.5520744323730466e-05, "hit": 40.0, "avg_fetch_time": 0.0009458700815836589, "fetch_time": 0.014188051223754883, "2xx": 15, "avg_2xx": 1.5, "mbps": 0.002269744873046875, "req": 25, "5xx": 0, "avg_3xx": 0.0, "no_fetch_time": 0.00085520744323730469, "totallen": 2975, "4xx": 10, "3xx": 0, "avg_time": 0.00060173034667968753, "avg_5xx": 0.0, "fetch": 15, "avg_4xx": 1.0}, "example.net": {"avg_fsize": 0.14404296875, "rps": 2.0, "avg_not_fetch_time": 8.5520744323730466e-05, "hit": 50.0, "avg_fetch_time": 0.00098485946655273442, "fetch_time": 0.0098485946655273438, "2xx": 10, "avg_2xx": 1.0, "mbps": 0.00225067138671875, "req": 20, "5xx": 0, "avg_3xx": 0.0, "no_fetch_time": 0.00085520744323730469, "totallen": 2950, "4xx": 10, "3xx": 0, "avg_time": 0.00053519010543823242, "avg_5xx": 0.0, "fetch": 10, "avg_4xx": 1.0}, "@start-time": 1391437481, "@end-time": 1391437490, "@info": ""}
{"hoge.example.net": {"avg_fsize": 0.0048828125, "rps": 0.5, "avg_not_fetch_time": 0.0, "hit": 0.0, "avg_fetch_time": 0.00083451271057128902, "fetch_time": 0.0041725635528564453, "2xx": 5, "avg_2xx": 0.5, "mbps": 1.9073486328125e-05, "req": 5, "5xx": 0, "avg_3xx": 0.0, "no_fetch_time": 0, "totallen": 25, "4xx": 0, "3xx": 0, "avg_time": 0.00083451271057128902, "avg_5xx": 0.0, "fetch": 5, "avg_4xx": 0.0}, "#alldata": {"avg_fsize": 0.1162109375, "rps": 2.5, "avg_not_fetch_time": 8.2373619079589844e-05, "hit": 40.0, "avg_fetch_time": 0.00090791384379069009, "fetch_time": 0.013618707656860352, "2xx": 15, "avg_2xx": 1.5, "mbps": 0.002269744873046875, "req": 25, "5xx": 0, "avg_3xx": 0.0, "no_fetch_time": 0.00082373619079589844, "totallen": 2975, "4xx": 10, "3xx": 0, "avg_time": 0.00057769775390624999, "avg_5xx": 0.0, "fetch": 15, "avg_4xx": 1.0}, "example.net": {"avg_fsize": 0.14404296875, "rps": 2.0, "avg_not_fetch_time": 8.2373619079589844e-05, "hit": 50.0, "avg_fetch_time": 0.00094461441040039062, "fetch_time": 0.0094461441040039062, "2xx": 10, "avg_2xx": 1.0, "mbps": 0.00225067138671875, "req": 20, "5xx": 0, "avg_3xx": 0.0, "no_fetch_time": 0.00082373619079589844, "totallen": 2950, "4xx": 10, "3xx": 0, "avg_time": 0.00051349401473999023, "avg_5xx": 0.0, "fetch": 10, "avg_4xx": 1.0}, "@start-time": 1391437491, "@end-time": 1391437500, "@info": ""}
{"hoge.example.net": {"fetch_time": 0.0037126541137695312, "2xx": 4, "req": 4, "5xx": 0, "no_fetch_time": 0, "totallen": 20, "4xx": 0, "3xx": 0, "fetch": 4}, "#alldata": {"fetch_time": 0.01218414306640625, "2xx": 13, "req": 23, "5xx": 0, "no_fetch_time": 0.00090909004211425781, "totallen": 2965, "4xx": 10, "3xx": 0, "fetch": 13}, "example.net": {"fetch_time": 0.0084714889526367188, "2xx": 9, "req": 19, "5xx": 0, "no_fetch_time": 0.00090909004211425781, "totallen": 2945, "4xx": 10, "3xx": 0, "fetch": 9}, "@start-time": 1391437527, "@end-time": 1391437536, "@info": ""}
{"hoge.example.net": {"fetch_time": 0.0052282810211181641, "2xx": 5, "req": 5, "5xx": 0, "no_fetch_time": 0, "totallen": 25, "4xx": 0, "3xx": 0, "fetch": 5}, "#alldata": {"fetch_time": 0.013852119445800781, "2xx": 15, "req": 25, "5xx": 0, "no_fetch_time": 0.00098705291748046875, "totallen": 2975, "4xx": 10, "3xx": 0, "fetch": 15}, "example.net": {"fetch_time": 0.0086238384246826172, "2xx": 10, "req": 20, "5xx": 0, "no_fetch_time": 0.00098705291748046875, "totallen": 2950, "4xx": 10, "3xx": 0, "fetch": 10}, "@start-time": 1391437537, "@end-time": 1391437546, "@info": ""}
-Default [start-time] - [end-time] (interval:[interval time]) [error message] Host | Mbps | rps | hit | time/req | (H)time/req | (M)time/req | KB/req | 2xx/s | 3xx/s | 4xx/s | 5xx/s ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Host or filter rule | Traffic | Request Per Sec | Hit per | Average response time | Average response time by hit request | Average response time by miss request | average response size | HTTP status 2xx rate | HTTP status 3xx rate | HTTP status 4xx rate | HTTP status 5xx rate * hit rate Decision as follows: Fetch to backend = Miss Not fetch = Hit -Raw [start-time] - [end-time] (interval:[interval time]) [error message] Host | req | fetch | fetch_time | no_fetch_time | totallen | 2xx | 3xx | 4xx | 5xx | -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| Host or filter rule | Request count | Fetch count | Total time(fetch to backend) | Total time(not fetch) | Total transfer size | HTTP-status count(2xx) | HTTP-status count(3xx) | HTTP-status count(4xx) | HTTP-status count(5xx) |
-r -j -i [interval] -a -F [filter pattern] -R [replace pattern] -f [field name(default:host)] --status [status,status,...] --start [second] --sopath [libvarnishapi.so] -w [file-name] -D -n [instance-name] -P [pid-file] -V
Raw data(no summarize)
Output json format
Specify interval second. Default is 10 second.
#10 second -i 10
Daemonize.
Write the process's PID to the specified file.(require -D option)
-F [host@url-pattern]
Specify filter pattern. Statistics for each domain separately ,if you do not specified.
#Filter by example.net (ends-with match) #This pattern is match to a.example.net and b.example.net and example.net -F example.net #Filter by example.net^/img/[0-9] #This pattern is match to a.example.net/img/0 and b.example.net/img/1 and example.net/img/2 -F "example.net@^/img/[0-9]" #Filter by example.net^/img/[0-9] and other example.net -F "example.net@^/img/[0-9]" -F example.net #Bad pattern #Not match to example.net@^/img/[0-9] -F example.net -F "example.net@^/img/[0-9]"
Specify replace pattern for the host header.
#Replace from [.] to [-]. #This pattern is replace from a.example.net to a-example-net -R "\./-" #Using group (see: https://docs.python.org/2.7/library/re.html ) -R "(example|varnish-cache).*/\1"
Specify field name. Default field is "host" This option not change output field name(Host)
#./varnishhoststat.py -f user-agent 2015-12-29 01:23:38 - 2015-12-29 01:23:47 (interval:10) Host | Mbps | rps | hit | time/req | (H)time/req | (M)time/req | KB/req | 2xx/s | 3xx/s | 4xx/s | 5xx/s | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| #alldata | 0.001697 | 0.400000 | 0.000000 | 0.000032 | 0.000000 | 0.000032 | 0.542969 | 0.000000 | 0.000000 | 0.000000 | 0.400000 | Wget/1.15 (linux-gnu) | 0.000845 | 0.200000 | 0.000000 | 0.000034 | 0.000000 | 0.000034 | 0.541016 | 0.000000 | 0.000000 | 0.000000 | 0.200000 | curl/7.35.0 | 0.000851 | 0.200000 | 0.000000 | 0.000029 | 0.000000 | 0.000029 | 0.544922 | 0.000000 | 0.000000 | 0.000000 | 0.200000 |
Additional filter. Show to statistics for each domain and filtered statistics.
Specify write log file-name. Move log file ,if you want rotation. (Don't send HUP)
Show version info.
Append summarize status
#./varnishhoststat.py -status 200,404 2016-12-30 15:33:40 - 2016-12-30 15:33:49 (interval:10) Host | Mbps | rps | hit | time/req | (H)time/req | (M)time/req | KB/req | 2xx/s | 3xx/s | 4xx/s | 5xx/s | 200/s | 404/s | -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| #alldata | 0.030865 | 1.200000 | 0.000000 | 0.000052 | 0.000000 | 0.000052 | 3.292236 | 1.100000 | 0.000000 | 0.100000 | 0.000000 | 1.100000 | 0.100000 | 192.168.1.37:6081 | 0.030865 | 1.200000 | 0.000000 | 0.000052 | 0.000000 | 0.000052 | 3.292236 | 1.100000 | 0.000000 | 0.100000 | 0.000000 | 1.100000 | 0.100000 |
Fix starting time.
Specify the libvarnishapi.so path, if you non-standard path.
Select which named Varnishd instance to use in multi-instance set-ups. (See -n for varnishstat, etc)
Version 52.20: Support Varnish5.2.x. Enhance perfomance.
Version 50.19: Fix if it's not come request within the interval-time, get an error. Support --status option.
Version 50.18: Use Python3. Enhanced performance. add @info (output sample:Log overrun, Log abandoned)
Version 50.17: Update varnishapi(include 50.18)
Version 0.16-varnish40: Update varnishapi(include 40.16)
Version 0.15-varnish40: Update varnishapi(include 40.15)
Version 0.14-varnish40: Update varnishapi(include 40.14)
Version 0.13-varnish40: Initial support for Python3. Feedback is welcome.
Version 0.12-varnish40: Support -f, -R option
Version 0.11-varnish40: Fix Crash if log abandoned.
Version 0.10-varnish40: Support --sopath option.
Version 0.9-varnish40: Support change to Varnish4(In development. Don't use production)
Version 0.8-varnish30: Support -n option. thx mrmonkington[#1]
Version 0.7-varnish30: Reopen VSM ,if Varnish restarted.
Version 0.6-varnish30: Support -a option
Version 0.5-varnish30: Support -D -P option
Version 0.4-varnish30: Support -w option
Version 0.3-varnish30: Support --start option, Bugfix
Version 0.2-varnish30: Support -r -j option
Version 0.1-varnish30: First version