This repository has been archived by the owner on Jul 21, 2023. It is now read-only.
forked from adamhadani/logtools
-
Notifications
You must be signed in to change notification settings - Fork 1
/
ChangeLog
105 lines (92 loc) · 5.71 KB
/
ChangeLog
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
version 0.8.4
Added the triggered-regexp script
Added the logtail utility and CLI
version 0.8.3
Added explicit error message when --format is missing to AccessLog log parser
Added uWSGIParser. Can now be used with logparse --parser uWSGIParser.
Field names (for key-based -f lookup): 'ip', 'timestamp', 'method',
'path', 'bytes', 'processing_time'.
Added unit-test for the uWSGIParser
Better exception handling for the logparse utility, now reports failure to match
used-specified --fields in more accurate manner, separate from failures in actual parsing of the logline.
version 0.8.2
Updated dependecy on sqlsoup (separated out of sqlalchemy). Thanks to
Robert Churchill for pointing it out.
Update SqlAlchemy dependency version to latest stable release
version 0.8.1
Fixes to logfilter to properly support --parser -based filtering
Updated documentation and examples in README
Cleanup and error checking to percentiles utility, fixed to work with osx
mktemp
version 0.8
Added the 'logfilter' utility with support for regexp/aho-corasick based matching. See README/documentation for examples
Added the 'percentiles' utility for quickly computing percentiles out of input data
Added unit-tests for the logfilter functionality
Added dependency on the acora string matching library (Implements Aho-Corasick in CPython)
Minor fixes to qps utility (formatting and parameter naming in README
example)
Updated README to be markdown based, cleaned up and edited
version 0.7.2
Added missing -d and -f parameters to logsample for enabling weighted reservoir sampling
Bugfix to geoip --filter default value triggering a filtering by default (regression introduced in in 0.7.1)
version 0.7.1
Small fix with geoip --filter default value
version 0.7
Added -q flag to urlparse (See README/documentation for examples)
Added -f flag to geoip, allowing one to filter inplace for a specific country (e.g -f 'United States')
Added --decode options to urlparse that allows one to url-decode (unquote) input stream
Added sumstat utility/API for generating some common summary statistics for a given histogram-like logfile. See README for examples
Added 'colsum' script utility that does an arithmetic summation over a column of data
Added stripping of leading whitespace from output of 'aggregate' script
Updated documentation, Added more examples for new utilities in README
Updated unit-tests to reflect additions/changes to tools
Unicode encoding in logplot line-plot using matplotlib backend (would crash with encoding problems in some cases)
Unicode encoding for the logparse, fixes some cases of errors, e.g when using JSONParser for a JSON that has non-string types (e.g integers)
Fixed bug in use of --ignore for logparse
Default value for --delimiter is now ' ' across commands (logjoin etc.)
version 0.6
Added --header flag to logparse, will prepend output with field names. useful e.g for dumping to CSV file
Added --parser and --ip-ua-fields options to filterbots, allows routing input through a parser for filtering bots
Updated logmerge to support indexed/named field-based merge-sorting when using --parser. See README for examples
Added Histogram plot type support to logplot (--type hist)
Added urlparse utility/API. See README for details
Added logjoin utility/API. See README for details
Added logtools.join_backends module with SQLAlchemy backend implementation (added sqlalchemy to install_requires)
version 0.5
Added qps utility - Parse timestamps from log stream and calculate qps over sliding time window
Added Prefix/Suffix matching to filterbots - Specify p'<prefix>' or s'<suffix>' in useragents file to use this feature
Added comment support for filterbots useragents/ips files - use '#' in start of line
Added JSONParser to parsing module, can now use JSON-formatted logs e.g for logmerge, logparse etc.
Added --ignore flag to logparse, surpresses missing field errors and skips those lines
Added the 'timeseries' plot type for plotting timeseries data
Added the matplotlib plotting backend supporting pie/line/timeseries charting
Added data/examples subdirectory, included example bots_useragents.txt files
Can now specify fields by field name in -f parameter for the logparse method/console script
Can now specify multiple comma-separated field names/indices in logparse method
Fixed memory bug resulting from un-needed use of readlines() on file descriptors
Changed programmatic interface for most methods to use explicit parameter naming instead of generic (opts, args)
Updated unit-tests, examples and README to reflect most of latest functionality
version 0.4
Added unit-testing for logtools.parsers module
Fixed small bug in GeoIP unit-testing
Fixed small bug in logplot option parsing from configuration file
Added logplot gchart backend 'line' chart type
Improved coverage of unit-testing for the plotting module
Added support for regular expressions in bots useragents file. use r'<reg_ex>' syntax to specify a regexp line
Added --format option for logparse, added set_format to Parser interface
Added --print option to geoip utility, will print the country along with original log line
version 0.3
Some bugfixes to command-line/options parsing
setup.py long_description documentation
Added logplot gchart backend chart type support (pie, histogram, line)
Changed most methods' programmatic interface to use generators (yield values)
Added 'aggregate' script
version 0.2
Added support for configuration profiles (-P <profile_name>)
Added logmerge utility and unittest
Added logplot utility with initial Google Chart API support and unittest (Experimental)
Added pygooglechart package dependency in setup.py
version 0.1
Added to PyPI Python Package Index
Initial check-in of project.
Version 0.1 Tag