[INDY-1187] Implement log processor #573
Conversation
skhoroshavin
force-pushed
the
log-processor
branch
from
f3c8100 to
b362209
Compare
skhoroshavin
changed the title
Signed-off-by: Sergey Khoroshavin <[email protected]>
skhoroshavin
force-pushed
the
log-processor
branch
from
b362209 to
4e1c319
Compare
skhoroshavin
changed the title
scripts/process_logs.yml
Outdated
| filename: warnings_<node>.log | ||
| pattern: <timestamp> | <message> | ||
| merge_nodes: no | ||
| commits: |
There was a problem hiding this comment.
What does commits mean here?
There was a problem hiding this comment.
This is just an identifier for log sink. It's used as a target in rule log line in chains.
scripts/process_logs.yml
Outdated
| only_timestamped: no # Whether to discard non-timestamped lines | ||
|
|
||
|
|
||
| chains: |
There was a problem hiding this comment.
We need documentation and description how to use it and what each parameter means here.
scripts/process_logs.yml
Outdated
| monitoring: warnings | ||
| - log line: warnings | ||
|
|
||
| tag_commits: |
There was a problem hiding this comment.
It would be a good idea to have a list of prepared chain items for common operations such as ViewChange, CatchUp, Connection, etc.
It would be great to have these rules in a separate file so that we don't need to edit neither the script nor config if the format of some log entries changes.
There was a problem hiding this comment.
Yes, this is next in my TODO list
scripts/process_logs.yml
Outdated
| merge_nodes: no | ||
| commits: | ||
| filename: commits.log | ||
| pattern: <timestamp> | <node> | <instId> <seqNo> |
There was a problem hiding this comment.
How to use it? Is it some pre-defined variables?
There was a problem hiding this comment.
Some of them (timestamp, node, level, source, func, message) are indeed pre-defined, others (in this case instId and seqNo) are values of user defined attributes by tag rule
Signed-off-by: Sergey Khoroshavin <[email protected]>
Signed-off-by: Sergey Khoroshavin <[email protected]>
skhoroshavin
force-pushed
the
log-processor
branch
from
f5de2df to
e3b67b5
Compare
Signed-off-by: Sergey Khoroshavin <[email protected]>
Signed-off-by: Sergey Khoroshavin <[email protected]>
Signed-off-by: Sergey Khoroshavin <[email protected]>
Signed-off-by: Sergey Khoroshavin <[email protected]>
Signed-off-by: Sergey Khoroshavin <[email protected]>
Signed-off-by: Sergey Khoroshavin <[email protected]>
Signed-off-by: Sergey Khoroshavin <[email protected]>
Signed-off-by: Sergey Khoroshavin <[email protected]>
|
test this please |
Signed-off-by: Sergey Khoroshavin <[email protected]>
Signed-off-by: Sergey Khoroshavin <[email protected]>
scripts/process_logs/README.md
Outdated
| - `node_group`: regex group number that matches node identifier | ||
| - `only_timestamped`: whether to discard non-timestamped messages | ||
|
|
||
| ### logs |
There was a problem hiding this comment.
Maybe better to name it output_logs?
Or even better move logs, timelogs and counters into output section, so that we have only 3 top-level configs: input, output, matchers (chains)
There was a problem hiding this comment.
output section makes sense, thanks
Signed-off-by: Sergey Khoroshavin <[email protected]>
skhoroshavin
force-pushed
the
log-processor
branch
from
98d75dc to
9cffb43
Compare
Signed-off-by: Sergey Khoroshavin <[email protected]>
skhoroshavin
force-pushed
the
log-processor
branch
from
d694e6c to
76687f4
Compare
| filename: output.log | ||
| pattern: <timestamp> | <node> | <source> | <func> | <message> | ||
| merge_nodes: yes | ||
| state_trie: |
There was a problem hiding this comment.
@ashcherbakov this is a leftover from debugging session when I found incorrect state trie condition while analyzing logs from INDY-1180, will remove
Signed-off-by: Sergey Khoroshavin <[email protected]>
skhoroshavin
changed the title
|
Only suggestion - add more example/predefined configs |
|
@dsurnin main problem with examples is my lack of experience with plenum, hope to catch up this soon |
|
test this please |
2 similar comments
|
test this please |
|
test this please |
Signed-off-by: Sergey Khoroshavin [email protected]