Capture unmatched lines #1421
Capture unmatched lines #1421
Conversation
…rue, all singular and multiline log lines that don't match the parser regex will be stored in an attribute called parse_fail, and eventually saved in the output destination.
|
Additional comments
|
| @@ -65,6 +65,8 @@ def initialize | |||
| config_param :read_lines_limit, :integer, default: 1000 | |||
| desc 'The interval of flushing the buffer for multiline format' | |||
| config_param :multiline_flush_interval, :time, default: nil | |||
| desc 'Enable the option to capture unmatched lines.' | |||
| config_param :enable_catch_all, :bool, default: false | |||
There was a problem hiding this comment.
emit_unmatched_lines or emit_parse_failed_lines is more better.
enable_catch_all is unclear name for me.
| if @enable_catch_all | ||
| record = {'parse_fail' => line} | ||
| record[@path_key] ||= tail_watcher.path unless @path_key.nil? | ||
| es.add(Time.now.to_i, record) |
There was a problem hiding this comment.
Use EventTime.now. Don't use Time directly for event time.
There was a problem hiding this comment.
We're using the Redhat version of td-agent which comes with Fluentd v0.12. Do you know when the 0.14 branch will be stable? If it's still quite long away, should I merge my branch into the 0.12 branch instead?
There was a problem hiding this comment.
If you want this feature in v0.12, you also need to send a patch to v0.12 branch.
|
Hi @repeatedly, I have implemented the tests for the new parameter, but the existing unit tests seem to fail randomly between the different ruby releases, and I've noticed when the CI runs again, those tests pass again. |
| if @emit_unmatched_lines | ||
| record = {'unmatched_line' => line} | ||
| record[@path_key] ||= tail_watcher.path unless @path_key.nil? | ||
| es.add(Time.now.to_i, record) |
| @@ -149,6 +149,29 @@ def test_emit(data) | |||
| assert_equal(1, d.emit_count) | |||
| end | |||
|
|
|||
| data(flat: config_element("", "", { "format" => "/^(?<message>test.*)/", "emit_unmatched_lines" => true }), | |||
There was a problem hiding this comment.
Seems same config_element. No need data?
…nd write tests for the new behaviour
jitran
force-pushed
the
emit-unmatched-lines
branch
from
f41943d to
d874db1
Compare
|
Thanks! |
|
Thanks for accepting the PR and merging it into the v.0.12 branch |
Essentially all unmatched lines will be placed inside the parse_fail attribute, i.e.
This will enable our analytics tools to pick up any pattern matching regexp issues.