…

[jitran]

For multiline events, if the first line doesn't match the prescribed pattern, we should still store the unmatched lines in the line buffer so that plugins such as fluent-plugin-grok-parser can report the log event as a grok parse error, see: fluent/fluent-plugin-grok-parser#25

Example:
input:

abcd efgh
asdfadsasda

output:

{"message":"abcd efgh\nasdfadsasda","grokfailure":"No grok pattern matched","server":"tranj-fluentd-s3b","stack":"my-app-pdev-01","application":"my-app","log_type":"filter.test","time":"2017-01-11T04:19:56Z"}

[repeatedly]

in_tail tests are broken. It means this patch breaks multiline mode result.
So I'm not sure this is proper patch for multiline_grok.

[jitran]

i wonder if the wider fluentd community are interested in being able to capture unmatched multiline events? currently the only way is through the td-agent. In logstash it is stored in the output tagged with grokparsefailure

[jitran]

Hi @repeatedly, I've redone the implementation so that the multiline-logstash behaviour is turned on when enable_multiline_catch_all is set to true. I'm not sure why some of the tests are failing though. Are they existing test failures?

[repeatedly]

Another idea is adding emit feature for unmatched logs instead of logging here:

fluentd/lib/fluent/plugin/in_tail.rb

Line 390 in 48526e4

log.warn "got incomplete line before first line from #{tail_watcher.path}: #{line.inspect}"

[jitran]

I'm not sure if emitting the event from parse_multilines() is the right way to go about it, I rather have them handled in flush_buffer()

Here's another implementation of handling unmatched lines: jitran@86d188f

Essentially all unmatched lines will be placed inside the parse_fail attribute, i.e.

{"parse_fail":"unmatched line....rest of the line","server":"myserver","stack":"my-app-dev-01","application":"my-app","log_type":"apache.access","time":"2017-01-16T23:06:49Z"}

Please let me know which implementation you prefer. Thanks

[jitran]

Closing this PR in favour of this #1421