Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Capture unmatched lines #1421

Merged
merged 4 commits into from
Jan 25, 2017
Merged

Capture unmatched lines #1421

Changes from 2 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
49ff636
Add a new option for in_tail called 'enable_catch_all'. When set to t…
jitran Jan 16, 2017
aba4b73
Use Time.now.to_i as its backwards compatible with the earlier versio…
jitran Jan 17, 2017
eba4c61
Renamed enable_catch_all param as emit_unmatched_lines for clarity; a…
jitran Jan 20, 2017
d874db1
Deprecated Time.now.to_i in favor of Fluent::EventTime.now, and updat…
jitran Jan 25, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions lib/fluent/plugin/in_tail.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,8 @@ def initialize
config_param :read_lines_limit, :integer, default: 1000
desc 'The interval of flushing the buffer for multiline format'
config_param :multiline_flush_interval, :time, default: nil
desc 'Enable the option to capture unmatched lines.'
config_param :enable_catch_all, :bool, default: false
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

emit_unmatched_lines or emit_parse_failed_lines is more better.
enable_catch_all is unclear name for me.

desc 'Enable the additional watch timer.'
config_param :enable_watch_timer, :bool, default: true
desc 'The encoding after conversion of the input.'
Expand Down Expand Up @@ -357,6 +359,11 @@ def convert_line_to_event(line, es, tail_watcher)
record[@path_key] ||= tail_watcher.path unless @path_key.nil?
es.add(time, record)
else
if @enable_catch_all
record = {'parse_fail' => line}
record[@path_key] ||= tail_watcher.path unless @path_key.nil?
es.add(Time.now.to_i, record)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use EventTime.now. Don't use Time directly for event time.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We're using the Redhat version of td-agent which comes with Fluentd v0.12. Do you know when the 0.14 branch will be stable? If it's still quite long away, should I merge my branch into the 0.12 branch instead?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you want this feature in v0.12, you also need to send a patch to v0.12 branch.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still Time.now.to_i

end
log.warn "pattern not match: #{line.inspect}"
end
}
Expand Down Expand Up @@ -387,6 +394,9 @@ def parse_multilines(lines, tail_watcher)
lb = line
else
if lb.nil?
if @enable_catch_all
convert_line_to_event(line, es, tail_watcher)
end
log.warn "got incomplete line before first line from #{tail_watcher.path}: #{line.inspect}"
else
lb << line
Expand Down