-
Notifications
You must be signed in to change notification settings - Fork 1.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Improve flush_interval handling #1442
Conversation
@tagomoris Could you check this patch? |
lib/fluent/plugin/output.rb
Outdated
@@ -310,6 +310,10 @@ def configure(conf) | |||
log.warn "'flush_at_shutdown' is false, and buffer plugin '#{buf_type}' is not persistent buffer." | |||
log.warn "your configuration will lose buffered data at shutdown. please confirm your configuration again." | |||
end | |||
|
|||
if @flush_mode != :interval && buffer_conf.has_key?('flush_interval') |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It looks better to set :interval
when flush_mode
is :default
, even when timekey
is specified, here:
https://github.com/fluent/fluentd/blob/master/lib/fluent/plugin/output.rb#L292
We need to know flush_interval
is set in <buffer>
section or not before setting :interval
... how about to add has_flush_interval
before super
as same with has_buffer_section
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
And it's better to warn when both of flush_interval
and flush_mode lazy
(or immediate) are configured explicitly by users (these are exclusive parameters).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It looks better to set :interval when flush_mode is :default
https://github.com/fluent/fluentd/blob/master/lib/fluent/plugin/output.rb#L293
From above line, :default
is changed to :lazy
or :interval
so :default
is not appeared here, right?
how about to add has_flush_interval before super as same with has_buffer_section?
It seems good.
these are exclusive parameters
If so, should we raise an ConfigError when flush_interval
and flush_mode lazy
are set explictly?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I mean to set @flush_mode = :interval
at L293 when flush_interval
is explicitly specified in configuration even if @chunk_key_time
is true.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If we doesn't take care about compatibility layer, it's better to raise configuration errors when both of flush_mode lazy
and flush_interval
are specified in configuration file.
But compatibility layer may specify these both if a plugin is TimeSlicedOutput and flush_interval
exists in configuration. So we can't raise errors for such cases. warn
is better.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
flush_interval is explicitly specified in configuration even if @chunk_key_time is true.
This means we should use :interval
with following configuration?
<buffer>
flush_interval 10s # use flush_mode interval
</buffer>
And warning for follwing configuration?
<buffer>
flush_interval 10s
flush_mode lazy
</buffer>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
1st example: No. It depends on default of plugins, and output.rb
should warn about ignored flush_interval
when plugins' default is not interval
.
My first point is that we cannot know the :interval
set in @flush_mode
is from plugin default or configuraitons at L314 (because it can be set at L293). So we should check conf
before super
to show appropriate warning.
2nd example: No. IMO it's better to raise error for inconsistent configuration.
The last my comment was wrong, because Fluent::Compat::TimeSlicedOutput#configure
will set flush_mode
as interval
if conf
has flush_interval
. So we can raise exception if both of flush_mode lazy
and flush_interval ...
are specified in configurations explicitly.
@repeatedly any updates? |
Applied reviews. |
@tagomoris could you check again? |
LGTM |
This patch improves flush_interval parameter handing.
flush_mode interval
when v0.12 configuration hasflush_interval
This avoids a trouble for existing out_file and out_s3 configurations
flush_mode interval
in<buffer>
Currenlty,
flush_interval
is ignored silently with following configuration. It is hard to debug for users.