Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve flush_interval handling #1442

Merged
merged 3 commits into from
Feb 17, 2017
Merged

Conversation

repeatedly
Copy link
Member

This patch improves flush_interval parameter handing.

  • Set flush_mode interval when v0.12 configuration has flush_interval

This avoids a trouble for existing out_file and out_s3 configurations

  • Log warning message for flush_interval without flush_mode interval in <buffer>

Currenlty, flush_interval is ignored silently with following configuration. It is hard to debug for users.

<buffer time>
  timekey 3600
  flush_interval 30s
</buffer>

@repeatedly repeatedly requested a review from tagomoris January 31, 2017 09:33
@repeatedly
Copy link
Member Author

@tagomoris Could you check this patch?

@repeatedly repeatedly added enhancement Feature request or improve operations v0.14 labels Jan 31, 2017
@@ -310,6 +310,10 @@ def configure(conf)
log.warn "'flush_at_shutdown' is false, and buffer plugin '#{buf_type}' is not persistent buffer."
log.warn "your configuration will lose buffered data at shutdown. please confirm your configuration again."
end

if @flush_mode != :interval && buffer_conf.has_key?('flush_interval')
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks better to set :interval when flush_mode is :default, even when timekey is specified, here:
https://github.com/fluent/fluentd/blob/master/lib/fluent/plugin/output.rb#L292
We need to know flush_interval is set in <buffer> section or not before setting :interval... how about to add has_flush_interval before super as same with has_buffer_section?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And it's better to warn when both of flush_interval and flush_mode lazy(or immediate) are configured explicitly by users (these are exclusive parameters).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks better to set :interval when flush_mode is :default

https://github.com/fluent/fluentd/blob/master/lib/fluent/plugin/output.rb#L293

From above line, :default is changed to :lazy or :interval so :default is not appeared here, right?

how about to add has_flush_interval before super as same with has_buffer_section?

It seems good.

these are exclusive parameters

If so, should we raise an ConfigError when flush_interval and flush_mode lazy are set explictly?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mean to set @flush_mode = :interval at L293 when flush_interval is explicitly specified in configuration even if @chunk_key_time is true.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we doesn't take care about compatibility layer, it's better to raise configuration errors when both of flush_mode lazy and flush_interval are specified in configuration file.
But compatibility layer may specify these both if a plugin is TimeSlicedOutput and flush_interval exists in configuration. So we can't raise errors for such cases. warn is better.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

flush_interval is explicitly specified in configuration even if @chunk_key_time is true.

This means we should use :interval with following configuration?

<buffer>
  flush_interval 10s # use flush_mode interval
</buffer>

And warning for follwing configuration?

<buffer>
  flush_interval 10s
  flush_mode lazy
</buffer>

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1st example: No. It depends on default of plugins, and output.rb should warn about ignored flush_interval when plugins' default is not interval.

My first point is that we cannot know the :interval set in @flush_mode is from plugin default or configuraitons at L314 (because it can be set at L293). So we should check conf before super to show appropriate warning.

2nd example: No. IMO it's better to raise error for inconsistent configuration.

The last my comment was wrong, because Fluent::Compat::TimeSlicedOutput#configure will set flush_mode as interval if conf has flush_interval. So we can raise exception if both of flush_mode lazy and flush_interval ... are specified in configurations explicitly.

@tagomoris
Copy link
Member

@repeatedly any updates?

@repeatedly
Copy link
Member Author

Applied reviews.

@repeatedly
Copy link
Member Author

@tagomoris could you check again?

@tagomoris
Copy link
Member

LGTM

@repeatedly repeatedly merged commit 8f9a766 into master Feb 17, 2017
@repeatedly repeatedly deleted the improve-flush_interval-handling branch February 17, 2017 11:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Feature request or improve operations v0.14
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants