Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor buffer code #3118

Merged
merged 3 commits into from
Sep 1, 2020
Merged

Refactor buffer code #3118

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
f554ac7
buffer: Skip timekeys calculation when no time chunk key
repeatedly Aug 31, 2020
9917626
buffer: Reduce object allocation in update_timekeys
repeatedly Aug 31, 2020
251b3d8
buffer: refactoring code
repeatedly Aug 31, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 15 additions & 7 deletions lib/fluent/plugin/buffer.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -192,6 +192,7 @@ def initialize

@stage_size = @queue_size = 0
@timekeys = Hash.new(0)
@enable_update_timekeys = false
@mutex = Mutex.new
end

Expand All @@ -207,6 +208,10 @@ def configure(conf)
end
end

def enable_update_timekeys
@enable_update_timekeys = true
end

def start
super

Expand Down Expand Up @@ -268,8 +273,9 @@ def new_metadata(timekey: nil, tag: nil, variables: nil)
Metadata.new(timekey, tag, variables)
end

# Keep this method for existing code
def metadata(timekey: nil, tag: nil, variables: nil)
meta = Metadata.new(timekey, tag, variables)
Metadata.new(timekey, tag, variables)
end

def timekeys
Expand Down Expand Up @@ -465,8 +471,10 @@ def enqueue_unstaged_chunk(chunk)

def update_timekeys
synchronize do
@timekeys = (@stage.values + @queue).each_with_object({}) do |chunk, keys|
if chunk && chunk.metadata && chunk.metadata.timekey
chunks = @stage.values
chunks.concat(@queue)
@timekeys = chunks.each_with_object({}) do |chunk, keys|
if chunk.metadata && chunk.metadata.timekey
t = chunk.metadata.timekey
keys[t] = keys.fetch(t, 0) + 1
end
Expand All @@ -477,7 +485,7 @@ def update_timekeys
# At flush_at_shutdown, all staged chunks should be enqueued for buffer flush. Set true to force_enqueue for it.
def enqueue_all(force_enqueue = false)
log.on_trace { log.trace "enqueueing all chunks in buffer", instance: self.object_id }
update_timekeys
update_timekeys if @enable_update_timekeys

if block_given?
synchronize{ @stage.keys }.each do |metadata|
Expand Down Expand Up @@ -787,11 +795,11 @@ def statistics
'total_queued_size' => stage_size + queue_size,
}

if (m = timekeys.min)
tkeys = timekeys
if (m = tkeys.min)
stats['oldest_timekey'] = m
end

if (m = timekeys.max)
if (m = tkeys.max)
stats['newest_timekey'] = m
end

Expand Down
1 change: 1 addition & 0 deletions lib/fluent/plugin/output.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -340,6 +340,7 @@ def configure(conf)
buffer_conf = conf.elements(name: 'buffer').first || Fluent::Config::Element.new('buffer', '', {}, [])
@buffer = Plugin.new_buffer(buffer_type, parent: self)
@buffer.configure(buffer_conf)
@buffer.enable_update_timekeys if @chunk_key_time

@flush_at_shutdown = @buffer_config.flush_at_shutdown
if @flush_at_shutdown.nil?
Expand Down