Skip to content

Commit

Permalink
in_sample: support to renew record functionality
Browse files Browse the repository at this point in the history
When in_sample plugin is used with filter parser which uses
remove_key_name_field, it raises the following error repeatedly.

  #0 dump an error event: error_class=ArgumentError error="message
  does not exist"

This kind of error occurs when key_name and remove_key_name_field
removes key from record with destructive change in filter parser.
It affects generated sample data.

To fix this issue, it is simple to just dup every record even though
it has a significant performance penalty.

Considering keeping compatibility and providing way to a workaround,
added option to enable previous behavior - reuse_record, disabled by default.

ref. #4575

Here is the small benchmark:

<source>
    @type sample
    tag test
    size xxx
    rate 100
    reuse_record
</source>
<match test>
  @type null
</match>

size: 100000

 master:

   PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
   330767 kenhys    20   0  364316  81036  13620 S 100.3   0.1   0:52.10 ruby

 reuse_record: true

   PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
   333640 kenhys    20   0  364328  80956  13560 S 100.0   0.1   0:17.04 ruby

 reuse_record: false

   PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
   335843 kenhys    20   0  366188 113300  13536 S 100.3   0.2   0:17.24 ruby

Signed-off-by: Kentaro Hayashi <[email protected]>
  • Loading branch information
kenhys committed Aug 16, 2024
1 parent a6978e5 commit e8441f1
Showing 1 changed file with 13 additions and 7 deletions.
20 changes: 13 additions & 7 deletions lib/fluent/plugin/in_sample.rb
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,8 @@ class SampleInput < Input
config_param :auto_increment_key, :string, default: nil
desc "The boolean to suspend-and-resume incremental value after restart"
config_param :suspend, :bool, default: false,deprecated: 'This parameters is ignored'
desc "Reuse the sample data to reduce the load when sending large amounts of data. You can enable it if filter does not do destructive change"
config_param :reuse_record, :bool, default: false
desc "The sample data to be generated. An array of JSON hashes or a single JSON hash."
config_param :sample, alias: :dummy, default: [{"message" => "sample"}] do |val|
begin
Expand Down Expand Up @@ -117,15 +119,19 @@ def emit(num)
end
end

def generate
d = @sample[@sample_index]
unless d
@sample_index = 0
d = @sample[@sample_index]
end
def next_sample
d = @reuse_record ? @sample[@sample_index] : @sample[@sample_index].dup
@sample_index += 1
return d if d

@sample_index = 0
next_sample
end

def generate
d = next_sample
if @auto_increment_key
d = d.dup
d = d.dup if @reuse_record
d[@auto_increment_key] = @storage.update(:auto_increment_value){|v| v + 1 }
end
d
Expand Down

0 comments on commit e8441f1

Please sign in to comment.