Create new out file plugins only for secondary #1154

ganmacs · 2016-08-10T08:47:21Z

Consider points

Plugin's name
This plugins's name is out_secondary_file now.
It may be too verbose.
Supporting params
I think supporting compress and append params.

Not for this PR tasks

Create a utility command that read dumped data

Close #873

This is new plugin only for <secondary> section.

tagomoris · 2016-08-18T07:00:57Z

lib/fluent/plugin/buffer.rb

@@ -55,7 +55,11 @@ class BufferChunkOverflowError < BufferError; end # A record size is larger than
      # if chunk size (or records) is 95% or more after #write, then that chunk will be enqueued
      config_param :chunk_full_threshold, :float, default: DEFAULT_CHUNK_FULL_THRESHOLD

-      Metadata = Struct.new(:timekey, :tag, :variables)
+      Metadata = Struct.new(:timekey, :tag, :variables) do
+        def nil?


Ruby's document says that "Only the object nil responds true to nil?".
This method breaks it. Change this method name to something else, like empty?.

I changed this method name empty?
https://github.com/fluent/fluentd/pull/1154/files#diff-7d794f77abc9222413b9120533cff327R59

tagomoris · 2016-08-26T05:31:33Z

test/plugin/test_out_secondary_file.rb

+      assert_equal 'out_file_test', d.instance.basename
+      assert_equal TMP_DIR, d.instance.directory
+      assert_equal :gzip, d.instance.compress
+    end


tagomoris · 2016-08-26T06:19:47Z

I added review comments.

* Add `directory` and `basename` config_pa rams(Remove `path` config_params) * Do not support a path that includes "*" * Add placeholder regex test * Do not use chunk_id as path name(pass default name) * Remove `gz` of compress type * Fix some typos

ganmacs · 2016-08-29T01:10:42Z

@tagomoris
I updated.

tagomoris · 2016-08-29T01:41:01Z

lib/fluent/plugin/out_secondary_file.rb

+                  ".gz"
+                end
+
+      test_path = generate_path(File.join(@directory, Time.now.strftime("%Y%m%d")))


test_path loses @basename.

Yes. But test_path is used for a test that directory is writable or not.
I think it is reasonable that test_path loses @basename.

Are there any positive reason not to use @basename for this test?
@basename might contain /... (users do everything possible).

@basename might contain /... (users do everything possible).

I didn't care about it.
I added a test for it.
And I use basename as test_path

eec9803

tagomoris · 2016-08-29T07:17:25Z

I found that Fluent::FileUtil.writable_p returns false if the directory of specified path doesn't exist.
It means configuration like directory /path/to/${tag}/%Y%m%d always raises configuration errors.
Hmm.

ganmacs · 2016-08-29T08:27:50Z

@tagomoris

I wrote a test that directory has placeholders ( https://github.com/fluent/fluentd/pull/1154/files#diff-39910dc690f56dd659cdccdca103e55aR417 ) and this test passed.

If configuration is like directory /path/to/${tag}/%Y%m%d, Fluent::FileUtil.writable_p check direcotry /path/to/ is writable or not (https://github.com/fluent/fluentd/blob/master/lib/fluent/compat/file_util.rb#L39).

I think it is good enough to check.

tagomoris · 2016-08-29T09:41:14Z

@ganmacs Ah, I misread the code... you're right.
LGTM.

tagomoris · 2016-09-01T06:27:03Z

Merged! It'll be released as v0.14.5. Thank you!

This explanation is missing since the beginning. * https://github.com/fluent/fluentd/blob/master/CHANGELOG.md#release-v0145---20160906 * fluent/fluentd#1154 Signed-off-by: Daijiro Fukuda <[email protected]>

tagomoris added feature request *Deprecated Label* Use enhancement label in general v0.14 labels Aug 10, 2016

ganmacs added 3 commits August 11, 2016 16:12

Add #nil? method to Metadata

f112cdd

Create out_seconary_file plugins

e8b318b

This is new plugin only for <secondary> section.

Fix failed test

717047b

ganmacs force-pushed the create-new-out_file-plugins-only-for-secondary branch 3 times, most recently from 769ae75 to b169908 Compare August 12, 2016 13:50

ganmacs added 3 commits August 12, 2016 23:05

update secondary_out_file

2325a80

Add conf that is out_secondary_file example

146ce2b

Support extract_placeholder in out_secondary_file's path

5283a77

ganmacs force-pushed the create-new-out_file-plugins-only-for-secondary branch 9 times, most recently from 7dd7218 to 486c0d8 Compare August 15, 2016 11:37

ganmacs changed the title ~~[WIP] Create new out file plugins only for secondary~~ Create new out file plugins only for secondary Aug 15, 2016

ganmacs force-pushed the create-new-out_file-plugins-only-for-secondary branch from 486c0d8 to a6a1d11 Compare August 17, 2016 01:14

tagomoris reviewed Aug 18, 2016
View reviewed changes

tagomoris reviewed Aug 26, 2016
View reviewed changes

ganmacs force-pushed the create-new-out_file-plugins-only-for-secondary branch from 8c7d4c7 to dcb294a Compare August 26, 2016 07:13

Update out_secondary_file

df0b7a9

* Add `directory` and `basename` config_pa rams(Remove `path` config_params) * Do not support a path that includes "*" * Add placeholder regex test * Do not use chunk_id as path name(pass default name) * Remove `gz` of compress type * Fix some typos

ganmacs force-pushed the create-new-out_file-plugins-only-for-secondary branch from dcb294a to a1f5a10 Compare August 26, 2016 07:45

Fix test's description and added test cases

cbc1f1f

ganmacs force-pushed the create-new-out_file-plugins-only-for-secondary branch from a1f5a10 to cbc1f1f Compare August 29, 2016 01:09

tagomoris reviewed Aug 29, 2016
View reviewed changes

Add test basename include / or not

eec9803

tagomoris merged commit 0dd02c8 into fluent:master Sep 1, 2016

ganmacs deleted the create-new-out_file-plugins-only-for-secondary branch November 28, 2019 00:06

daipom mentioned this pull request Mar 6, 2023

out_secondary_file: add missing option explanation fluent/fluentd-docs-gitbook#445

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create new out file plugins only for secondary #1154

Create new out file plugins only for secondary #1154

ganmacs commented Aug 10, 2016

tagomoris Aug 18, 2016

ganmacs Aug 18, 2016

tagomoris Aug 26, 2016

tagomoris commented Aug 26, 2016

ganmacs commented Aug 29, 2016

tagomoris Aug 29, 2016

ganmacs Aug 29, 2016

tagomoris Aug 29, 2016

ganmacs Aug 29, 2016

tagomoris commented Aug 29, 2016

ganmacs commented Aug 29, 2016

tagomoris commented Aug 29, 2016

tagomoris commented Sep 1, 2016

Create new out file plugins only for secondary #1154

Create new out file plugins only for secondary #1154

Conversation

ganmacs commented Aug 10, 2016

Consider points

Not for this PR tasks

tagomoris Aug 18, 2016

Choose a reason for hiding this comment

ganmacs Aug 18, 2016

Choose a reason for hiding this comment

tagomoris Aug 26, 2016

Choose a reason for hiding this comment

tagomoris commented Aug 26, 2016

ganmacs commented Aug 29, 2016

tagomoris Aug 29, 2016

Choose a reason for hiding this comment

ganmacs Aug 29, 2016

Choose a reason for hiding this comment

tagomoris Aug 29, 2016

Choose a reason for hiding this comment

ganmacs Aug 29, 2016

Choose a reason for hiding this comment

tagomoris commented Aug 29, 2016

ganmacs commented Aug 29, 2016

tagomoris commented Aug 29, 2016

tagomoris commented Sep 1, 2016