[WIP] v0.14 plugin APIs #562

tagomoris · 2015-03-16T13:30:57Z

Plugins and PluginSupport APIs for v0.14 plugin system.

repeatedly · 2015-07-22T13:07:46Z

example/in_status.conf

@@ -0,0 +1,10 @@
+<source>


in_status will be revmoed. so no need this configuration.

repeatedly · 2015-07-22T13:16:40Z

plugin_support should be plugin/support?
I'm not sure Ruby's manner.

@nurse What do you think?

repeatedly · 2015-07-22T13:27:27Z

lib/fluent/plugin/buffer.rb

+
+          @queue_length_limit = @buffer_config.queue_length_limit
+          if @queue_length_limit && @total_bytes_limit > @chunk_bytes_limit * @queue_length_limit
+            @total_bytes_limit = @chunk_bytes_limit * @queue_length_limit


log.warn is needed.

nurse · 2015-07-24T20:20:36Z

plugin_support should be plugin/support?
I'm not sure Ruby's manner.

Ruby doesn't have related manner, but on Rails "plugin_helper" or "helpers" seem related words.

cosmo0920 · 2015-08-25T14:25:04Z

lib/fluent/plugin/buffer.rb

+        end
+      end
+
+      def metadata(timekey: nil, tag: nil, key_value_pairs={})


typo? key_value_pairs should be keyword argument.

-def metadata(timekey: nil, tag: nil, key_value_pairs={}) +def metadata(timekey: nil, tag: nil, key_value_pairs: {})

* add `paths` to add optional load_path for each registory instance * add `require`s to load plugin classes in each tests

tagomoris · 2016-02-02T06:54:08Z

Now I'm rebasing on current master.

tagomoris · 2016-02-02T06:54:57Z

TODO:

syntax conversion w/ recent ruby style
add raise NotImplementedError in methods to be overridden in sub classes (currently commented out)
pass plugin logger to buffer plugins from buffered output plugins
add missing descriptions for plugin configurations

* and separate each output plugin classes into files * Fluent::** are preserved for compatibility of traditional plugins

* to reduce duplicated implementation in Input and Output

…tions instead of Actor of v11 Plugins should include using PluginSupport Mixin individually.

* and mark old TestDriver as obsolete

* not to specify default run_timeout & expected_emit_length * to miss to revise obsolete method/variable names

* DEFAULT_TYPE 'json' * storage.synchronize should be callable more than once from the same thread * to create transactional get-put operation * by using Monitor * non-saved data storage is needed * for consistent API between configurations which stores data and which doesn't store data * Storage::Memory is for configurations not to store data * Storage::Base #save_at_shutdown is for cases not to store data by plugin code * add #delete method to Storage to clean internal data before #save or so * create parent directories just before actual #save in Storage::JSON

…t fluentd

* Merging this branch is too early because of some bugs This reverts commit 0b9a79e.

* for plugins to call `Engine.emit` or `Fluent::Engine.emit`

* add method to return user-specified id (or nil)

…buffer API

* and APIs for flushing are not included in this change * to fix these APIs/hooks w/ implementations of BufferedOutput APIs

…plugins

tagomoris · 2016-02-02T11:47:35Z

Rebased.

…re timing

cosmo0920 · 2016-02-08T05:23:52Z

lib/fluent/plugin/in_http.rb

-        header['Content-length'] ||= body.bytesize
-        header['Content-type'] ||= 'text/plain'
+        if body
+          header['Content-length'] ||= body.bytesize


In RFC 7230, content length header is described as Content-Length.
Should we follow the RFC or keep as-is for backward compatibility?

tagomoris · 2016-02-08T10:43:34Z

I found some inconsistent changes after rebase, and now, I think that it's inappropriate to push this change as a huge pull-request.
So I'll create some different patches and pull-requests for diffs which should be contained to this.

Anyway, this change is good reference to re-create diffs. Please leave this as is for now.

coveralls · 2016-11-14T18:20:09Z

Changes Unknown when pulling 9c58f42 on v0.14 into * on master*.

tagomoris force-pushed the v0.14 branch from 913d7ac to b71a18f Compare April 23, 2015 04:57

tagomoris force-pushed the v0.14 branch 3 times, most recently from 4f62aa2 to edf37b2 Compare May 29, 2015 11:17

repeatedly added feature request *Deprecated Label* Use enhancement label in general v0.14 labels Jul 7, 2015

repeatedly reviewed Jul 22, 2015
View reviewed changes

example/in_status.conf

@@ -0,0 +1,10 @@

<source>

Copy link

Member

repeatedly Jul 22, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in_status will be revmoed. so no need this configuration.

repeatedly reviewed Jul 22, 2015
View reviewed changes

cosmo0920 reviewed Aug 25, 2015
View reviewed changes

Refactor to use Registry as Plugin dictionary, instead of original impl.

6a8b284

* add `paths` to add optional load_path for each registory instance * add `require`s to load plugin classes in each tests

tagomoris added 11 commits February 2, 2016 16:09

Plugin base class change: Fluent::** -> Fluent::Plugin::**

ba61202

* and separate each output plugin classes into files * Fluent::** are preserved for compatibility of traditional plugins

Add Fluent::Plugin::Base

249bd0e

* to reduce duplicated implementation in Input and Output

Add Fluent::PluginSupport to provide thread/timer/child_process opera…

31f56b2

…tions instead of Actor of v11 Plugins should include using PluginSupport Mixin individually.

Add new Test Driver with simplified API

29eeaa3

* and mark old TestDriver as obsolete

Re-implement in_exec w/ v0.14 API (tests: green)

f7e7c39

Fix bug

06bac36

* not to specify default run_timeout & expected_emit_length * to miss to revise obsolete method/variable names

fix not to use unreliable "sleep"

65d6356

fix tests w/ modern test-unit styles

542f4d1

fix bug not to count emit_times

a4e5f15

add Fluent::PluginSupport::Timer to provide timer helpers

5e38ba5

Re-implement in_status w/ v0.14 API (tests: green)

218aaef

tagomoris added 19 commits February 2, 2016 19:26

Add suspend option to restart incremental value after stop-and-star…

5be758b

…t fluentd

Revert "Fix bug to overwrite all subsections definition by subclass"

bf21a72

* Merging this branch is too early because of some bugs This reverts commit 0b9a79e.

Fix bug not to publish method writing? on connection handler

369f802

Fix to enforce to use router.emit instead of Engine.emit

8abdd4a

Add compatibility layer for output plugin

b206b33

* for plugins to call `Engine.emit` or `Fluent::Engine.emit`

fix to specify file/dir permissions

0d4bde5

Writing v0.14 Buffer API (not tested now)

5f7c9ec

Implementing new Buffer APIs, especially emitting data

6d01de6

Fix plugin_id to check duplication

08ebd9a

* add method to return user-specified id (or nil)

Re-fix file path encoding, metadata storing and many others w/ fixed …

8b9d76f

…buffer API

Fix v0.14 Buffer API and common implementations

71b18c9

Add initial implementations of MemoryBuffer/FileBuffer, but not tested

522aa3d

* and APIs for flushing are not included in this change * to fix these APIs/hooks w/ implementations of BufferedOutput APIs

add tests for reserved fields of sections

68b3758

first commit for new BufferedOutput plugin class (not tested at all)

e0c890c

Fix to use server cert or CA cert as far as possible

b39cab4

buffer chunk/total bytes limits are managed by buffer implementation …

478bab1

…plugins

fix not to leak metadata_list

e5e1bf4

implement buffered output plugin base classes with v0.14 API

51c1377

tagomoris force-pushed the v0.14 branch from edf37b2 to 51c1377 Compare February 2, 2016 11:41

fix require paths and syntax misusage

55d7f8a

tagomoris added 2 commits February 4, 2016 18:56

remove already obsoleted plugins and these tests

e33dd5c

move all plugin registries to fluent/plugin, and fix some APIs, requi…

9c58f42

…re timing

cosmo0920 reviewed Feb 8, 2016
View reviewed changes

tagomoris closed this Mar 8, 2016

tagomoris deleted the v0.14 branch March 8, 2016 00:05

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] v0.14 plugin APIs #562

[WIP] v0.14 plugin APIs #562

tagomoris commented Mar 16, 2015

repeatedly Jul 22, 2015

repeatedly commented Jul 22, 2015

repeatedly Jul 22, 2015

nurse commented Jul 24, 2015

cosmo0920 Aug 25, 2015

tagomoris commented Feb 2, 2016

tagomoris commented Feb 2, 2016

tagomoris commented Feb 2, 2016

cosmo0920 Feb 8, 2016

tagomoris commented Feb 8, 2016

coveralls commented Nov 14, 2016 •

edited

Loading

[WIP] v0.14 plugin APIs #562

[WIP] v0.14 plugin APIs #562

Conversation

tagomoris commented Mar 16, 2015

repeatedly Jul 22, 2015

Choose a reason for hiding this comment

repeatedly commented Jul 22, 2015

repeatedly Jul 22, 2015

Choose a reason for hiding this comment

nurse commented Jul 24, 2015

cosmo0920 Aug 25, 2015

Choose a reason for hiding this comment

tagomoris commented Feb 2, 2016

tagomoris commented Feb 2, 2016

tagomoris commented Feb 2, 2016

cosmo0920 Feb 8, 2016

Choose a reason for hiding this comment

tagomoris commented Feb 8, 2016

coveralls commented Nov 14, 2016 • edited Loading

coveralls commented Nov 14, 2016 •

edited

Loading