Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Apply newer service_discovery plugin helper methods #3362

Merged
merged 4 commits into from
May 10, 2021
Merged

Conversation

ashie
Copy link
Member

@ashie ashie commented May 7, 2021

Which issue(s) this PR fixes:
none

What this PR does / why we need it:
Apply a new helper method added at #3299 to out_forward.
This PR is revised version of #3300 which was closed unexpectedly.

Docs Changes:
Same with #3299

Release Note:
Same with #3299

@ashie
Copy link
Member Author

ashie commented May 7, 2021

I'll merge it after CI is completed.

@ashie
Copy link
Member Author

ashie commented May 7, 2021

I'll merge it after CI is completed.

Hmm, 2 tests are failed on all platforms 🤔

2021-05-07T07:25:01.5492123Z ===============================================================================
2021-05-07T07:25:01.5512385Z Failure: test: server is an abbreviation of static type of service_discovery(ForwardOutputTest)
2021-05-07T07:25:01.5516251Z /home/runner/work/fluentd/fluentd/test/plugin/test_out_forward.rb:282:in `block in <class:ForwardOutputTest>'
2021-05-07T07:25:01.5518316Z <1234> expected but was
2021-05-07T07:25:01.5519533Z <1235>
2021-05-07T07:25:01.5520310Z 
2021-05-07T07:25:01.5521282Z diff:
2021-05-07T07:25:01.5522222Z ? 1234
2021-05-07T07:25:01.5523154Z ?    5
2021-05-07T07:25:01.5584243Z ===============================================================================
2021-05-07T07:25:03.8035052Z ......F
2021-05-07T07:25:03.8035472Z ===============================================================================
2021-05-07T07:25:03.8036046Z Failure: test: when out_forward has @id(ForwardOutputTest):
2021-05-07T07:25:03.8041597Z   Exception raised:
2021-05-07T07:25:03.8042557Z   RR::Errors::DoubleNotFoundError(<On subject Fluent::Plugin,
2021-05-07T07:25:03.8043314Z   unexpected method invocation:
2021-05-07T07:25:03.8044075Z     new_sd("static", {:parent=>#<Fluent::Plugin::ForwardOutput:000000000329d8>})
2021-05-07T07:25:03.8044761Z   expected invocations:
2021-05-07T07:25:03.8045902Z   - new_sd(:static, {:parent=>anything})>)
2021-05-07T07:25:03.8047111Z   /home/runner/work/fluentd/fluentd/lib/fluent/plugin_helper/service_discovery/manager.rb:43:in `block in configure'
2021-05-07T07:25:03.8184718Z   /home/runner/work/fluentd/fluentd/lib/fluent/plugin_helper/service_discovery/manager.rb:36:in `each'
2021-05-07T07:25:03.8186338Z   /home/runner/work/fluentd/fluentd/lib/fluent/plugin_helper/service_discovery/manager.rb:36:in `configure'
2021-05-07T07:25:03.8187781Z   /home/runner/work/fluentd/fluentd/lib/fluent/plugin_helper/service_discovery.rb:107:in `service_discovery_create_manager'
2021-05-07T07:25:03.8189483Z   /home/runner/work/fluentd/fluentd/lib/fluent/plugin_helper/service_discovery.rb:78:in `service_discovery_configure'
2021-05-07T07:25:03.8190784Z   /home/runner/work/fluentd/fluentd/lib/fluent/plugin/out_forward.rb:230:in `configure'
2021-05-07T07:25:03.8191917Z   /home/runner/work/fluentd/fluentd/lib/fluent/test/driver/base_owner.rb:57:in `configure'
2021-05-07T07:25:03.8193485Z   /home/runner/work/fluentd/fluentd/test/plugin/test_out_forward.rb:1087:in `block (2 levels) in <class:ForwardOutputTest>'
2021-05-07T07:25:03.8195312Z   /opt/hostedtoolcache/Ruby/2.7.3/x64/lib/ruby/gems/2.7.0/gems/test-unit-3.4.1/lib/test/unit/assertions.rb:627:in `block in assert_nothing_raised'
2021-05-07T07:25:03.8197012Z   /opt/hostedtoolcache/Ruby/2.7.3/x64/lib/ruby/gems/2.7.0/gems/test-unit-3.4.1/lib/test/unit/assertions.rb:1633:in `_wrap_assertion'
2021-05-07T07:25:03.8198672Z   /opt/hostedtoolcache/Ruby/2.7.3/x64/lib/ruby/gems/2.7.0/gems/test-unit-3.4.1/lib/test/unit/assertions.rb:618:in `assert_nothing_raised'
2021-05-07T07:25:03.8200271Z   /home/runner/work/fluentd/fluentd/test/plugin/test_out_forward.rb:1086:in `block in <class:ForwardOutputTest>'
2021-05-07T07:25:03.8201774Z   /opt/hostedtoolcache/Ruby/2.7.3/x64/lib/ruby/gems/2.7.0/gems/test-unit-3.4.1/lib/test/unit/testcase.rb:832:in `run_test'
2021-05-07T07:25:03.8203340Z   /opt/hostedtoolcache/Ruby/2.7.3/x64/lib/ruby/gems/2.7.0/gems/test-unit-3.4.1/lib/test/unit/testcase.rb:530:in `block (2 levels) in run'
2021-05-07T07:25:03.8205099Z   /opt/hostedtoolcache/Ruby/2.7.3/x64/lib/ruby/gems/2.7.0/gems/test-unit-3.4.1/lib/test/unit/fixture.rb:276:in `block in create_fixtures_runner'
2021-05-07T07:25:03.8206675Z   /opt/hostedtoolcache/Ruby/2.7.3/x64/lib/ruby/gems/2.7.0/gems/test-unit-3.4.1/lib/test/unit/fixture.rb:276:in `block in create_fixtures_runner'
2021-05-07T07:25:03.8208467Z   /opt/hostedtoolcache/Ruby/2.7.3/x64/lib/ruby/gems/2.7.0/gems/test-unit-3.4.1/lib/test/unit/fixture.rb:276:in `block in create_fixtures_runner'
2021-05-07T07:25:03.8209955Z   /opt/hostedtoolcache/Ruby/2.7.3/x64/lib/ruby/gems/2.7.0/gems/test-unit-3.4.1/lib/test/unit/fixture.rb:257:in `run_fixture'
2021-05-07T07:25:03.8211374Z   /opt/hostedtoolcache/Ruby/2.7.3/x64/lib/ruby/gems/2.7.0/gems/test-unit-3.4.1/lib/test/unit/fixture.rb:292:in `run_setup'
2021-05-07T07:25:03.8212785Z   /opt/hostedtoolcache/Ruby/2.7.3/x64/lib/ruby/gems/2.7.0/gems/test-unit-3.4.1/lib/test/unit/testcase.rb:528:in `block in run'
2021-05-07T07:25:03.8214162Z   /opt/hostedtoolcache/Ruby/2.7.3/x64/lib/ruby/gems/2.7.0/gems/test-unit-3.4.1/lib/test/unit/testcase.rb:527:in `catch'
2021-05-07T07:25:03.8216699Z   /opt/hostedtoolcache/Ruby/2.7.3/x64/lib/ruby/gems/2.7.0/gems/test-unit-3.4.1/lib/test/unit/testcase.rb:527:in `run'
2021-05-07T07:25:03.8218133Z   /opt/hostedtoolcache/Ruby/2.7.3/x64/lib/ruby/gems/2.7.0/gems/test-unit-3.4.1/lib/test/unit/testsuite.rb:124:in `run_test'
2021-05-07T07:25:03.8219514Z   /opt/hostedtoolcache/Ruby/2.7.3/x64/lib/ruby/gems/2.7.0/gems/test-unit-3.4.1/lib/test/unit/testsuite.rb:53:in `run'
2021-05-07T07:25:03.8220914Z   /opt/hostedtoolcache/Ruby/2.7.3/x64/lib/ruby/gems/2.7.0/gems/test-unit-3.4.1/lib/test/unit/testsuite.rb:124:in `run_test'
2021-05-07T07:25:03.8222285Z   /opt/hostedtoolcache/Ruby/2.7.3/x64/lib/ruby/gems/2.7.0/gems/test-unit-3.4.1/lib/test/unit/testsuite.rb:53:in `run'
2021-05-07T07:25:03.8223801Z   /opt/hostedtoolcache/Ruby/2.7.3/x64/lib/ruby/gems/2.7.0/gems/test-unit-3.4.1/lib/test/unit/ui/testrunnermediator.rb:67:in `run_suite'
2021-05-07T07:25:03.8225603Z   /opt/hostedtoolcache/Ruby/2.7.3/x64/lib/ruby/gems/2.7.0/gems/test-unit-3.4.1/lib/test/unit/ui/testrunnermediator.rb:45:in `block (2 levels) in run'
2021-05-07T07:25:03.8227437Z   /opt/hostedtoolcache/Ruby/2.7.3/x64/lib/ruby/gems/2.7.0/gems/test-unit-3.4.1/lib/test/unit/ui/testrunnermediator.rb:102:in `with_listener'
2021-05-07T07:25:03.8229110Z   /opt/hostedtoolcache/Ruby/2.7.3/x64/lib/ruby/gems/2.7.0/gems/test-unit-3.4.1/lib/test/unit/ui/testrunnermediator.rb:41:in `block in run'
2021-05-07T07:25:03.8230721Z   /opt/hostedtoolcache/Ruby/2.7.3/x64/lib/ruby/gems/2.7.0/gems/test-unit-3.4.1/lib/test/unit/ui/testrunnermediator.rb:39:in `catch'
2021-05-07T07:25:03.8232424Z   /opt/hostedtoolcache/Ruby/2.7.3/x64/lib/ruby/gems/2.7.0/gems/test-unit-3.4.1/lib/test/unit/ui/testrunnermediator.rb:39:in `run'
2021-05-07T07:25:03.8233995Z   /opt/hostedtoolcache/Ruby/2.7.3/x64/lib/ruby/gems/2.7.0/gems/test-unit-3.4.1/lib/test/unit/ui/testrunner.rb:40:in `start_mediator'
2021-05-07T07:25:03.8235450Z   /opt/hostedtoolcache/Ruby/2.7.3/x64/lib/ruby/gems/2.7.0/gems/test-unit-3.4.1/lib/test/unit/ui/testrunner.rb:25:in `start'
2021-05-07T07:25:03.8248847Z   /opt/hostedtoolcache/Ruby/2.7.3/x64/lib/ruby/gems/2.7.0/gems/test-unit-3.4.1/lib/test/unit/ui/testrunnerutilities.rb:24:in `run'
2021-05-07T07:25:03.8250792Z   /opt/hostedtoolcache/Ruby/2.7.3/x64/lib/ruby/gems/2.7.0/gems/test-unit-3.4.1/lib/test/unit/autorunner.rb:446:in `block in run'
2021-05-07T07:25:03.8252447Z   /opt/hostedtoolcache/Ruby/2.7.3/x64/lib/ruby/gems/2.7.0/gems/test-unit-3.4.1/lib/test/unit/autorunner.rb:502:in `change_work_directory'
2021-05-07T07:25:03.8253941Z   /opt/hostedtoolcache/Ruby/2.7.3/x64/lib/ruby/gems/2.7.0/gems/test-unit-3.4.1/lib/test/unit/autorunner.rb:445:in `run'
2021-05-07T07:25:03.8255331Z   /opt/hostedtoolcache/Ruby/2.7.3/x64/lib/ruby/gems/2.7.0/gems/test-unit-3.4.1/lib/test/unit/autorunner.rb:66:in `run'
2021-05-07T07:25:03.8256863Z   /opt/hostedtoolcache/Ruby/2.7.3/x64/lib/ruby/gems/2.7.0/gems/test-unit-3.4.1/lib/test/unit.rb:518:in `block (2 levels) in <top (required)>'
2021-05-07T07:25:03.8259268Z /home/runner/work/fluentd/fluentd/test/plugin/test_out_forward.rb:1086:in `block in <class:ForwardOutputTest>'
2021-05-07T07:25:03.8259965Z      1083:       end
2021-05-07T07:25:03.8260277Z      1084:     }
2021-05-07T07:25:03.8260557Z      1085: 
2021-05-07T07:25:03.8260930Z   => 1086:     assert_nothing_raised do
2021-05-07T07:25:03.8261421Z      1087:       output.configure(CONFIG + %[
2021-05-07T07:25:03.8261910Z      1088:         @id unique_out_forward
2021-05-07T07:25:03.8262255Z      1089:       ])
2021-05-07T07:25:03.8262653Z ===============================================================================

ashie added 2 commits May 7, 2021 17:57
Plugin type is a string value, so the test should exepect a string
value while the previous out_forward code pass a symbol value as a
service_discovery plugin type by one-off code.

Signed-off-by: Takuro Ashie <[email protected]>
A test should ensure to have only one assertion because trailing
assertions aren't evalueated when an assertions is failed. It makes
debugging hard. Here is an example output of the test:

before:

  Failure: test: server is an abbreviation of static type of service_discovery(ForwardOutputTest)
  /home/aho/Projects/Fluentd/fluentd/test/plugin/test_out_forward.rb:282:in `block in <class:ForwardOutputTest>'
  <1234> expected but was
  <1235>

  diff:
  ? 1234
  ?    5

after:

  Failure: test: server is an abbreviation of static type of service_discovery(ForwardOutputTest)
  /home/aho/Projects/Fluentd/fluentd/test/plugin/test_out_forward.rb:281:in `block in <class:ForwardOutputTest>'
  <[{:host=>"127.0.0.1", :port=>1234}, {:host=>"127.0.0.1", :port=>1235}]> expected but was
  <[{:host=>"127.0.0.1", :port=>1235}, {:host=>"127.0.0.1", :port=>1234}]>

  diff:
  ? [{:host=>"127.0.0.1", :port=>1234}, {:host=>"127.0.0.1", :port=>1235}]
  ?                                 5                                  4

The later one is easy to understand that the array is inverted.

Signed-off-by: Takuro Ashie <[email protected]>
@ashie
Copy link
Member Author

ashie commented May 7, 2021

BTW although tests in fluentd tend to check arrays by separated assertions like this:

    assert_equal 2, d.instance.discovery_manager.services.size
    assert_equal '127.0.0.1', d.instance.discovery_manager.services[0].host
    assert_equal 1234, d.instance.discovery_manager.services[0].port
    assert_equal '127.0.0.1', d.instance.discovery_manager.services[1].host
    assert_equal 1235, d.instance.discovery_manager.services[1].port

I don't like such tests because it makes hard to debug.
A test should ensure to have only one assertion because trailing assertions aren't evaluated when an assertion is failed.
For above case, it can be replaced by the following code:

    assert_equal(
      [
        { host: '127.0.0.1', port: 1234 },
        { host: '127.0.0.1', port: 1235 },
      ],
      d.instance.discovery_manager.services.collect do |service|
        { host: service.host, port: service.port }
      end
    )

Here is an example output:

before:

Failure: test: server is an abbreviation of static type of service_discovery(ForwardOutputTest)
/home/aho/Projects/Fluentd/fluentd/test/plugin/test_out_forward.rb:282:in `block in <class:ForwardOutputTest>'
<1234> expected but was
<1235>

diff:
? 1234
?    5

after:

Failure: test: server is an abbreviation of static type of service_discovery(ForwardOutputTest)
/home/aho/Projects/Fluentd/fluentd/test/plugin/test_out_forward.rb:281:in `block in <class:ForwardOutputTest>'
<[{:host=>"127.0.0.1", :port=>1234}, {:host=>"127.0.0.1", :port=>1235}]> expected but was
<[{:host=>"127.0.0.1", :port=>1235}, {:host=>"127.0.0.1", :port=>1234}]>

diff:
? [{:host=>"127.0.0.1", :port=>1234}, {:host=>"127.0.0.1", :port=>1235}]
?                                 5                                  4

The later one is easy to understand that the array is inverted.

"server" directive should have higher priority than "service" in
"service_discovery" to keep backward compatibility.

Signed-off-by: Takuro Ashie <[email protected]>
@ashie
Copy link
Member Author

ashie commented May 10, 2021

tests passed (without a known unstable test).

@tagomoris
Copy link
Member

Oops, I missed the notification about the merge of #3299 and your follow-up comments.
@ashie I'm sorry and thank you for creating this change.

@ashie
Copy link
Member Author

ashie commented Jun 14, 2021

Don't worry 😃
BTW we also modified the document for it: fluent/fluentd-docs-gitbook#321
Please let me know if you find any problem.

@tagomoris
Copy link
Member

That looks perfect!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants