Add auth forward protocol #1136

tagomoris · 2016-08-03T06:02:47Z

Revised branch of #813.

tagomoris · 2016-08-23T08:21:52Z

CI is green now.

tagomoris · 2016-08-23T12:09:27Z

This error sometimes occurs :(

Program: C:\Ruby23\bin\ruby.exe
File: ../../../../ext/cool.io/../libev/ev_select.c, Line 133
Expression: ("libev: fd >= FD_SETSIZE passed to fd_set-based select backend", fd < FD_SETSIZE)

tagomoris · 2016-08-24T04:23:56Z

I completed the implementation, and tests about communication patterns below (I'll paste configurations later):

out_secure_forward -> (TLS terminator: nginx) -> in_forward with shared key and user authorization
out_forward -> (TLS proxy: nginx) -> in_secure_forward with shared key and user authorization

tagomoris · 2016-08-24T04:24:06Z

@repeatedly Could you review this change?

tagomoris · 2016-08-24T04:59:43Z

I found that adding deny_keepalive option changed the behavior which test code expected.
I'll fix it soon.

tagomoris · 2016-08-24T06:24:16Z

The configuration set for out_secure_forward -> ssl terminator (nginx) -> in_forward on localhost.
Certification files are in example/ of the repository of fluent-plugin-secure-forward.

# in_forward
<system>
  rpc_endpoint 0.0.0.0:24444
</system>

<source>
  @type forward
  port 24224
  bind 0.0.0.0
  <security>
    self_hostname input.testing.local
    shared_key    secure_communication_is_awesome
    user_auth     yes
    allow_anonymous_source no
    <user>
      username user1
      password yes_this_is_user1
    </user>
    <user>
      username user2
      password yes_this_is_really_user2
    </user>
    <user>
      username user3
      password noooooo_this_may_not_be_user3
    </user>
    <client>
      # host  127.0.0.1
      network 127.0.0.0/24
      shared_key using_different_key_makes_us_secure
      users user1,user2
    </client>
  </security>
</source>

<match {test,test2,test3,test4}>
  @type stdout
</match>

# out_secure_forward
<source>
  @type dummy
  tag test
  dummy {"message":"yay"}
</source>
<match test.**>
    @type secure_forward
    secure yes
    self_hostname client
    shared_key "using_different_key_makes_us_secure"
    ca_cert_path "#{Dir.pwd}/example/cacerts1/ca_cert.pem"
    enable_strict_verification yes
    <server>
      host localhost
      port 24284
      hostlabel "SecureForward CA"
      username user1
      password yes_this_is_user1
    </server>
    flush_interval 0
</match>

# nginx.conf
worker_processes auto;

error_log /Users/tagomoris/nginx.err;

events {
  worker_connections 64;
}

stream {
  upstream backend {
    server 127.0.0.1:24224;
  }
  server {
    listen 24284 ssl;
    proxy_pass backend;
    ssl_protocols       TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers         AES128-SHA:AES256-SHA:RC4-SHA:DES-CBC3-SHA:RC4-MD5;
    ssl_certificate     /Users/tagomoris/github/fluent-plugin-secure-forward/example/cacerts1/ca_cert.pem;
    ssl_certificate_key /Users/tagomoris/github/fluent-plugin-secure-forward/example/cacerts1/ca_key.pem;
    ssl_password_file   /Users/tagomoris/github/fluent-plugin-secure-forward/pass.txt;
    ssl_session_cache   shared:SSL:10m;
    ssl_session_timeout 10m;
  }
}

Logs:

$ bundle exec bin/fluentd -c example/in_forward_client.conf 
2016-08-24 11:43:53 +0900 [info]: reading config file path="example/in_forward_client.conf"
2016-08-24 11:43:53 +0900 [info]: starting fluentd-0.14.2
2016-08-24 11:43:53 +0900 [info]: spawn command to main: /Users/tagomoris/.rbenv/versions/2.3.1/bin/ruby -Eascii-8bit:ascii-8bit  -rbundler/setup bin/fluentd -c example/in_forward_client.conf --under-supervisor
2016-08-24 11:43:53 +0900 [info]: reading config file path="example/in_forward_client.conf"
2016-08-24 11:43:53 +0900 [info]: starting fluentd-0.14.2 without supervision
2016-08-24 11:43:53 +0900 [info]: gem 'fluentd' version '0.14.2'
2016-08-24 11:43:53 +0900 [info]: adding match pattern="{test,test2,test3,test4}" type="stdout"
2016-08-24 11:43:53 +0900 [info]: adding source type="forward"
2016-08-24 11:43:53 +0900 [info]: using configuration file: <ROOT>
  <system>
    rpc_endpoint "0.0.0.0:24444"
  </system>
  <source>
    @type forward
    port 24224
    bind "0.0.0.0"
    <security>
      self_hostname "input.testing.local"
      shared_key "secure_communication_is_awesome"
      user_auth yes
      allow_anonymous_source no
      <user>
        username "user1"
        password "yes_this_is_user1"
      </user>
      <user>
        username "user2"
        password "yes_this_is_really_user2"
      </user>
      <user>
        username "user3"
        password "noooooo_this_may_not_be_user3"
      </user>
      <client>
        network "127.0.0.0/24"
        shared_key "using_different_key_makes_us_secure"
        users user1,user2
      </client>
    </security>
  </source>
  <match {test,test2,test3,test4}>
    @type stdout
  </match>
</ROOT>
2016-08-24 11:43:53 +0900 [info]: listening fluent socket on 0.0.0.0:24224
2016-08-24 11:44:24 +0900 test: {"message":"yay"}
2016-08-24 11:44:25 +0900 test: {"message":"yay"}
2016-08-24 11:44:26 +0900 test: {"message":"yay"}
2016-08-24 11:44:27 +0900 test: {"message":"yay"}
2016-08-24 11:44:28 +0900 test: {"message":"yay"}
2016-08-24 11:44:29 +0900 test: {"message":"yay"}
2016-08-24 11:44:30 +0900 test: {"message":"yay"}
^C2016-08-24 11:44:31 +0900 [info]: Received graceful stop
2016-08-24 11:44:31 +0900 test: {"message":"yay"}
2016-08-24 11:44:32 +0900 [info]: shutting down fluentd
2016-08-24 11:44:32 +0900 [info]: preparing shutdown input plugin type=:forward plugin_id="object:3fc700d54fb0"
2016-08-24 11:44:32 +0900 [info]: preparing shutdown output plugin type=:stdout plugin_id="object:3fc700e41194"
2016-08-24 11:44:32 +0900 [info]: shutting down input plugin type=:forward plugin_id="object:3fc700d54fb0"
2016-08-24 11:44:32 +0900 [info]: shutting down output plugin type=:stdout plugin_id="object:3fc700e41194"
2016-08-24 11:44:32 +0900 [info]: closing input plugin type=:forward plugin_id="object:3fc700d54fb0"
2016-08-24 11:44:32 +0900 [info]: closing output plugin type=:stdout plugin_id="object:3fc700e41194"
2016-08-24 11:44:32 +0900 [info]: Worker 0 finished with status 0

tagomoris · 2016-08-24T06:33:23Z

The another configuration set for out_forward -> TLS terminator -> in_secure_forward.
Certificate files are from the repository of fluent-plugin-secure-forward too.

# out_forward
<source>
  @type dummy
  tag test
</source>

<match test>
  @type forward
  flush_interval 0
  <security>
    self_hostname output.testing.local
    shared_key    secure_communication_is_awesome
  </security>
  <server>
    host 127.0.0.1
    port 24224
    username user1
    password yes_this_is_user1
    # shared_key using_different_key_makes_us_secure
    shared_key hogeposxxx0
  </server>
</match>

# in_secure_forward
<source>
  @type secure_forward
  port 24284
  secure yes
  self_hostname server_a.local
  shared_key hogeposxxx0
  ca_cert_path        "#{Dir.pwd}/example/cacerts1/ca_cert.pem"
  ca_private_key_path "#{Dir.pwd}/example/cacerts1/ca_key.pem"
  ca_private_key_passphrase "my secret"
  allow_anonymous_source yes
  authentication yes
  <user>
    username "user1"
    password "yes_this_is_user1"
  </user>
</source>

<match test.**>
  @type stdout
</match>

# nginx.conf
worker_processes auto;

error_log /Users/tagomoris/nginx.err;

events {
  worker_connections 64;
}

stream {
  upstream backend {
    server 127.0.0.1:24284;
  }
  server {
    listen 24224;
    proxy_pass backend;
    proxy_ssl on;
  }
}

Logs:

$ bundle exec fluentd -c example/cert_copy_server_a.conf 
2016-08-24 12:11:46 +0900 [info]: reading config file path="example/cert_copy_server_a.conf"
2016-08-24 12:11:46 +0900 [info]: starting fluentd-0.14.2
2016-08-24 12:11:46 +0900 [info]: spawn command to main: /Users/tagomoris/.rbenv/versions/2.3.1/bin/ruby -Eascii-8bit:ascii-8bit  -rbundler/setup /Users/tagomoris/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/bin/fluentd -c example/cert_copy_server_a.conf --under-supervisor
2016-08-24 12:11:47 +0900 [info]: reading config file path="example/cert_copy_server_a.conf"
2016-08-24 12:11:47 +0900 [info]: starting fluentd-0.14.2 without supervision
2016-08-24 12:11:47 +0900 [info]: gem 'fluentd' version '0.14.2'
2016-08-24 12:11:47 +0900 [info]: gem 'fluent-plugin-secure-forward' version '0.4.3'
2016-08-24 12:11:47 +0900 [info]: adding match pattern="test.**" type="stdout"
2016-08-24 12:11:47 +0900 [info]: adding source type="secure_forward"
2016-08-24 12:11:47 +0900 [info]: using configuration file: <ROOT>
  <source>
    @type secure_forward
    port 24284
    secure yes
    self_hostname "server_a.local"
    shared_key xxxxxx
    ca_cert_path "/Users/tagomoris/github/fluent-plugin-secure-forward/example/cacerts1/ca_cert.pem"
    ca_private_key_path "/Users/tagomoris/github/fluent-plugin-secure-forward/example/cacerts1/ca_key.pem"
    ca_private_key_passphrase xxxxxx
    allow_anonymous_source yes
    authentication yes
    <user>
      username "user1"
      password xxxxxx
    </user>
  </source>
  <match test.**>
    @type stdout
  </match>
</ROOT>
2016-08-24 12:21:16 +0900 test: {"message":"dummy"}
2016-08-24 12:21:17 +0900 test: {"message":"dummy"}
2016-08-24 12:21:18 +0900 test: {"message":"dummy"}
2016-08-24 12:21:19 +0900 test: {"message":"dummy"}
2016-08-24 12:21:20 +0900 test: {"message":"dummy"}
2016-08-24 12:21:21 +0900 test: {"message":"dummy"}
2016-08-24 12:21:22 +0900 test: {"message":"dummy"}
2016-08-24 12:21:23 +0900 test: {"message":"dummy"}
2016-08-24 12:21:24 +0900 test: {"message":"dummy"}
2016-08-24 12:21:25 +0900 test: {"message":"dummy"}
2016-08-24 12:21:26 +0900 test: {"message":"dummy"}
2016-08-24 12:21:27 +0900 test: {"message":"dummy"}
2016-08-24 12:21:28 +0900 test: {"message":"dummy"}
2016-08-24 12:21:29 +0900 test: {"message":"dummy"}
2016-08-24 12:21:30 +0900 test: {"message":"dummy"}
2016-08-24 12:21:31 +0900 test: {"message":"dummy"}
2016-08-24 12:21:32 +0900 test: {"message":"dummy"}
2016-08-24 12:21:33 +0900 test: {"message":"dummy"}
2016-08-24 12:21:34 +0900 test: {"message":"dummy"}
2016-08-24 12:21:35 +0900 test: {"message":"dummy"}
2016-08-24 12:21:36 +0900 test: {"message":"dummy"}
2016-08-24 12:21:37 +0900 test: {"message":"dummy"}
2016-08-24 12:21:38 +0900 test: {"message":"dummy"}
2016-08-24 12:21:39 +0900 test: {"message":"dummy"}
2016-08-24 12:21:40 +0900 test: {"message":"dummy"}
^C2016-08-24 15:15:18 +0900 [info]: Received graceful stop
2016-08-24 15:15:19 +0900 [info]: shutting down fluentd
2016-08-24 15:15:19 +0900 [info]: preparing shutdown input plugin type=:secure_forward plugin_id="object:3fffa253db94"
2016-08-24 15:15:19 +0900 [info]: preparing shutdown output plugin type=:stdout plugin_id="object:3fffa25a4e0c"
2016-08-24 15:15:19 +0900 [info]: shutting down input plugin type=:secure_forward plugin_id="object:3fffa253db94"
2016-08-24 15:15:19 +0900 [warn]: super was not called in #shutdown: calling it forcedly plugin=Fluent::SecureForwardInput
2016-08-24 15:15:19 +0900 [info]: shutting down output plugin type=:stdout plugin_id="object:3fffa25a4e0c"
2016-08-24 15:15:19 +0900 [info]: closing input plugin type=:secure_forward plugin_id="object:3fffa253db94"
2016-08-24 15:15:19 +0900 [info]: closing output plugin type=:stdout plugin_id="object:3fffa25a4e0c"
2016-08-24 15:15:19 +0900 [info]: Worker 0 finished with status 0

tagomoris · 2016-08-26T04:19:53Z

@repeatedly ping?
I'll merge this for v0.14.4 after next release.

tagomoris · 2016-08-26T04:23:07Z

Mmm, is it better to release this as v0.14.3, and see whether we can think it as stable or not?

repeatedly · 2016-08-26T05:20:53Z

@tagomoris Okay, I will review it.
I think release v0.14.3 is better after merged Windows related patch.

repeatedly · 2016-08-26T05:23:32Z

lib/fluent/plugin/in_forward.rb

@@ -52,8 +54,77 @@ def initialize
    desc "The field name of the client's hostname."
    config_param :source_hostname_key, :string, default: nil

+    config_section :security, required: false, multi: false do


Off topic. Can we write desc for config_section?

I have no idea about it...

…r plugin helpers)

…n immediately This option is not supported in any implementations (including out_secure_forward), but it's needed to add it now to support that behavior in following versions.

in_forward was changed not to disconnect connections immediately in default (it's same with v0.12 in_forward behavior). In this commit, tests was also changed that expected behavior is ACK timeout, not disconnected.

It was originally for JSONL (json string per lines). But currently it isn't needed because Yajl parser can process it in #read_message method. I added a test case to confirm it.

tagomoris · 2016-09-02T04:47:51Z

Rebased on current master.
I'll merge this after CI green.

tagomoris · 2016-09-02T06:33:47Z

Done!

tagomoris changed the title ~~Add auth forward protocol~~ [WIP] Add auth forward protocol Aug 10, 2016

tagomoris added feature request *Deprecated Label* Use enhancement label in general v0.14 labels Aug 10, 2016

tagomoris force-pushed the add-auth-forward-protocol branch from f0a01fb to 0bfb348 Compare August 10, 2016 07:59

tagomoris changed the title ~~[WIP] Add auth forward protocol~~ Add auth forward protocol Aug 10, 2016

tagomoris force-pushed the add-auth-forward-protocol branch from 27ece74 to 71b6acb Compare August 18, 2016 10:29

tagomoris force-pushed the add-auth-forward-protocol branch from 295276e to 172a855 Compare August 23, 2016 08:21

tagomoris mentioned this pull request Aug 23, 2016

Add authentication feature to in_forward #813

Closed

tagomoris force-pushed the add-auth-forward-protocol branch from 172a855 to a26f3a7 Compare August 23, 2016 10:53

repeatedly reviewed Aug 26, 2016
View reviewed changes

tagomoris added 8 commits September 2, 2016 13:24

add configuration example for authentication/authorization

5f912b9

mixin for tests to support SocketManager (temporary hack before serve…

f125d40

…r plugin helpers)

implemented authentication/authorization in forward protocol

0ae5aae

add tests for authentication/authorization

46ea54a

add "deny_keepalive" option to tell clients to disconnect each sessio…

6f67707

…n immediately This option is not supported in any implementations (including out_secure_forward), but it's needed to add it now to support that behavior in following versions.

fix tests with forward plugin default behavior

36da142

in_forward was changed not to disconnect connections immediately in default (it's same with v0.12 in_forward behavior). In this commit, tests was also changed that expected behavior is ACK timeout, not disconnected.

removed unused kwarg "delimiter"

b8ab659

It was originally for JSONL (json string per lines). But currently it isn't needed because Yajl parser can process it in #read_message method. I added a test case to confirm it.

fix to add lost line in rebase

bed4aed

tagomoris force-pushed the add-auth-forward-protocol branch from ead3c77 to bed4aed Compare September 2, 2016 04:45

tagomoris merged commit 7061c1c into master Sep 2, 2016

tagomoris deleted the add-auth-forward-protocol branch September 2, 2016 06:33

t-osakada mentioned this pull request Sep 21, 2016

out_forward: fix undefined variables #1238

Merged

fujimotos mentioned this pull request May 25, 2018

Add a guide to set up password authentication for in/out_forward fluent/fluentd-docs#498

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add auth forward protocol #1136

Add auth forward protocol #1136

tagomoris commented Aug 3, 2016

tagomoris commented Aug 23, 2016

tagomoris commented Aug 23, 2016

tagomoris commented Aug 24, 2016

tagomoris commented Aug 24, 2016

tagomoris commented Aug 24, 2016

tagomoris commented Aug 24, 2016 •

edited

Loading

tagomoris commented Aug 24, 2016 •

edited

Loading

tagomoris commented Aug 26, 2016

tagomoris commented Aug 26, 2016

repeatedly commented Aug 26, 2016

repeatedly Aug 26, 2016

tagomoris Aug 26, 2016

tagomoris commented Sep 2, 2016

tagomoris commented Sep 2, 2016

Add auth forward protocol #1136

Add auth forward protocol #1136

Conversation

tagomoris commented Aug 3, 2016

tagomoris commented Aug 23, 2016

tagomoris commented Aug 23, 2016

tagomoris commented Aug 24, 2016

tagomoris commented Aug 24, 2016

tagomoris commented Aug 24, 2016

tagomoris commented Aug 24, 2016 • edited Loading

tagomoris commented Aug 24, 2016 • edited Loading

tagomoris commented Aug 26, 2016

tagomoris commented Aug 26, 2016

repeatedly commented Aug 26, 2016

repeatedly Aug 26, 2016

Choose a reason for hiding this comment

tagomoris Aug 26, 2016

Choose a reason for hiding this comment

tagomoris commented Sep 2, 2016

tagomoris commented Sep 2, 2016

tagomoris commented Aug 24, 2016 •

edited

Loading

tagomoris commented Aug 24, 2016 •

edited

Loading