Fluentd 4.1.1 is crashing on logging TLS errors

[konstantin-kornienko]

Describe the bug

Fluentd 4.1.1 thread exits sometime, trying to log TLS errors.
Fluentd 4.1.0 is working fine (just logs an error and continue working).

To Reproduce

looks like this behavior is introduced here: 53df5d9

Expected behavior

FluentD shouldn't crash on logging

Your Environment

- Fluentd version: 4.1.1
- Operating system: Ubuntu 18.0.4

Your Configuration

<source>
  @id forward
  @type forward
  port 24224
  <security>
    self_hostname xxx
    shared_key xxx
  </security>
  <transport tls>
    version TLSv1_2
    insecure false
    ca_path /etc/td-agent/ca.crt
    cert_path /etc/td-agent/server.crt
    private_key_path /etc/td-agent/server.key
    client_cert_auth true
  </transport>
</source>

Your Error Log

2021-07-29 12:24:01 +0000 [warn]: #1 [forward] thread exited by unexpected error plugin=Fluent::Plugin::ForwardInput title=:event_loop error_class=Errno::ENOTCONN error="Transport endpoint is not connected - getpeername(2)"
2021-07-29 12:24:01 +0000 [error]: #1 unexpected error error_class=Errno::ENOTCONN error="Transport endpoint is not connected - getpeername(2)"
  2021-07-29 12:24:01 +0000 [error]: #1 /opt/td-agent/lib/ruby/2.7.0/openssl/ssl.rb:239:in `peeraddr'
  2021-07-29 12:24:01 +0000 [error]: #1 /opt/td-agent/lib/ruby/2.7.0/openssl/ssl.rb:239:in `peeraddr'
  2021-07-29 12:24:01 +0000 [error]: #1 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.12.3/lib/fluent/plugin_helper/server.rb:713:in `rescue in try_tls_accept'
  2021-07-29 12:24:01 +0000 [error]: #1 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.12.3/lib/fluent/plugin_helper/server.rb:696:in `try_tls_accept'
  2021-07-29 12:24:01 +0000 [error]: #1 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.12.3/lib/fluent/plugin_helper/server.rb:726:in `on_connect'
  2021-07-29 12:24:01 +0000 [error]: #1 /opt/td-agent/lib/ruby/gems/2.7.0/gems/cool.io-1.7.1/lib/cool.io/server.rb:41:in `on_connection'
  2021-07-29 12:24:01 +0000 [error]: #1 /opt/td-agent/lib/ruby/gems/2.7.0/gems/cool.io-1.7.1/lib/cool.io/listener.rb:46:in `on_readable'
  2021-07-29 12:24:01 +0000 [error]: #1 /opt/td-agent/lib/ruby/gems/2.7.0/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88:in `run_once'
  2021-07-29 12:24:01 +0000 [error]: #1 /opt/td-agent/lib/ruby/gems/2.7.0/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88:in `run'
  2021-07-29 12:24:01 +0000 [error]: #1 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.12.3/lib/fluent/plugin_helper/event_loop.rb:93:in `block in start'
  2021-07-29 12:24:01 +0000 [error]: #1 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.12.3/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
2021-07-29 12:24:01 +0000 [error]: #1 unexpected error error_class=Errno::ENOTCONN error="Transport endpoint is not connected - getpeername(2)"
  2021-07-29 12:24:01 +0000 [error]: #1 suppressed same stacktrace

Additional context

It's AliCloud :( And here are a lot of issues with TLS.

[ashie]

refs: #3330 #3309