ENOTCONN error="Transport endpoint is not connected - getpeername(2) in fluentd http input plugin

[pratikshavyas]

Describe the bug

We are using http input plugin and sending data to elasticsearch. This setup works fine but in between get below mentioned error logs:
2022-02-08 13:49:39 +0100 [error]: #2 unexpected error error_class=Errno::ENOTCONN error="Transport endpoint is not connected - getpeername(2)"
2022-02-08 13:49:39 +0100 [error]: #2 /opt/td-agent/lib/ruby/2.7.0/openssl/ssl.rb:239:in peeraddr' 2022-02-08 13:49:39 +0100 [error]: #2 /opt/td-agent/lib/ruby/2.7.0/openssl/ssl.rb:239:in peeraddr'
2022-02-08 13:49:39 +0100 [error]: #2 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin_helper/server.rb:718:in rescue in try_tls_accept' 2022-02-08 13:49:39 +0100 [error]: #2 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin_helper/server.rb:696:in try_tls_accept'
2022-02-08 13:49:39 +0100 [error]: #2 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin_helper/server.rb:730:in on_readable' 2022-02-08 13:49:39 +0100 [error]: #2 /opt/td-agent/lib/ruby/gems/2.7.0/gems/cool.io-1.7.1/lib/cool.io/io.rb:186:in on_readable'
2022-02-08 13:49:39 +0100 [error]: #2 /opt/td-agent/lib/ruby/gems/2.7.0/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88:in run_once' 2022-02-08 13:49:39 +0100 [error]: #2 /opt/td-agent/lib/ruby/gems/2.7.0/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88:in run'
2022-02-08 13:49:39 +0100 [error]: #2 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin_helper/event_loop.rb:93:in block in start' 2022-02-08 13:49:39 +0100 [error]: #2 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin_helper/thread.rb:78:in block in thread_create'
2022-02-08 13:49:39 +0100 [error]: #2 unexpected error error_class=Errno::ENOTCONN error="Transport endpoint is not connected - getpeername(2)"
2022-02-08 13:49:39 +0100 [error]: #2 suppressed same stacktrace

To Reproduce

Deploy fluentd with http as input plugin and elasticsearch as output.

Expected behavior

These errors should not appear in fluentd logs

Your Environment

- Fluentd version: 1.13.3
- TD Agent version: 4.2.0

Your Configuration

<system>
    workers 4
  </system>
  <source>
    @type http
    @log_level "error"
    @id input_http_ipv4
    port 9000
    bind "0.0.0.0"
    <transport tls>
      ca_path "/etc/certs/es-root-ca.pem"
      cert_path "/etc/certs/fluentd_cert.pem"
      private_key_path "/etc/certs/fluentd_key.pem"
    </transport>
  </source>

Your Error Log

#2 unexpected error error_class=Errno::ENOTCONN error="Transport endpoint is not connected - getpeername(2)"
  2022-02-08 13:49:39 +0100 [error]: #2 /opt/td-agent/lib/ruby/2.7.0/openssl/ssl.rb:239:in `peeraddr'
  2022-02-08 13:49:39 +0100 [error]: #2 /opt/td-agent/lib/ruby/2.7.0/openssl/ssl.rb:239:in `peeraddr'
  2022-02-08 13:49:39 +0100 [error]: #2 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin_helper/server.rb:718:in `rescue in try_tls_accept'
  2022-02-08 13:49:39 +0100 [error]: #2 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin_helper/server.rb:696:in `try_tls_accept'
  2022-02-08 13:49:39 +0100 [error]: #2 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin_helper/server.rb:730:in `on_readable'
  2022-02-08 13:49:39 +0100 [error]: #2 /opt/td-agent/lib/ruby/gems/2.7.0/gems/cool.io-1.7.1/lib/cool.io/io.rb:186:in `on_readable'
  2022-02-08 13:49:39 +0100 [error]: #2 /opt/td-agent/lib/ruby/gems/2.7.0/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88:in `run_once'
  2022-02-08 13:49:39 +0100 [error]: #2 /opt/td-agent/lib/ruby/gems/2.7.0/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88:in `run'
  2022-02-08 13:49:39 +0100 [error]: #2 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin_helper/event_loop.rb:93:in `block in start'
  2022-02-08 13:49:39 +0100 [error]: #2 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
2022-02-08 13:49:39 +0100 [error]: #2 unexpected error error_class=Errno::ENOTCONN error="Transport endpoint is not connected - getpeername(2)"
  2022-02-08 13:49:39 +0100 [error]: #2 suppressed same stacktrace
#<Thread:0x0000000003b2a298@event_loop /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin_helper/thread.rb:70 run> terminated with exception (report_on_exception is true):
/opt/td-agent/lib/ruby/2.7.0/openssl/ssl.rb:239:in `peeraddr': Transport endpoint is not connected - getpeername(2) (Errno::ENOTCONN)
	from /opt/td-agent/lib/ruby/2.7.0/openssl/ssl.rb:239:in `peeraddr'
	from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin_helper/server.rb:718:in `rescue in try_tls_accept'
	from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin_helper/server.rb:696:in `try_tls_accept'
	from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin_helper/server.rb:730:in `on_readable'
	from /opt/td-agent/lib/ruby/gems/2.7.0/gems/cool.io-1.7.1/lib/cool.io/io.rb:186:in `on_readable'
	from /opt/td-agent/lib/ruby/gems/2.7.0/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88:in `run_once'
	from /opt/td-agent/lib/ruby/gems/2.7.0/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88:in `run'
	from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin_helper/event_loop.rb:93:in `block in start'
	from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin_helper/server.rb:697:in `accept_nonblock': SSL_accept SYSCALL returned=5 errno=0 state=SSLv3 read client certificate A (OpenSSL::SSL::SSLError)
	from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin_helper/server.rb:697:in `try_tls_accept'
	from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin_helper/server.rb:730:in `on_readable'
	from /opt/td-agent/lib/ruby/gems/2.7.0/gems/cool.io-1.7.1/lib/cool.io/io.rb:186:in `on_readable'
	from /opt/td-agent/lib/ruby/gems/2.7.0/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88:in `run_once'
	from /opt/td-agent/lib/ruby/gems/2.7.0/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88:in `run'
	from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin_helper/event_loop.rb:93:in `block in start'
	from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'

Additional context

Please explain what could possibly be causing these errors and how this can be resolved.

[ashie]

It's already fixed in #3483

[pratikshavyas]

The above shared PR fixes fluentd crashing issue after TLS error, Can you please share what could be the possible cause for the 'transport endpoint not connected error' as the setup works fine initially and then after a while it throws this error.