Getting Errno::ENOTCONN error="Transport endpoint is not connected - getpeername(2) with fluentd

[pratikshavyas]

Describe the bug

We are using http input plugin and sending data to elasticsearch. This setup was working fine until we started receiving below error logs and fluentd stopped responding.
2022-02-08 13:49:39 +0100 [error]: #2 unexpected error error_class=Errno::ENOTCONN error="Transport endpoint is not connected - getpeername(2)"
2022-02-08 13:49:39 +0100 [error]: #2 /opt/td-agent/lib/ruby/2.7.0/openssl/ssl.rb:239:in peeraddr' 2022-02-08 13:49:39 +0100 [error]: #2 /opt/td-agent/lib/ruby/2.7.0/openssl/ssl.rb:239:in peeraddr'
2022-02-08 13:49:39 +0100 [error]: #2 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin_helper/server.rb:718:in rescue in try_tls_accept' 2022-02-08 13:49:39 +0100 [error]: #2 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin_helper/server.rb:696:in try_tls_accept'
2022-02-08 13:49:39 +0100 [error]: #2 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin_helper/server.rb:730:in on_readable' 2022-02-08 13:49:39 +0100 [error]: #2 /opt/td-agent/lib/ruby/gems/2.7.0/gems/cool.io-1.7.1/lib/cool.io/io.rb:186:in on_readable'
2022-02-08 13:49:39 +0100 [error]: #2 /opt/td-agent/lib/ruby/gems/2.7.0/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88:in run_once' 2022-02-08 13:49:39 +0100 [error]: #2 /opt/td-agent/lib/ruby/gems/2.7.0/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88:in run'
2022-02-08 13:49:39 +0100 [error]: #2 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin_helper/event_loop.rb:93:in block in start' 2022-02-08 13:49:39 +0100 [error]: #2 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin_helper/thread.rb:78:in block in thread_create'
2022-02-08 13:49:39 +0100 [error]: #2 unexpected error error_class=Errno::ENOTCONN error="Transport endpoint is not connected - getpeername(2)"
2022-02-08 13:49:39 +0100 [error]: #2 suppressed same stacktrace
#<Thread:0x0000000003b2a298@event_loop /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin_helper/thread.rb:70 run> terminated with exception (report_on_exception is true):
/opt/td-agent/lib/ruby/2.7.0/openssl/ssl.rb:239:in peeraddr': Transport endpoint is not connected - getpeername(2) (Errno::ENOTCONN) from /opt/td-agent/lib/ruby/2.7.0/openssl/ssl.rb:239:in peeraddr'

To Reproduce

Deploy fluentd with http as input plugin and elasticsearch as output.

Expected behavior

Fluentd should not stop and keep working without any error.

Your Environment

- Fluentd version: 1.13.3
- TD Agent version: 4.2.0

Your Configuration

<source>
    @type http
    @log_level "error"
    @id input_http_ipv4
    port 9000
    bind "0.0.0.0"
  </source>
Buffer section in match configured with values:
 <buffer tag, date, facility, vnf_name>
        @type "file"
        @log_level error
        path "/data/fluentdlogs/logs"
        timekey 1d
        flush_thread_count 4
        chunk_limit_size 4MB
        overflow_action block
        flush_mode interval
        flush_interval 5s
        total_limit_size 5GB
  </buffer>

Your Error Log

#2 unexpected error error_class=Errno::ENOTCONN error="Transport endpoint is not connected - getpeername(2)"
  2022-02-08 13:49:39 +0100 [error]: #2 /opt/td-agent/lib/ruby/2.7.0/openssl/ssl.rb:239:in `peeraddr'
  2022-02-08 13:49:39 +0100 [error]: #2 /opt/td-agent/lib/ruby/2.7.0/openssl/ssl.rb:239:in `peeraddr'
  2022-02-08 13:49:39 +0100 [error]: #2 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin_helper/server.rb:718:in `rescue in try_tls_accept'
  2022-02-08 13:49:39 +0100 [error]: #2 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin_helper/server.rb:696:in `try_tls_accept'
  2022-02-08 13:49:39 +0100 [error]: #2 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin_helper/server.rb:730:in `on_readable'
  2022-02-08 13:49:39 +0100 [error]: #2 /opt/td-agent/lib/ruby/gems/2.7.0/gems/cool.io-1.7.1/lib/cool.io/io.rb:186:in `on_readable'
  2022-02-08 13:49:39 +0100 [error]: #2 /opt/td-agent/lib/ruby/gems/2.7.0/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88:in `run_once'
  2022-02-08 13:49:39 +0100 [error]: #2 /opt/td-agent/lib/ruby/gems/2.7.0/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88:in `run'
  2022-02-08 13:49:39 +0100 [error]: #2 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin_helper/event_loop.rb:93:in `block in start'
  2022-02-08 13:49:39 +0100 [error]: #2 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
2022-02-08 13:49:39 +0100 [error]: #2 unexpected error error_class=Errno::ENOTCONN error="Transport endpoint is not connected - getpeername(2)"
  2022-02-08 13:49:39 +0100 [error]: #2 suppressed same stacktrace
#<Thread:0x0000000003b2a298@event_loop /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin_helper/thread.rb:70 run> terminated with exception (report_on_exception is true):
/opt/td-agent/lib/ruby/2.7.0/openssl/ssl.rb:239:in `peeraddr': Transport endpoint is not connected - getpeername(2) (Errno::ENOTCONN)
	from /opt/td-agent/lib/ruby/2.7.0/openssl/ssl.rb:239:in `peeraddr'
	from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin_helper/server.rb:718:in `rescue in try_tls_accept'
	from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin_helper/server.rb:696:in `try_tls_accept'
	from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin_helper/server.rb:730:in `on_readable'
	from /opt/td-agent/lib/ruby/gems/2.7.0/gems/cool.io-1.7.1/lib/cool.io/io.rb:186:in `on_readable'
	from /opt/td-agent/lib/ruby/gems/2.7.0/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88:in `run_once'
	from /opt/td-agent/lib/ruby/gems/2.7.0/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88:in `run'
	from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin_helper/event_loop.rb:93:in `block in start'
	from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin_helper/server.rb:697:in `accept_nonblock': SSL_accept SYSCALL returned=5 errno=0 state=SSLv3 read client certificate A (OpenSSL::SSL::SSLError)
	from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin_helper/server.rb:697:in `try_tls_accept'
	from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin_helper/server.rb:730:in `on_readable'
	from /opt/td-agent/lib/ruby/gems/2.7.0/gems/cool.io-1.7.1/lib/cool.io/io.rb:186:in `on_readable'
	from /opt/td-agent/lib/ruby/gems/2.7.0/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88:in `run_once'
	from /opt/td-agent/lib/ruby/gems/2.7.0/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88:in `run'
	from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin_helper/event_loop.rb:93:in `block in start'
	from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'

Additional context

No response

[ashie]

It's already fixed in #3483, please use v1.14.0 or later.

[pratikshavyas]

Can you please explain the cause behind these errors as they seem to appear randomly and stopping fluentd process.