Plugin retries even when retry_max_times 0

[willemveerman]

Describe the bug
Fluentd will retry chunks even when configured with retry_max_times 0

The buffer config below will cause retries, as per logs pasted below

          retry_max_times 0
          retry_wait 3s
          retry_timeout 100s
          retry_max_interval 30s
          retry_forever false

See below that certain chunk IDs are repeated, with a delay of 3 seconds as per config.

So retry_max_times is not being obeyed, but retry_wait parameter is being obeyed.

2020-10-26 12:15:39 +0000 [debug]: #0 taking back chunk for errors. chunk="5b291e18e2edb64d253b478e2b5e20d7"
2020-10-26 12:15:39 +0000 [warn]: #0 failed to flush the buffer. retry_time=0 next_retry_seconds=2020-10-26 12:15:42 +0000 chunk="5b291e18e2edb64d253b478e2b5e20d7" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster ({:host=>\"elasticsearch.default.svc.cluster.local\", :port=>9200, :scheme=>\"http\", :user=>\"\", :password=>\"obfuscated\"}): getaddrinfo: Name or service not known (SocketError)"
  2020-10-26 12:15:39 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluent-plugin-elasticsearch-4.0.11/lib/fluent/plugin/out_elasticsearch.rb:1005:in `rescue in send_bulk'
  2020-10-26 12:15:39 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluent-plugin-elasticsearch-4.0.11/lib/fluent/plugin/out_elasticsearch.rb:967:in `send_bulk'
  2020-10-26 12:15:39 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluent-plugin-elasticsearch-4.0.11/lib/fluent/plugin/out_elasticsearch.rb:794:in `block in write'
  2020-10-26 12:15:39 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluent-plugin-elasticsearch-4.0.11/lib/fluent/plugin/out_elasticsearch.rb:793:in `each'
  2020-10-26 12:15:39 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluent-plugin-elasticsearch-4.0.11/lib/fluent/plugin/out_elasticsearch.rb:793:in `write'
  2020-10-26 12:15:39 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluentd-1.11.4/lib/fluent/plugin/output.rb:1136:in `try_flush'
  2020-10-26 12:15:39 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluentd-1.11.4/lib/fluent/plugin/output.rb:1442:in `flush_thread_run'
  2020-10-26 12:15:39 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluentd-1.11.4/lib/fluent/plugin/output.rb:462:in `block (2 levels) in start'
  2020-10-26 12:15:39 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluentd-1.11.4/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
2020-10-26 12:15:42 +0000 [debug]: #0 taking back chunk for errors. chunk="5b291e18e2edb64d253b478e2b5e20d7"
2020-10-26 12:15:42 +0000 [error]: #0 failed to flush the buffer, and hit limit for retries. dropping all chunks in the buffer queue. retry_times=0 records=2 error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster ({:host=>\"elasticsearch.default.svc.cluster.local\", :port=>9200, :scheme=>\"http\", :user=>\"\", :password=>\"obfuscated\"}): getaddrinfo: Name or service not known (SocketError)"
  2020-10-26 12:15:42 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluent-plugin-elasticsearch-4.0.11/lib/fluent/plugin/out_elasticsearch.rb:1005:in `rescue in send_bulk'
  2020-10-26 12:15:42 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluent-plugin-elasticsearch-4.0.11/lib/fluent/plugin/out_elasticsearch.rb:967:in `send_bulk'
  2020-10-26 12:15:42 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluent-plugin-elasticsearch-4.0.11/lib/fluent/plugin/out_elasticsearch.rb:794:in `block in write'
  2020-10-26 12:15:42 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluent-plugin-elasticsearch-4.0.11/lib/fluent/plugin/out_elasticsearch.rb:793:in `each'
  2020-10-26 12:15:42 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluent-plugin-elasticsearch-4.0.11/lib/fluent/plugin/out_elasticsearch.rb:793:in `write'
  2020-10-26 12:15:42 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluentd-1.11.4/lib/fluent/plugin/output.rb:1136:in `try_flush'
  2020-10-26 12:15:42 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluentd-1.11.4/lib/fluent/plugin/output.rb:1442:in `flush_thread_run'
  2020-10-26 12:15:42 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluentd-1.11.4/lib/fluent/plugin/output.rb:462:in `block (2 levels) in start'
  2020-10-26 12:15:42 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluentd-1.11.4/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
2020-10-26 12:15:42 +0000 [debug]: #0 buffer queue cleared
2020-10-26 12:15:55 +0000 [debug]: #0 taking back chunk for errors. chunk="5b291e2a0fd2981e73850bb311ed7cb1"
2020-10-26 12:15:55 +0000 [warn]: #0 failed to flush the buffer. retry_time=0 next_retry_seconds=2020-10-26 12:15:58 +0000 chunk="5b291e2a0fd2981e73850bb311ed7cb1" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster ({:host=>\"elasticsearch.default.svc.cluster.local\", :port=>9200, :scheme=>\"http\", :user=>\"\", :password=>\"obfuscated\"}): getaddrinfo: Name or service not known (SocketError)"
  2020-10-26 12:15:55 +0000 [warn]: #0 suppressed same stacktrace
2020-10-26 12:15:58 +0000 [debug]: #0 taking back chunk for errors. chunk="5b291e2a0fd2981e73850bb311ed7cb1"
2020-10-26 12:15:58 +0000 [error]: #0 failed to flush the buffer, and hit limit for retries. dropping all chunks in the buffer queue. retry_times=0 records=69 error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster ({:host=>\"elasticsearch.default.svc.cluster.local\", :port=>9200, :scheme=>\"http\", :user=>\"\", :password=>\"obfuscated\"}): getaddrinfo: Name or service not known (SocketError)"
  2020-10-26 12:15:58 +0000 [error]: #0 suppressed same stacktrace
2020-10-26 12:15:58 +0000 [debug]: #0 buffer queue cleared

To Reproduce

1. Apply this yaml to k8s cluster - https://gist.github.com/willemveerman/b8c37d766b3cb625fc4669dff4c03b14 - e.g.:
   • minikube start
   • kubectl apply -f retry_issue.yaml
2. Wait until fluentd and ES pods are in Running status
3. Delete ES service: kubectl delete svc elasticsearch
4. View fluentd logs - kubectl logs fluentd-7sw4h - you will observe chunks being retried, even though configuration is retry_max_times 0

Expected behavior

When configuration is retry_max_times 0 chunk are sent once and not retried.

Your Environment

• OS version:
  😄 minikube v1.12.1 on Darwin 10.15.7
  ✨ Automatically selected the docker driver. Other choices: hyperkit, virtualbox
  🔥 Creating docker container (CPUs=2, Memory=4000MB) ...
  🐳 Preparing Kubernetes v1.18.3 on Docker 19.03.2 ...
• Fluentd v1.11.4
  • image fluent/fluentd-kubernetes-daemonset:v1.11.4-debian-elasticsearch6-1.0
• ES plugin 4.0.11

  2020-10-26 12:19:51 +0000 [info]: gem 'fluent-plugin-concat' version '2.4.0'
  2020-10-26 12:19:51 +0000 [info]: gem 'fluent-plugin-detect-exceptions' version '0.0.13'
  2020-10-26 12:19:51 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '4.0.11'
  2020-10-26 12:19:51 +0000 [info]: gem 'fluent-plugin-grok-parser' version '2.6.2'
  2020-10-26 12:19:51 +0000 [info]: gem 'fluent-plugin-json-in-json-2' version '1.0.2'
  2020-10-26 12:19:51 +0000 [info]: gem 'fluent-plugin-kubernetes_metadata_filter' version '2.3.0'
  2020-10-26 12:19:51 +0000 [info]: gem 'fluent-plugin-multi-format-parser' version '1.0.0'
  2020-10-26 12:19:51 +0000 [info]: gem 'fluent-plugin-prometheus' version '1.6.1'
  2020-10-26 12:19:51 +0000 [info]: gem 'fluent-plugin-record-modifier' version '2.0.1'
  2020-10-26 12:19:51 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.2.0'
  2020-10-26 12:19:51 +0000 [info]: gem 'fluent-plugin-systemd' version '1.0.2'
  2020-10-26 12:19:51 +0000 [info]: gem 'fluentd' version '1.11.4'
  

• ES version 6.8.12