Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fluentd DaemonSet failed to flush the buffer sending to Opensearch cluster with undefined error #55

Closed
pashtet04 opened this issue May 22, 2022 · 2 comments

Comments

@pashtet04
Copy link

Steps to replicate

match configured to send container logs from Kubernetes Fluentd DaemonSet to secured Opensearch 1.3.2 with config

        <match kube.var.log.containers.**>
          @type opensearch
          hosts "#{ENV['DP_ELASTICSEARCH_HOSTS']}"
          port "#{ENV['DP_ELASTICSEARCH_PORT']}"
          scheme "#{ENV['DP_ELASTICSEARCH_SCHEME']}"
          user "#{ENV['DP_ELASTICSEARCH_USER']}"
          password "#{ENV['DP_ELASTICSEARCH_PASSWORD']}"
          index_name dev
          request_timeout 30s
          suppress_type_name true
          reconnect_on_error true
          reload_on_failure true
          reload_connections false
          ssl_verify false
          <buffer tag>
            timekey 1h
            flush_mode interval
            flush_interval 60s
            retry_wait 1.0
          </buffer>
        </match>

And fluentd got an error and no logs send:

2022-05-22 19:27:31 +0000 [warn]: #0 failed to flush the buffer. retry_times=0 next_retry_time=2022-05-22 19:27:33 +0000 chunk="5df9eb36ea559fe9862c44aff9c51945" error_class=Fluent::Plugin::OpenSearchOutput::RecoverableRequestFailure error="could not push logs to OpenSearch cluster ({:host=>\"HOST1\", :port=>9200, :scheme=>\"https\", :user=>\"fluentd\", :password=>\"obfuscated\"}, {:host=>\"HOST2\", :port=>9200, :scheme=>\"https\", :user=>\"fluentd\", :password=>\"obfuscated\"}, {:host=>\"HOST3\", :port=>9200, :scheme=>\"https\", :user=>\"fluentd\", :password=>\"obfuscated\"}): undefined methodcall' for nil:NilClass"
`

Expected Behavior or What you need to ask

No undefined errors in logs
Events/logs sends from Fluentd DaemonSet to the Opensearch cluster

Using Fluentd and OpenSearch plugin versions

  • fluent/fluentd-kubernetes-daemonset:v1.14.6-debian-opensearch-1.0

  • Kubernetes

  • fluentd 1.14.6

  • OpenSearch plugin version
    fluent-plugin-opensearch (1.0.3)
    opensearch-api (1.0.0)
    opensearch-ruby (1.0.0)
    opensearch-transport (1.0.0)
    fluent-config-regexp-type (1.0.0)
    fluent-plugin-concat (2.5.0)
    fluent-plugin-detect-exceptions (0.0.14)
    fluent-plugin-grok-parser (2.6.2)
    fluent-plugin-json-in-json-2 (1.0.2)
    fluent-plugin-kubernetes_metadata_filter (2.9.5)
    fluent-plugin-multi-format-parser (1.0.0)
    fluent-plugin-opensearch (1.0.3)
    fluent-plugin-parser-cri (0.1.1)
    fluent-plugin-prometheus (2.0.2)
    fluent-plugin-record-modifier (2.1.0)
    fluent-plugin-rewrite-tag-filter (2.4.0)
    fluent-plugin-systemd (1.0.5)
    fluentd (1.14.6)

  • OpenSearch version 1.3.2

@sdwerwed
Copy link

Probably duplicate of #42 ?

@pashtet04
Copy link
Author

seems to be

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants