Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Elasticsearch & Opensearch output plugins have unexpected/undocumented behavior #419

Open
brian-finisher opened this issue Jul 14, 2022 · 0 comments

Comments

@brian-finisher
Copy link

Describe the bug

It took me hours to debug this issue, and it wasn't until digging into the plugin's README files that I figured out what is going on!

Both the Opensearch & Elasticsearch plugins mention the logstash_format parameter impacting the index. What it does not say is that if you set logstash_format = false, it also sets include_timestamp=false. This results in records not being usable in Opensearch Dashboards & Kibana, as there are no time records.

Link to the problematic documentation

https://docs.fluentd.org/output/opensearch#logstash_format-optional
https://docs.fluentd.org/output/elasticsearch#logstash_format-optional

Expected explanation

If true, Fluentd uses the conventional index name format logstash-%Y.%m.%d (default: false). This option supersedes the index_name option.
In addition, include_timestamp (default: false) is set to true, which sends timestamp information which can be used by (Opensearch Dashboards / Kibana).

Additional context

The documentation here: https://github.com/fluent/fluent-plugin-opensearch#include_timestamp does indicate this behavior.

Note that the documentation here is much more thorough and authoritative overall.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant