From 32049a2cd65ccfe9783526d0ad3ce9dbd5d8cd1a Mon Sep 17 00:00:00 2001 From: Hiroshi Hatake Date: Tue, 1 Feb 2022 17:14:16 +0900 Subject: [PATCH 1/4] 1.0: Add OpenSearch plugin documentation Signed-off-by: Hiroshi Hatake --- .gitbook.yaml | 1 + SUMMARY.md | 1 + output/opensearch.md | 163 +++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 165 insertions(+) create mode 100644 output/opensearch.md diff --git a/.gitbook.yaml b/.gitbook.yaml index 8ccab13b..369aca69 100644 --- a/.gitbook.yaml +++ b/.gitbook.yaml @@ -89,6 +89,7 @@ redirects: v1.0/articles/out_exec: plugins/output/exec.md v1.0/articles/out_mongo_replset: plugins/output/mongo_replset.md v1.0/articles/out_relabel: plugins/output/relabel.md + v1.0/articles/out_opensearch: plugins/output/opensearch.md # Plugin/Filter v1.0/articles/filter_geoip: plugins/filter/geoip.md diff --git a/SUMMARY.md b/SUMMARY.md index b7f046e7..6b9e5b32 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -79,6 +79,7 @@ * [s3](output/s3.md) * [kafka](output/kafka.md) * [elasticsearch](output/elasticsearch.md) + * [opensearch](output/opensearch.md) * [mongo](output/mongo.md) * [mongo\_replset](output/mongo_replset.md) * [rewrite\_tag\_filter](output/rewrite_tag_filter.md) diff --git a/output/opensearch.md b/output/opensearch.md new file mode 100644 index 00000000..70b5aec1 --- /dev/null +++ b/output/opensearch.md @@ -0,0 +1,163 @@ +# opensearch + +The `out_opensearch` Output plugin writes records into Opensearch. By default, it creates records using [bulk api](https://opensearch.org/docs/latest/opensearch/rest-api/document-apis/bulk/) which performs multiple indexing operations in a single API call. This reduces overhead and can greatly increase indexing speed. This means that when you first import records using the plugin, records are not immediately pushed to OpenSearch. + +Records will be sent to OpenSearch when the `chunk_keys` condition has been met. To change the output frequency, please specify the `time` in `chunk_keys` and specify `timekey` value in the configuration. + +This document does not describe all the parameters. For details, refer to the **Further Reading** section. + +## Installation + +Since `out_opensearch` has been included in the standard distribution of `td-agent` since v4.4.0 and the alternative distribution of `calyptia-fluentd` since v1.3.4, `td-agent`/`calyptia-fluentd` users do not need to install it manually. + +If you have installed Fluentd without `td-agent` or `calyptia-fluentd`, please install this plugin using `fluent-gem`: + +```text +$ fluent-gem install fluent-plugin-opensearch +``` + +## Example Configuration + +Here is a simple working configuration which should serve as a good starting point for most users: + +```text + + @type opensearch + host localhost + port 9200 + logstash_format true + +``` + +For more details on each option, read the section on [Parameters](opensearch.md#parameters). + +## Plugin Helpers + +* [`event_emitter`](../plugin-helper-overview/api-plugin-helper-event_emitter.md) +* [`compat_parameters`](../plugin-helper-overview/api-plugin-helper-compat_parameters.md) + +## Parameters + +### `@type` \(required\) + +This option must be always `opensearch`. + +### `host` \(optional\) + +The hostname of your OpenSearch node \(default: `localhost`\). + +### `port` \(optional\) + +The port number of your OpenSearch node \(default: `9200`\). + +### `hosts` \(optional\) + +If you want to connect to more than one OpenSearch nodes, specify this option in the following format: + +```text +hosts host1:port1,host2:port2,host3:port3 +# or +hosts https://customhost.com:443/path,https://username:password@host-failover.com:443 +``` + +If you use this option, the `host` and `port` options are ignored. + +### `user`, `password` \(optional\) + +The login credentials to connect to the OpenSearch node \(default: `nil`\): + +```text +user fluent +password mysecret +``` + +### `scheme` \(optional\) + +Specify `https` if your OpenSearch endpoint supports SSL \(default: `http`\). + +### `path` \(optional\) + +The REST API endpoint of OpenSearch to post write requests \(default: `nil`\). + +### `index_name` \(optional\) + +The index name to write events to \(default: `fluentd`\). + +This option supports the placeholder syntax of Fluentd plugin API. For example, if you want to partition the index by tags, you can specify it like this: + +```text +index_name fluentd.${tag} +``` + +Here is a more practical example which partitions the OpenSearch index by tags and timestamps: + +```text +index_name fluentd.${tag}.%Y%m%d +``` + +Time placeholder needs to set up tag and time in `chunk_keys`. Also, it needs to specify timekey for time slice of chunk: + +```text + + timekey 1h # chunks per hours ("3600" also available) + +``` + +For more information about buffer options checkout the [Buffer Section Configuration](../configuration/buffer-section.md). + +### `logstash_format` \(optional\) + +If `true`, Fluentd uses the conventional index name format `logstash-%Y.%m.%d` \(default: `false`\). This option supersedes the `index_name` option. + +#### `@log_level` option + +The `@log_level` option allows the user to set different levels of logging for each plugin. + +Supported log levels: `fatal`, `error`, `warn`, `info`, `debug`, `trace`. + +Please see the [logging article](../deployment/logging.md) for further details. + +### `logstash_prefix` \(optional\) + +The logstash prefix index name to write events when `logstash_format` is `true` \(default: `logstash`\). + +## Miscellaneous + +You can use `%{}` style placeholders to escape for URL encoding needed characters. + +Valid configuration: + +```text +user %{demo+} +password %{@secret} +``` + +Valid configuration: + +```text +hosts https://%{j+hn}:%{passw@rd}@host1:443/elastic/,http://host2 +``` + +Invalid configuration: + +```text +user demo+ +password @secret +``` + +## Common Output / Buffer parameters + +For common output / buffer parameters, please check the following articles: + +* [Output Plugin Overview](./) +* [Buffer Section Configuration](../configuration/buffer-section.md) + +## Troubleshooting + +Please refer to the [OpenSearch's troubleshooting](https://github.com/fluent/fluent-plugin-opensearch#troubleshooting) section. + +## Further Reading + +* [`fluent-plugin-opensearch`](https://github.com/fluent/fluent-plugin-opensearch) + +If this article is incorrect or outdated, or omits critical information, please [let us know](https://github.com/fluent/fluentd-docs-gitbook/issues?state=open). [Fluentd](http://www.fluentd.org/) is an open-source project under [Cloud Native Computing Foundation \(CNCF\)](https://cncf.io/). All components are available under the Apache 2 License. From 61ab2cde385ac3d067be109438d5584211adef60 Mon Sep 17 00:00:00 2001 From: Hiroshi Hatake Date: Tue, 15 Feb 2022 14:23:36 +0900 Subject: [PATCH 2/4] Use "OpenSearch" notation Co-authored-by: Kentaro Hayashi Signed-off-by: Hiroshi Hatake --- output/opensearch.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/output/opensearch.md b/output/opensearch.md index 70b5aec1..09c4d0a5 100644 --- a/output/opensearch.md +++ b/output/opensearch.md @@ -1,6 +1,6 @@ # opensearch -The `out_opensearch` Output plugin writes records into Opensearch. By default, it creates records using [bulk api](https://opensearch.org/docs/latest/opensearch/rest-api/document-apis/bulk/) which performs multiple indexing operations in a single API call. This reduces overhead and can greatly increase indexing speed. This means that when you first import records using the plugin, records are not immediately pushed to OpenSearch. +The `out_opensearch` Output plugin writes records into OpenSearch. By default, it creates records using [bulk api](https://opensearch.org/docs/latest/opensearch/rest-api/document-apis/bulk/) which performs multiple indexing operations in a single API call. This reduces overhead and can greatly increase indexing speed. This means that when you first import records using the plugin, records are not immediately pushed to OpenSearch. Records will be sent to OpenSearch when the `chunk_keys` condition has been met. To change the output frequency, please specify the `time` in `chunk_keys` and specify `timekey` value in the configuration. From 798582a31dffd7c5dc2a71ccc267e4ac37ced8e2 Mon Sep 17 00:00:00 2001 From: Hiroshi Hatake Date: Tue, 15 Feb 2022 14:39:29 +0900 Subject: [PATCH 3/4] Use more concrete sentence for "the first configuration" Signed-off-by: Hiroshi Hatake --- output/opensearch.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/output/opensearch.md b/output/opensearch.md index 09c4d0a5..f2154a3a 100644 --- a/output/opensearch.md +++ b/output/opensearch.md @@ -18,7 +18,7 @@ $ fluent-gem install fluent-plugin-opensearch ## Example Configuration -Here is a simple working configuration which should serve as a good starting point for most users: +Here is a simple working configuration for OpenSearch instance that is running on localhost: ```text From 76ef4607b1fe4620d8b4b12e82553109746f51f6 Mon Sep 17 00:00:00 2001 From: Hiroshi Hatake Date: Wed, 16 Feb 2022 11:25:01 +0900 Subject: [PATCH 4/4] Update opensearch plugin installation step Current stable version of td-agent does not include OpenSearch plugin yet. We should add / modify descriptions for td-agent users later. Signed-off-by: Hiroshi Hatake --- output/opensearch.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/output/opensearch.md b/output/opensearch.md index f2154a3a..0763b5d5 100644 --- a/output/opensearch.md +++ b/output/opensearch.md @@ -8,14 +8,18 @@ This document does not describe all the parameters. For details, refer to the ** ## Installation -Since `out_opensearch` has been included in the standard distribution of `td-agent` since v4.4.0 and the alternative distribution of `calyptia-fluentd` since v1.3.4, `td-agent`/`calyptia-fluentd` users do not need to install it manually. +Since `out_opensearch` has been included in the alternative distribution of `calyptia-fluentd` since v1.3.4, `calyptia-fluentd` users do not need to install it manually. -If you have installed Fluentd without `td-agent` or `calyptia-fluentd`, please install this plugin using `fluent-gem`: +If you have installed Fluentd without `calyptia-fluentd`, please install this plugin using `fluent-gem` or `td-agent-gem` (for td-agent users): ```text $ fluent-gem install fluent-plugin-opensearch ``` +```text +$ td-agent-gem install fluent-plugin-opensearch +``` + ## Example Configuration Here is a simple working configuration for OpenSearch instance that is running on localhost: