diff --git a/_beats/script/build_docs.sh b/_beats/script/build_docs.sh index f21983e1dc6..c06bd45dd76 100755 --- a/_beats/script/build_docs.sh +++ b/_beats/script/build_docs.sh @@ -38,5 +38,5 @@ do params="$params --resource=${resource_dir}" fi - $docs_dir/build_docs $params --doc "$index" --out "$dest_dir" + $docs_dir/build_docs --asciidoctor $params --doc "$index" --out "$dest_dir" done diff --git a/docs/copied-from-beats/command-reference.asciidoc b/docs/copied-from-beats/command-reference.asciidoc index 1688aa49cc2..a0254871714 100644 --- a/docs/copied-from-beats/command-reference.asciidoc +++ b/docs/copied-from-beats/command-reference.asciidoc @@ -618,7 +618,7 @@ This command sets up the environment without actually running *SYNOPSIS* -tag::setup-command-tag[] +// tag::setup-command-tag[] ["source","sh",subs="attributes"] ---- {beatname_lc} setup [FLAGS] @@ -662,7 +662,7 @@ template, ILM policy, and write alias (if supported and configured). ifdef::apm-server[] *`--pipelines`*:: Registers the <> definitions set in `ingest/pipeline/definition.json`. -end::apm-server[] +endif::apm-server[] *`--template`*:: deprecated:[7.2] @@ -722,7 +722,7 @@ ifdef::apm-server[] endif::apm-server[] endif::[] -end::setup-command-tag[] +// end::setup-command-tag[] [[test-command]] ==== `test` command diff --git a/docs/copied-from-beats/monitoring/monitoring-beats.asciidoc b/docs/copied-from-beats/monitoring/monitoring-beats.asciidoc index 4b231288151..3334fcc52b1 100644 --- a/docs/copied-from-beats/monitoring/monitoring-beats.asciidoc +++ b/docs/copied-from-beats/monitoring/monitoring-beats.asciidoc @@ -7,7 +7,7 @@ You can use the {stack} {monitor-features} to gain insight into the health of ifndef::apm-server[] -{beatname_uc} agents running in your environment. +{beatname_uc} instances running in your environment. endif::[] ifdef::apm-server[] {beatname_uc}. diff --git a/docs/copied-from-beats/monitoring/monitoring-internal-collection.asciidoc b/docs/copied-from-beats/monitoring/monitoring-internal-collection.asciidoc index 2222f05fd89..ba0380c9ae0 100644 --- a/docs/copied-from-beats/monitoring/monitoring-internal-collection.asciidoc +++ b/docs/copied-from-beats/monitoring/monitoring-internal-collection.asciidoc @@ -48,9 +48,13 @@ monitoring: password: somepassword -------------------- + -If you configured a different output, such as {ls} or you want to send {beatname_uc} -monitoring events to a separate {es} cluster (referred to as the _monitoring cluster_), -you must specify additional configuration options. For example: +If you +ifndef::no_logstash_output[] +configured a different output, such as {ls} or you +endif::[] +want to send {beatname_uc} monitoring events to a separate {es} cluster +(referred to as the _monitoring cluster_), you must specify additional +configuration options. For example: + ["source","yml",subs="attributes"] -------------------- @@ -65,11 +69,11 @@ monitoring: that are part of the monitoring cluster. ifndef::serverless[] -. <<{beatname_lc}-starting,Start {beatname_uc}>>. +. Start {beatname_uc}. endif::[] ifdef::serverless[] -. <<{beatname_lc}-deploying,Deploy {beatname_uc}>>. +. Deploy {beatname_uc}. endif::[] . {kibana-ref}/monitoring-data.html[View the monitoring data in {kib}]. diff --git a/docs/copied-from-beats/outputconfig.asciidoc b/docs/copied-from-beats/outputconfig.asciidoc index 150273de453..e0c87e9263b 100644 --- a/docs/copied-from-beats/outputconfig.asciidoc +++ b/docs/copied-from-beats/outputconfig.asciidoc @@ -10,20 +10,20 @@ //// Make sure this content appears below a level 2 heading. ////////////////////////////////////////////////////////////////////////// +// tag::shared-outputconfig[] + [[configuring-output]] == Configure the output -ifdef::only-elasticsearch[] -You configure {beatname_uc} to write to Elasticsearch by setting options -in the `output.elasticsearch` section of the +{beatname_lc}.yml+ config file -endif::[] - -ifndef::only-elasticsearch[] You configure {beatname_uc} to write to a specific output by setting options -in the `Outputs` section of the +{beatname_lc}.yml+ config file. Only a single +in the Outputs section of the +{beatname_lc}.yml+ config file. Only a single output may be defined. -The following topics describe how to configure each supported output: +The following topics describe how to configure each supported output. If you've +secured the {stack}, also read <> for more about +security-related configuration options. + +// end::shared-outputconfig[] * <> * <> @@ -35,1735 +35,22 @@ endif::[] * <> * <> -If you've secured the {stack}, also read <> for more about -security-related configuration options. - -endif::[] - ifdef::beat-specific-output-config[] include::{beat-specific-output-config}[] endif::[] -[[elasticsearch-output]] -=== Configure the Elasticsearch output - -++++ -Elasticsearch -++++ - -When you specify Elasticsearch for the output, {beatname_uc} sends the transactions directly to Elasticsearch by using the Elasticsearch HTTP API. - -Example configuration: - -["source","yaml",subs="attributes"] ------------------------------------------------------------------------------- - -output.elasticsearch: - hosts: ["https://localhost:9200"] - index: "{beat_default_index_prefix}-%{[{beat_version_key}]}-%{+yyyy.MM.dd}" - ssl.certificate_authorities: ["/etc/pki/root/ca.pem"] - ssl.certificate: "/etc/pki/client/cert.pem" - ssl.key: "/etc/pki/client/cert.key" ------------------------------------------------------------------------------- - -To enable SSL, just add `https` to all URLs defined under __hosts__. - -["source","yaml",subs="attributes,callouts"] ------------------------------------------------------------------------------- - -output.elasticsearch: - hosts: ["https://localhost:9200"] - username: "{beatname_lc}_internal" - password: "{pwd}" ------------------------------------------------------------------------------- - -If the Elasticsearch nodes are defined by `IP:PORT`, then add `protocol: https` to the yaml file. - -["source","yaml",subs="attributes,callouts"] ------------------------------------------------------------------------------- -output.elasticsearch: - hosts: ["localhost"] - protocol: "https" - username: "{beatname_lc}_internal" - password: "{pwd}" ------------------------------------------------------------------------------- - - -For more information about securing {beatname_uc}, see -<>. - -ifndef::no_ilm[] -If you are indexing large amounts of time-series data, you might also want to -configure {beatname_uc} to use index lifecycle management. For more information -about configuring and using index lifecycle management with {beatname_uc}, see -<>. -endif::no_ilm[] - -==== Compatibility - -This output works with all compatible versions of Elasticsearch. See the -https://www.elastic.co/support/matrix#matrix_compatibility[Elastic Support -Matrix]. - -==== Configuration options - -You can specify the following options in the `elasticsearch` section of the +{beatname_lc}.yml+ config file: - -===== `enabled` - -The enabled config is a boolean setting to enable or disable the output. If set -to false, the output is disabled. - -The default value is true. - - -[[hosts-option]] -===== `hosts` - -The list of Elasticsearch nodes to connect to. The events are distributed to -these nodes in round robin order. If one node becomes unreachable, the event is -automatically sent to another node. Each Elasticsearch node can be defined as a `URL` or `IP:PORT`. -For example: `http://192.15.3.2`, `https://es.found.io:9230` or `192.24.3.2:9300`. -If no port is specified, `9200` is used. - -NOTE: When a node is defined as an `IP:PORT`, the _scheme_ and _path_ are taken from the -<> and <> config options. - -[source,yaml] ------------------------------------------------------------------------------- -output.elasticsearch: - hosts: ["10.45.3.2:9220", "10.45.3.1:9230"] - protocol: https - path: /elasticsearch ------------------------------------------------------------------------------- - -In the previous example, the Elasticsearch nodes are available at `https://10.45.3.2:9220/elasticsearch` and -`https://10.45.3.1:9230/elasticsearch`. - -===== `compression_level` - -The gzip compression level. Setting this value to 0 disables compression. -The compression level must be in the range of 1 (best speed) to 9 (best compression). - -Increasing the compression level will reduce the network usage but will increase the cpu usage. - -The default value is 0. - -===== `escape_html` - -Configure escaping of HTML in strings. Set to `true` to enable escaping. - -The default value is `false`. - - -===== `worker` - -The number of workers per configured host publishing events to Elasticsearch. This -is best used with load balancing mode enabled. Example: If you have 2 hosts and -3 workers, in total 6 workers are started (3 for each host). - -The default value is 1. - -===== `username` - -The basic authentication username for connecting to Elasticsearch. - -===== `password` - -The basic authentication password for connecting to Elasticsearch. - -===== `parameters` - -Dictionary of HTTP parameters to pass within the url with index operations. - -[[protocol-option]] -===== `protocol` - -The name of the protocol Elasticsearch is reachable on. The options are: -`http` or `https`. The default is `http`. However, if you specify a URL for -<>, the value of `protocol` is overridden by whatever scheme you -specify in the URL. - -[[path-option]] -===== `path` - -An HTTP path prefix that is prepended to the HTTP API calls. This is useful for -the cases where Elasticsearch listens behind an HTTP reverse proxy that exports -the API under a custom prefix. - -===== `headers` - -Custom HTTP headers to add to each request created by the Elasticsearch output. -Example: - -[source,yaml] ------------------------------------------------------------------------------- -output.elasticsearch.headers: - X-My-Header: Header contents ------------------------------------------------------------------------------- - -It is generally possible to specify multiple header values for the same header -name by separating them with a comma. - -===== `proxy_url` - -The URL of the proxy to use when connecting to the Elasticsearch servers. The -value may be either a complete URL or a "host[:port]", in which case the "http" -scheme is assumed. If a value is not specified through the configuration file -then proxy environment variables are used. See the -https://golang.org/pkg/net/http/#ProxyFromEnvironment[Go documentation] -for more information about the environment variables. - -[[index-option-es]] -===== `index` - -ifndef::apm-server[] -The index name to write events to when you're using daily indices. The default is -+"{beatname_lc}-%{[{beat_version_key}]}-%{+yyyy.MM.dd}"+ (for example, -+"{beatname_lc}-{version}-{localdate}"+). If you change this setting, you also -need to configure the `setup.template.name` and `setup.template.pattern` options -(see <>). -endif::apm-server[] - -ifdef::apm-server[] -The index name to write events to. The default is -+"apm-%{[{beat_version_key}]}-{type}-%{+yyyy.MM.dd}"+ (for example, -+"apm-{version}-transaction-{localdate}"+). See -<> for more information on -default index configuration. - -IMPORTANT: If you change this setting, -you need to configure the `setup.template.name` and `setup.template.pattern` options -(see <>). You also must set the default index configuration -in the `apm-server.yml` file. - -NOTE: +{beat_version_key}+ is a field managed by Beats that is added to every document. -It holds the current version of APM Server. -endif::apm-server[] - -ifndef::no_dashboards[] -If you are using the pre-built Kibana -dashboards, you also need to set the `setup.dashboards.index` option (see -<>). -endif::no_dashboards[] - -ifndef::apm-server[] -ifndef::no_ilm[] -The `index` setting is ignored when index lifecycle management is enabled. If -you’re sending events to a cluster that supports index lifecycle management, see -<> to learn how to change the index name. -endif::no_ilm[] - -You can set the index dynamically by using a format string to access any event -field. For example, this configuration uses a custom field, `fields.log_type`, -to set the index: - -["source","yaml",subs="attributes"] ------------------------------------------------------------------------------- -output.elasticsearch: - hosts: ["http://localhost:9200"] - index: "%{[fields.log_type]}-%{[{beat_version_key}]}-%{+yyyy.MM.dd}" <1> ------------------------------------------------------------------------------- - -<1> We recommend including +{beat_version_key}+ in the name to avoid mapping issues -when you upgrade. - -With this configuration, all events with `log_type: normal` are sent to an -index named +normal-{version}-{localdate}+, and all events with -`log_type: critical` are sent to an index named -+critical-{version}-{localdate}+. -endif::apm-server[] - -ifdef::apm-server[] -You can set the index dynamically by using a format string to access any event -field. For example, this configuration uses the field, `processor.event`, -to set the index: - -["source","yaml",subs="attributes"] ------------------------------------------------------------------------------- -output.elasticsearch: - hosts: ["http://localhost:9200"] - index: "apm-%{[observer.version]}-%{[processor.event]}-%{+yyyy.MM.dd}" <1> ------------------------------------------------------------------------------- - -<1> `observer` refers to {beatname_uc}. We recommend including -+{beat_version_key}+ in the name to avoid mapping issues when you upgrade -{beatname_uc}. - -With this configuration, -all events are separated by their `processor.event` into different indices. -endif::apm-server[] - -TIP: To learn how to add custom fields to events, see the -<> option. - -See the <> setting for other ways to set the index -dynamically. - - -[[indices-option-es]] -===== `indices` - -An array of index selector rules. Each rule specifies the index to use for -events that match the rule. During publishing, {beatname_uc} uses the first -matching rule in the array. Rules can contain conditionals, format string-based -fields, and name mappings. If the `indices` setting is missing or no rule -matches, the <> setting is used. - -Rule settings: - -*`index`*:: The index format string to use. If this string contains field -references, such as `%{[fields.name]}`, the fields must exist, or the rule fails. - -*`mappings`*:: A dictionary that takes the value returned by `index` and maps it -to a new name. - -*`default`*:: The default string value to use if `mappings` does not find a -match. - -*`when`*:: A condition that must succeed in order to execute the current rule. -ifndef::no-processors[] -All the <> supported by processors are also supported -here. -endif::no-processors[] - -ifndef::apm-server[] -The following example sets the index based on whether the `message` field -contains the specified string: - -["source","yaml",subs="attributes"] ------------------------------------------------------------------------------- -output.elasticsearch: - hosts: ["http://localhost:9200"] - indices: - - index: "warning-%{[{beat_version_key}]}-%{+yyyy.MM.dd}" - when.contains: - message: "WARN" - - index: "error-%{[{beat_version_key}]}-%{+yyyy.MM.dd}" - when.contains: - message: "ERR" ------------------------------------------------------------------------------- - - -This configuration results in indices named +warning-{version}-{localdate}+ -and +error-{version}-{localdate}+ (plus the default index if no matches are -found). - -The following example sets the index by taking the name returned by the `index` -format string and mapping it to a new name that's used for the index: - -["source","yaml"] ------------------------------------------------------------------------------- -output.elasticsearch: - hosts: ["http://localhost:9200"] - indices: - - index: "%{[fields.log_type]}" - mappings: - critical: "sev1" - normal: "sev2" - default: "sev3" ------------------------------------------------------------------------------- - - -This configuration results in indices named `sev1`, `sev2`, and `sev3`. - -The `mappings` setting simplifies the configuration, but is limited to string -values. You cannot specify format strings within the mapping pairs. -endif::apm-server[] - -ifdef::apm-server[] -The following example sets the index based on whether the `processor.event` field -contains the specified string: - -["source","yaml",subs="attributes"] ------------------------------------------------------------------------------- -output.elasticsearch: - hosts: ["http://localhost:9200"] - indices: - - index: "apm-%{[observer.version]}-sourcemap" - when.contains: - processor.event: "sourcemap" - - - index: "apm-%{[observer.version]}-error-%{+yyyy.MM.dd}" - when.contains: - processor.event: "error" - - - index: "apm-%{[observer.version]}-transaction-%{+yyyy.MM.dd}" - when.contains: - processor.event: "transaction" - - - index: "apm-%{[observer.version]}-span-%{+yyyy.MM.dd}" - when.contains: - processor.event: "span" - - - index: "apm-%{[observer.version]}-metric-%{+yyyy.MM.dd}" - when.contains: - processor.event: "metric" - - - index: "apm-%{[observer.version]}-onboarding-%{+yyyy.MM.dd}" - when.contains: - processor.event: "onboarding" ------------------------------------------------------------------------------- - -NOTE: `observer` refers to {beatname_uc}. We recommend including -+{beat_version_key}+ in the name to avoid mapping issues when you upgrade -{beatname_uc}. - -This is the default configuration for {beatname_uc} and results in indices -named in the following format: +"apm-%{[{beat_version_key}]}-{type}-%{+yyyy.MM.dd}"+ -For example: +"apm-{version}-transaction-{localdate}"+. - -The following example sets the index by taking the name returned by the `index` -format string and mapping it to a new name that's used for the index: - -["source","yaml"] ------------------------------------------------------------------------------- -output.elasticsearch: - hosts: ["http://localhost:9200"] - indices: - - index: "%{[processor.event]}" - mappings: - sourcemap: "apm-sourcemap" - error: "apm-error" - transaction: "apm-transaction" - span: "apm-span" - metric: "apm-metric" - onboarding: "apm-onboarding" - default: "apm" ------------------------------------------------------------------------------- - -This configuration results in indices named `apm-sourcemap`, `apm-error`, etc. - -The `mappings` setting simplifies the configuration, but is limited to string -values. You cannot specify format strings within the mapping pairs. -endif::apm-server[] - -//TODO: MOVE ILM OPTIONS TO APPEAR LOGICALLY BASED ON LOCATION IN THE YAML FILE. - -ifndef::no_ilm[] -[[ilm-es]] -===== `ilm` - -Configuration options for index lifecycle management. - -See <> for more information. -endif::no_ilm[] - -ifndef::no-pipeline[] -[[pipeline-option-es]] -===== `pipeline` - -A format string value that specifies the ingest node pipeline to write events to. - -["source","yaml"] ------------------------------------------------------------------------------- -output.elasticsearch: - hosts: ["http://localhost:9200"] - pipeline: my_pipeline_id ------------------------------------------------------------------------------- - -For more information, see <>. - -ifndef::apm-server[] -You can set the ingest node pipeline dynamically by using a format string to -access any event field. For example, this configuration uses a custom field, -`fields.log_type`, to set the pipeline for each event: - -["source","yaml",subs="attributes"] ------------------------------------------------------------------------------- -output.elasticsearch: - hosts: ["http://localhost:9200"] - pipeline: "%{[fields.log_type]}_pipeline" ------------------------------------------------------------------------------- - - -With this configuration, all events with `log_type: normal` are sent to a pipeline -named `normal_pipeline`, and all events with `log_type: critical` are sent to a -pipeline named `critical_pipeline`. -endif::apm-server[] - -ifdef::apm-server[] -You can set the ingest node pipeline dynamically by using a format string to -access any event field. For example, this configuration uses the field, -`processor.event`, to set the pipeline for each event: - -["source","yaml",subs="attributes"] ------------------------------------------------------------------------------- -output.elasticsearch: - hosts: ["http://localhost:9200"] - pipeline: "%{[processor.event]}_pipeline" ------------------------------------------------------------------------------- - - -With this configuration, all events with `processor.event: transaction` are sent to a pipeline -named `transaction_pipeline`. Similarly, all events with `processor.event: error` are sent to a -pipeline named `error_pipeline`. - -The default pipeline is `apm`. It adds user agent and geo ip information to events. -To disable this, or any other pipeline, set `output.elasticsearch.pipeline: _none`. -endif::apm-server[] - -TIP: To learn how to add custom fields to events, see the -<> option. - -See the <> setting for other ways to set the -ingest node pipeline dynamically. - -[[pipelines-option-es]] -===== `pipelines` - -An array of pipeline selector rules. Each rule specifies the ingest node -pipeline to use for events that match the rule. During publishing, {beatname_uc} -uses the first matching rule in the array. Rules can contain conditionals, -format string-based fields, and name mappings. If the `pipelines` setting is -missing or no rule matches, the <> setting is -used. - -Rule settings: - -*`pipeline`*:: The pipeline format string to use. If this string contains field -references, such as `%{[fields.name]}`, the fields must exist, or the rule -fails. - -*`mappings`*:: A dictionary that takes the value returned by `pipeline` and maps -it to a new name. - -*`default`*:: The default string value to use if `mappings` does not find a -match. - -*`when`*:: A condition that must succeed in order to execute the current rule. -ifndef::no-processors[] -All the <> supported by processors are also supported -here. -endif::no-processors[] - -ifndef::apm-server[] -The following example sends events to a specific pipeline based on whether the -`message` field contains the specified string: - -["source","yaml"] ------------------------------------------------------------------------------- -output.elasticsearch: - hosts: ["http://localhost:9200"] - pipelines: - - pipeline: "warning_pipeline" - when.contains: - message: "WARN" - - pipeline: "error_pipeline" - when.contains: - message: "ERR" ------------------------------------------------------------------------------- - - -The following example sets the pipeline by taking the name returned by the -`pipeline` format string and mapping it to a new name that's used for the -pipeline: - -["source","yaml"] ------------------------------------------------------------------------------- -output.elasticsearch: - hosts: ["http://localhost:9200"] - pipelines: - - pipeline: "%{[fields.log_type]}" - mappings: - critical: "sev1_pipeline" - normal: "sev2_pipeline" - default: "sev3_pipeline" ------------------------------------------------------------------------------- - - -With this configuration, all events with `log_type: critical` are sent to -`sev1_pipeline`, all events with `log_type: normal` are sent to a -`sev2_pipeline`, and all other events are sent to `sev3_pipeline`. -endif::apm-server[] - -ifdef::apm-server[] -The following example sends events to a specific pipeline based on whether the -`processor.event` field contains the specified string: - -["source","yaml"] ------------------------------------------------------------------------------- -output.elasticsearch: - hosts: ["http://localhost:9200"] - pipelines: - - pipeline: "sourcemap_pipeline" - when.contains: - processor.event: "sourcemap" - - - pipeline: "error_pipeline" - when.contains: - processor.event: "error" - - - pipeline: "transaction_pipeline" - when.contains: - processor.event: "transaction" - - - pipeline: "span_pipeline" - when.contains: - processor.event: "span" - - - pipeline: "metric_pipeline" - when.contains: - processor.event: "metric" - - - pipeline: "onboarding_pipeline" - when.contains: - processor.event: "onboarding" ------------------------------------------------------------------------------- - - -The following example sets the pipeline by taking the name returned by the -`pipeline` format string and mapping it to a new name that's used for the -pipeline: - -["source","yaml"] ------------------------------------------------------------------------------- -output.elasticsearch: - hosts: ["http://localhost:9200"] - pipelines: - - pipeline: "%{[processor.event]}" - mappings: - sourcemap: "sourcemap_pipeline" - error: "error_pipeline" - transaction: "transaction_pipeline" - span: "span_pipeline" - metric: "metric_pipeline" - onboarding: "onboarding_pipeline" - default: "apm_pipeline" ------------------------------------------------------------------------------- - -With this configuration, all events with `processor.event: transaction` are sent to a pipeline -named `transaction_pipeline`, all events with `processor.event: error` are sent to a -pipeline named `error_pipeline`, etc. - -NOTE: Defining any pipeline will deactivate the default `apm` pipeline. -endif::apm-server[] - -For more information about ingest node pipelines, see -<>. - -endif::[] - -===== `max_retries` - -ifdef::ignores_max_retries[] -{beatname_uc} ignores the `max_retries` setting and retries indefinitely. -endif::[] - -ifndef::ignores_max_retries[] -The number of times to retry publishing an event after a publishing failure. -After the specified number of retries, the events are typically dropped. - -Set `max_retries` to a value less than 0 to retry until all events are published. - -The default is 3. -endif::[] - - -===== `bulk_max_size` - -The maximum number of events to bulk in a single Elasticsearch bulk API index request. The default is 50. - -Events can be collected into batches. {beatname_uc} will split batches larger than `bulk_max_size` -into multiple batches. - -Specifying a larger batch size can improve performance by lowering the overhead of sending events. -However big batch sizes can also increase processing times, which might result in -API errors, killed connections, timed-out publishing requests, and, ultimately, lower -throughput. - -Setting `bulk_max_size` to values less than or equal to 0 disables the -splitting of batches. When splitting is disabled, the queue decides on the -number of events to be contained in a batch. - -===== `backoff.init` - -The number of seconds to wait before trying to reconnect to Elasticsearch after -a network error. After waiting `backoff.init` seconds, {beatname_uc} tries to -reconnect. If the attempt fails, the backoff timer is increased exponentially up -to `backoff.max`. After a successful connection, the backoff timer is reset. The -default is 1s. - - -===== `backoff.max` - -The maximum number of seconds to wait before attempting to connect to -Elasticsearch after a network error. The default is 60s. - -===== `timeout` - -The http request timeout in seconds for the Elasticsearch request. The default is 90. - -===== `ssl` - -Configuration options for SSL parameters like the certificate authority to use -for HTTPS-based connections. If the `ssl` section is missing, the host CAs are used for HTTPS connections to -Elasticsearch. - -See <> for more information. - -//begin outer exclude for ls, kafka, redis, file, and console -ifndef::only-elasticsearch[] - -[[logstash-output]] -=== Configure the Logstash output - -++++ -Logstash -++++ - -The Logstash output sends events directly to Logstash by using the lumberjack -protocol, which runs over TCP. Logstash allows for additional processing and routing of -generated events. - -include::./shared-logstash-config.asciidoc[] - -==== Accessing metadata fields - -Every event sent to Logstash contains the following metadata fields that you can -use in Logstash for indexing and filtering: - -ifndef::apm-server[] -["source","json",subs="attributes"] ------------------------------------------------------------------------------- -{ - ... - "@metadata": { <1> - "beat": "{beat_default_index_prefix}", <2> - "version": "{stack-version}" <3> - } -} ------------------------------------------------------------------------------- -<1> {beatname_uc} uses the `@metadata` field to send metadata to Logstash. See the -{logstash-ref}/event-dependent-configuration.html#metadata[Logstash documentation] -for more about the `@metadata` field. -<2> The default is {beat_default_index_prefix}. To change this value, set the -<> option in the {beatname_uc} config file. -<3> The current version of {beatname_uc}. -endif::[] - -ifdef::apm-server[] -["source","json",subs="attributes"] ------------------------------------------------------------------------------- -{ - ... - "@metadata": { <1> - "beat": "{beat_default_index_prefix}", <2> - "pipeline":"apm", <3> - "version": "{stack-version}" <4> - } -} ------------------------------------------------------------------------------- -<1> {beatname_uc} uses the `@metadata` field to send metadata to Logstash. See the -{logstash-ref}/event-dependent-configuration.html#metadata[Logstash documentation] -for more about the `@metadata` field. -<2> The default is {beat_default_index_prefix}. To change this value, set the -<> option in the {beatname_uc} config file. -<3> The default pipeline configuration: `apm`. Additional pipelines can be enabled -with a {logstash-ref}/use-ingest-pipelines.html[Logstash pipeline config]. -<4> The current version of {beatname_uc}. -endif::[] - -You can access this metadata from within the Logstash config file to set values -dynamically based on the contents of the metadata. - -For example, the following Logstash configuration file for version 7.x sets -Logstash to use the index reported by {beatname_uc} for indexing events -into Elasticsearch: - -ifndef::apm-server[] -[source,logstash] ------------------------------------------------------------------------------- - -input { - beats { - port => 5044 - } -} - -output { - elasticsearch { - hosts => ["http://localhost:9200"] - index => "%{[@metadata][beat]}-%{[@metadata][version]}" <1> - } -} ------------------------------------------------------------------------------- -<1> `%{[@metadata][beat]}` sets the first part of the index name to the value -of the `beat` metadata field and `%{[@metadata][version]}` sets the second part to -the Beat's version. For example: -+{beat_default_index_prefix}-{version}+. -endif::[] - -ifdef::apm-server[] -[source,logstash] ------- -input { - beats { - port => 5044 - } -} - -filter { - if [@metadata][beat] == "apm" { - if [processor][event] == "sourcemap" { - mutate { - add_field => { "[@metadata][index]" => "%{[@metadata][beat]}-%{[@metadata][version]}-%{[processor][event]}" } <1> - } - } else { - mutate { - add_field => { "[@metadata][index]" => "%{[@metadata][beat]}-%{[@metadata][version]}-%{[processor][event]}-%{+yyyy.MM.dd}" } <2> - } - } - } -} - -output { - elasticsearch { - hosts => ["http://localhost:9200"] - index => "%{[@metadata][index]}" - } -} ------- -<1> Creates a new field named `@metadata.index`. -`%{[@metadata][beat]}` sets the first part of the index name to the value of the `beat` metadata field. -`%{[@metadata][version]}` sets the second part to {beatname_uc}'s version. -`%{[processor][event]}` sets the final part based on the APM event type. -For example: +{beat_default_index_prefix}-{version}-sourcemap+. -<2> In addition to the above rules, this pattern appends a date to the `index` name so Logstash creates a new index each day. -For example: +{beat_default_index_prefix}-{version}-transaction-{sample_date_0}+. -endif::[] - -Events indexed into Elasticsearch with the Logstash configuration shown here -will be similar to events directly indexed by {beatname_uc} into Elasticsearch. - -ifndef::apm-server[] -NOTE: If ILM is not being used, set `index` to `%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}` instead so Logstash creates an index per day, based on the `@timestamp` value of the events coming from Beats. -endif::[] - -ifdef::apm-server[] -==== Logstash and ILM - -When used with {apm-server-ref}/manual-ilm-setup.html[Index lifecycle management], Logstash does not need to create a new index each day. -Here's a sample Logstash configuration file that would accomplish this: - -[source,logstash] ------- -input { - beats { - port => 5044 - } -} - -output { - elasticsearch { - hosts => ["http://localhost:9200"] - index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{[processor][event]}" - } -} ------- -endif::[] - -==== Compatibility - -This output works with all compatible versions of Logstash. See the -https://www.elastic.co/support/matrix#matrix_compatibility[Elastic Support -Matrix]. - -==== Configuration options - -You can specify the following options in the `logstash` section of the -+{beatname_lc}.yml+ config file: - -===== `enabled` - -The enabled config is a boolean setting to enable or disable the output. If set -to false, the output is disabled. - -The default value is `true`. - -[[hosts]] -===== `hosts` - -The list of known Logstash servers to connect to. If load balancing is disabled, but -multiple hosts are configured, one host is selected randomly (there is no precedence). -If one host becomes unreachable, another one is selected randomly. - -All entries in this list can contain a port number. The default port number 5044 will be used, if no number is given. - -===== `compression_level` - -The gzip compression level. Setting this value to 0 disables compression. -The compression level must be in the range of 1 (best speed) to 9 (best compression). - -Increasing the compression level will reduce the network usage but will increase the cpu usage. - -The default value is 3. - -===== `escape_html` - -Configure escaping of HTML in strings. Set to `true` to enable escaping. - -The default value is `false`. - -===== `worker` - -The number of workers per configured host publishing events to Logstash. This -is best used with load balancing mode enabled. Example: If you have 2 hosts and -3 workers, in total 6 workers are started (3 for each host). - -[[loadbalance]] -===== `loadbalance` - -If set to true and multiple Logstash hosts are configured, the output plugin -load balances published events onto all Logstash hosts. If set to false, -the output plugin sends all events to only one host (determined at random) and -will switch to another host if the selected one becomes unresponsive. The default value is false. - -["source","yaml",subs="attributes"] ------------------------------------------------------------------------------- -output.logstash: - hosts: ["localhost:5044", "localhost:5045"] - loadbalance: true - index: {beatname_lc} ------------------------------------------------------------------------------- - -===== `ttl` - -Time to live for a connection to Logstash after which the connection will be re-established. -Useful when Logstash hosts represent load balancers. Since the connections to Logstash hosts -are sticky, operating behind load balancers can lead to uneven load distribution between the instances. -Specifying a TTL on the connection allows to achieve equal connection distribution between the -instances. Specifying a TTL of 0 will disable this feature. - -The default value is 0. - -NOTE: The "ttl" option is not yet supported on an async Logstash client (one with the "pipelining" option set). - -===== `pipelining` - -Configures number of batches to be sent asynchronously to logstash while waiting -for ACK from logstash. Output only becomes blocking once number of `pipelining` -batches have been written. Pipelining is disabled if a values of 0 is -configured. The default value is 2. - -===== `proxy_url` - -The URL of the SOCKS5 proxy to use when connecting to the Logstash servers. The -value must be a URL with a scheme of `socks5://`. The protocol used to -communicate to Logstash is not based on HTTP so a web-proxy cannot be used. - -If the SOCKS5 proxy server requires client authentication, then a username and -password can be embedded in the URL as shown in the example. - -When using a proxy, hostnames are resolved on the proxy server instead of on the -client. You can change this behavior by setting the -<> option. - -["source","yaml",subs="attributes"] ------------------------------------------------------------------------------- -output.logstash: - hosts: ["remote-host:5044"] - proxy_url: socks5://user:password@socks5-proxy:2233 ------------------------------------------------------------------------------- - -[[logstash-proxy-use-local-resolver]] -===== `proxy_use_local_resolver` - -The `proxy_use_local_resolver` option determines if Logstash hostnames are -resolved locally when using a proxy. The default value is false which means -that when a proxy is used the name resolution occurs on the proxy server. - -[[logstash-index]] -===== `index` - -The index root name to write events to. The default is the Beat name. For -example +"{beat_default_index_prefix}"+ generates +"[{beat_default_index_prefix}-]{version}-YYYY.MM.DD"+ -indices (for example, +"{beat_default_index_prefix}-{version}-2017.04.26"+). - -===== `ssl` - -Configuration options for SSL parameters like the root CA for Logstash connections. See -<> for more information. To use SSL, you must also configure the -https://www.elastic.co/guide/en/logstash/current/plugins-inputs-beats.html[Beats input plugin for Logstash] to use SSL/TLS. - -===== `timeout` - -The number of seconds to wait for responses from the Logstash server before timing out. The default is 30 (seconds). - -===== `max_retries` - -ifdef::ignores_max_retries[] -{beatname_uc} ignores the `max_retries` setting and retries indefinitely. -endif::[] - -ifndef::ignores_max_retries[] -The number of times to retry publishing an event after a publishing failure. -After the specified number of retries, the events are typically dropped. - -Set `max_retries` to a value less than 0 to retry until all events are published. - -The default is 3. -endif::[] - -===== `bulk_max_size` - -The maximum number of events to bulk in a single Logstash request. The default is 2048. - -If the Beat sends single events, the events are collected into batches. If the Beat publishes -a large batch of events (larger than the value specified by `bulk_max_size`), the batch is -split. - -Specifying a larger batch size can improve performance by lowering the overhead of sending events. -However big batch sizes can also increase processing times, which might result in -API errors, killed connections, timed-out publishing requests, and, ultimately, lower -throughput. - -Setting `bulk_max_size` to values less than or equal to 0 disables the -splitting of batches. When splitting is disabled, the queue decides on the -number of events to be contained in a batch. - - -===== `slow_start` - -If enabled only a subset of events in a batch of events is transferred per transaction. -The number of events to be sent increases up to `bulk_max_size` if no error is encountered. -On error the number of events per transaction is reduced again. - -The default is `false`. - -===== `backoff.init` - -The number of seconds to wait before trying to reconnect to Logstash after -a network error. After waiting `backoff.init` seconds, {beatname_uc} tries to -reconnect. If the attempt fails, the backoff timer is increased exponentially up -to `backoff.max`. After a successful connection, the backoff timer is reset. The -default is 1s. - -===== `backoff.max` - -The maximum number of seconds to wait before attempting to connect to -Logstash after a network error. The default is 60s. - -[[kafka-output]] -=== Configure the Kafka output - -++++ -Kafka -++++ - -The Kafka output sends the events to Apache Kafka. - -Example configuration: - -[source,yaml] ------------------------------------------------------------------------------- -output.kafka: - # initial brokers for reading cluster metadata - hosts: ["kafka1:9092", "kafka2:9092", "kafka3:9092"] - - # message topic selection + partitioning - topic: '%{[fields.log_topic]}' - partition.round_robin: - reachable_only: false - - required_acks: 1 - compression: gzip - max_message_bytes: 1000000 ------------------------------------------------------------------------------- - -NOTE: Events bigger than <> will be dropped. To avoid this problem, make sure {beatname_uc} does not generate events bigger than <>. - -[[kafka-compatibility]] -==== Compatibility - -This output works with all Kafka versions in between 0.11 and 2.1.0. Older versions -might work as well, but are not supported. - -==== Configuration options - -You can specify the following options in the `kafka` section of the +{beatname_lc}.yml+ config file: - -===== `enabled` - -The `enabled` config is a boolean setting to enable or disable the output. If set -to false, the output is disabled. - -The default value is true. - -===== `hosts` - -The list of Kafka broker addresses from where to fetch the cluster metadata. -The cluster metadata contain the actual Kafka brokers events are published to. - -===== `version` - -Kafka version {beatname_lc} is assumed to run against. Defaults to 1.0.0. - -Event timestamps will be added, if version 0.10.0.0+ is enabled. - -Valid values are all kafka releases in between `0.8.2.0` and `2.0.0`. - -See <> for information on supported versions. - -===== `username` - -The username for connecting to Kafka. If username is configured, the password -must be configured as well. Only SASL/PLAIN is supported. - -===== `password` - -The password for connecting to Kafka. - -[[topic-option-kafka]] -===== `topic` - -The Kafka topic used for produced events. - -You can set the topic dynamically by using a format string to access any -event field. For example, this configuration uses a custom field, -`fields.log_topic`, to set the topic for each event: - -[source,yaml] ------ -topic: '%{[fields.log_topic]}' ------ - -TIP: To learn how to add custom fields to events, see the -<> option. - -See the <> setting for other ways to set the -topic dynamically. - -[[topics-option-kafka]] -===== `topics` - -An array of topic selector rules. Each rule specifies the `topic` to use for -events that match the rule. During publishing, {beatname_uc} sets the `topic` -for each event based on the first matching rule in the array. Rules -can contain conditionals, format string-based fields, and name mappings. If the -`topics` setting is missing or no rule matches, the -<> field is used. - -Rule settings: - -*`topic`*:: The topic format string to use. If this string contains field -references, such as `%{[fields.name]}`, the fields must exist, or the rule -fails. - -*`mappings`*:: A dictionary that takes the value returned by `topic` and maps it -to a new name. - -*`default`*:: The default string value to use if `mappings` does not find a -match. - -*`when`*:: A condition that must succeed in order to execute the current rule. -ifndef::no-processors[] -All the <> supported by processors are also supported -here. -endif::no-processors[] - -The following example sets the topic based on whether the message field contains -the specified string: - -["source","yaml",subs="attributes"] ------------------------------------------------------------------------------- -output.kafka: - hosts: ["localhost:9092"] - topic: "logs-%{[beat.version]}" - topics: - - topic: "critical-%{[beat.version]}" - when.contains: - message: "CRITICAL" - - topic: "error-%{[beat.version]}" - when.contains: - message: "ERR" ------------------------------------------------------------------------------- - - -This configuration results in topics named +critical-{version}+, -+error-{version}+, and +logs-{version}+. - -===== `key` - -Optional Kafka event key. If configured, the event key must be unique and can be extracted from the event using a format string. - -===== `partition` - -Kafka output broker event partitioning strategy. Must be one of `random`, -`round_robin`, or `hash`. By default the `hash` partitioner is used. - -*`random.group_events`*: Sets the number of events to be published to the same - partition, before the partitioner selects a new partition by random. The - default value is 1 meaning after each event a new partition is picked randomly. - -*`round_robin.group_events`*: Sets the number of events to be published to the - same partition, before the partitioner selects the next partition. The default - value is 1 meaning after each event the next partition will be selected. - -*`hash.hash`*: List of fields used to compute the partitioning hash value from. - If no field is configured, the events `key` value will be used. - -*`hash.random`*: Randomly distribute events if no hash or key value can be computed. - -All partitioners will try to publish events to all partitions by default. If a -partition's leader becomes unreachable for the beat, the output might block. All -partitioners support setting `reachable_only` to overwrite this -behavior. If `reachable_only` is set to `true`, events will be published to -available partitions only. - -NOTE: Publishing to a subset of available partitions potentially increases resource usage because events may become unevenly distributed. - -===== `client_id` - -The configurable ClientID used for logging, debugging, and auditing purposes. The default is "beats". - -===== `worker` - -The number of concurrent load-balanced Kafka output workers. - -===== `codec` - -Output codec configuration. If the `codec` section is missing, events will be json encoded. - -See <> for more information. - -===== `metadata` - -Kafka metadata update settings. The metadata do contain information about -brokers, topics, partition, and active leaders to use for publishing. - -*`refresh_frequency`*:: Metadata refresh interval. Defaults to 10 minutes. - -*`full`*:: Strategy to use when fetching metadata, when this option is `true`, the client will maintain -a full set of metadata for all the available topics, if the this option is set to `false` it will only refresh the -metadata for the configured topics. The default is false. - -*`retry.max`*:: Total number of metadata update retries when cluster is in middle of leader election. The default is 3. - -*`retry.backoff`*:: Waiting time between retries during leader elections. Default is 250ms. - -===== `max_retries` - -ifdef::ignores_max_retries[] -{beatname_uc} ignores the `max_retries` setting and retries indefinitely. -endif::[] - -ifndef::ignores_max_retries[] -The number of times to retry publishing an event after a publishing failure. -After the specified number of retries, the events are typically dropped. - -Set `max_retries` to a value less than 0 to retry until all events are published. - -The default is 3. -endif::[] - -===== `bulk_max_size` - -The maximum number of events to bulk in a single Kafka request. The default is 2048. - -===== `bulk_flush_frequency` - -Duration to wait before sending bulk Kafka request. 0 is no delay. The default is 0. - -===== `timeout` - -The number of seconds to wait for responses from the Kafka brokers before timing -out. The default is 30 (seconds). - -===== `broker_timeout` - -The maximum duration a broker will wait for number of required ACKs. The default is 10s. - -===== `channel_buffer_size` - -Per Kafka broker number of messages buffered in output pipeline. The default is 256. - -===== `keep_alive` - -The keep-alive period for an active network connection. If 0s, keep-alives are disabled. The default is 0 seconds. - -===== `compression` - -Sets the output compression codec. Must be one of `none`, `snappy`, `lz4` and `gzip`. The default is `gzip`. - -===== `compression_level` - -Sets the compression level used by gzip. Setting this value to 0 disables compression. -The compression level must be in the range of 1 (best speed) to 9 (best compression). - -Increasing the compression level will reduce the network usage but will increase the cpu usage. - -The default value is 4. - -[[kafka-max_message_bytes]] -===== `max_message_bytes` - -The maximum permitted size of JSON-encoded messages. Bigger messages will be dropped. The default value is 1000000 (bytes). This value should be equal to or less than the broker's `message.max.bytes`. - -===== `required_acks` - -The ACK reliability level required from broker. 0=no response, 1=wait for local commit, -1=wait for all replicas to commit. The default is 1. - -Note: If set to 0, no ACKs are returned by Kafka. Messages might be lost silently on error. - -===== `ssl` - -Configuration options for SSL parameters like the root CA for Kafka connections. - The Kafka host keystore should be created with the -`-keyalg RSA` argument to ensure it uses a cipher supported by -https://github.com/Shopify/sarama/wiki/Frequently-Asked-Questions#why-cant-sarama-connect-to-my-kafka-cluster-using-ssl[Filebeat's Kafka library]. -See <> for more information. - -//begin inner exclude for redis -ifndef::no-redis-output[] -[[redis-output]] -=== Configure the Redis output - -++++ -Redis -++++ - -The Redis output inserts the events into a Redis list or a Redis channel. -This output plugin is compatible with -the https://www.elastic.co/guide/en/logstash/current/plugins-inputs-redis.html[Redis input plugin] for Logstash. - -Example configuration: - -["source","yaml",subs="attributes"] ------------------------------------------------------------------------------- -output.redis: - hosts: ["localhost"] - password: "my_password" - key: "{beatname_lc}" - db: 0 - timeout: 5 ------------------------------------------------------------------------------- - -==== Compatibility - -This output works with Redis 3.2.4. - -==== Configuration options - -You can specify the following options in the `redis` section of the +{beatname_lc}.yml+ config file: - -===== `enabled` - -The enabled config is a boolean setting to enable or disable the output. If set -to false, the output is disabled. - -The default value is true. - -===== `hosts` - -The list of Redis servers to connect to. If load balancing is enabled, the events are -distributed to the servers in the list. If one server becomes unreachable, the events are -distributed to the reachable servers only. You can define each Redis server by specifying -`HOST` or `HOST:PORT`. For example: `"192.15.3.2"` or `"test.redis.io:12345"`. If you -don't specify a port number, the value configured by `port` is used. - -===== `index` - -The index name added to the events metadata for use by Logstash. The default is "{beatname_lc}". - -[[key-option-redis]] -===== `key` - -The name of the Redis list or channel the events are published to. If not -configured, the value of the `index` setting is used. - -You can set the key dynamically by using a format string to access any event -field. For example, this configuration uses a custom field, `fields.list`, to -set the Redis list key. If `fields.list` is missing, `fallback` is used: - -["source","yaml"] ------------------------------------------------------------------------------- -output.redis: - hosts: ["localhost"] - key: "%{[fields.list]:fallback}" ------------------------------------------------------------------------------- - - -TIP: To learn how to add custom fields to events, see the -<> option. - -See the <> setting for other ways to set the key -dynamically. - -[[keys-option-redis]] -===== `keys` - -An array of key selector rules. Each rule specifies the `key` to use for events -that match the rule. During publishing, {beatname_uc} uses the first matching -rule in the array. Rules can contain conditionals, format string-based fields, -and name mappings. If the `keys` setting is missing or no rule matches, the -<> setting is used. - -Rule settings: - -*`index`*:: The key format string to use. If this string contains field -references, such as `%{[fields.name]}`, the fields must exist, or the rule -fails. - -*`mappings`*:: A dictionary that takes the value returned by `key` and maps it to -a new name. - -*`default`*:: The default string value to use if `mappings` does not find a match. - -*`when`*:: A condition that must succeed in order to execute the current rule. -ifndef::no-processors[] -All the <> supported by processors are also supported -here. -endif::no-processors[] - -Example `keys` settings: - -["source","yaml"] ------------------------------------------------------------------------------- -output.redis: - hosts: ["localhost"] - key: "default_list" - keys: - - key: "info_list" # send to info_list if `message` field contains INFO - when.contains: - message: "INFO" - - key: "debug_list" # send to debug_list if `message` field contains DEBUG - when.contains: - message: "DEBUG" - - key: "%{[fields.list]}" - mappings: - http: "frontend_list" - nginx: "frontend_list" - mysql: "backend_list" ------------------------------------------------------------------------------- - -===== `password` - -The password to authenticate with. The default is no authentication. - -===== `db` - -The Redis database number where the events are published. The default is 0. - -===== `datatype` - -The Redis data type to use for publishing events.If the data type is `list`, the -Redis RPUSH command is used and all events are added to the list with the key defined under `key`. -If the data type `channel` is used, the Redis `PUBLISH` command is used and means that all events -are pushed to the pub/sub mechanism of Redis. The name of the channel is the one defined under `key`. -The default value is `list`. - -===== `codec` - -Output codec configuration. If the `codec` section is missing, events will be json encoded. - -See <> for more information. - -===== `host_topology` - -deprecated[5.0.0] - -The Redis host to connect to when using topology map support. Topology map support is disabled if this option is not set. - -===== `password_topology` - -deprecated[5.0.0] - -The password to use for authenticating with the Redis topology server. The default is no authentication. - -===== `db_topology` - -deprecated[5.0.0] - -The Redis database number where the topology information is stored. The default is 1. - -===== `worker` - -The number of workers to use for each host configured to publish events to Redis. Use this setting along with the -`loadbalance` option. For example, if you have 2 hosts and 3 workers, in total 6 workers are started (3 for each host). - -===== `loadbalance` - -If set to true and multiple hosts or workers are configured, the output plugin load balances published events onto all -Redis hosts. If set to false, the output plugin sends all events to only one host (determined at random) and will switch -to another host if the currently selected one becomes unreachable. The default value is true. - -===== `timeout` - -The Redis connection timeout in seconds. The default is 5 seconds. - -===== `backoff.init` - -The number of seconds to wait before trying to reconnect to Redis after -a network error. After waiting `backoff.init` seconds, {beatname_uc} tries to -reconnect. If the attempt fails, the backoff timer is increased exponentially up -to `backoff.max`. After a successful connection, the backoff timer is reset. The -default is 1s. - -===== `backoff.max` - -The maximum number of seconds to wait before attempting to connect to -Redis after a network error. The default is 60s. - -===== `max_retries` - -ifdef::ignores_max_retries[] -{beatname_uc} ignores the `max_retries` setting and retries indefinitely. -endif::[] - -ifndef::ignores_max_retries[] -The number of times to retry publishing an event after a publishing failure. -After the specified number of retries, the events are typically dropped. - -Set `max_retries` to a value less than 0 to retry until all events are published. - -The default is 3. -endif::[] - - -===== `bulk_max_size` - -The maximum number of events to bulk in a single Redis request or pipeline. The default is 2048. - -If the Beat sends single events, the events are collected into batches. If the -Beat publishes a large batch of events (larger than the value specified by -`bulk_max_size`), the batch is split. - -Specifying a larger batch size can improve performance by lowering the overhead -of sending events. However big batch sizes can also increase processing times, -which might result in API errors, killed connections, timed-out publishing -requests, and, ultimately, lower throughput. - -Setting `bulk_max_size` to values less than or equal to 0 disables the -splitting of batches. When splitting is disabled, the queue decides on the -number of events to be contained in a batch. - -===== `ssl` - -Configuration options for SSL parameters like the root CA for Redis connections -guarded by SSL proxies (for example https://www.stunnel.org[stunnel]). See -<> for more information. - -===== `proxy_url` - -The URL of the SOCKS5 proxy to use when connecting to the Redis servers. The -value must be a URL with a scheme of `socks5://`. You cannot use a web proxy -because the protocol used to communicate with Redis is not based on HTTP. - -If the SOCKS5 proxy server requires client authentication, you can embed -a username and password in the URL. - -When using a proxy, hostnames are resolved on the proxy server instead of on the -client. You can change this behavior by setting the -<> option. - -[[redis-proxy-use-local-resolver]] -===== `proxy_use_local_resolver` - -This option determines whether Redis hostnames are resolved locally when using a proxy. -The default value is false, which means that name resolution occurs on the proxy server. - -//end inner exclude for redis -endif::[] - -[[file-output]] -=== Configure the File output - -++++ -File -++++ - -The File output dumps the transactions into a file where each transaction is in a JSON format. -Currently, this output is used for testing, but it can be used as input for -Logstash. - -["source","yaml",subs="attributes"] ------------------------------------------------------------------------------- -output.file: - path: "/tmp/{beatname_lc}" - filename: {beatname_lc} - #rotate_every_kb: 10000 - #number_of_files: 7 - #permissions: 0600 ------------------------------------------------------------------------------- - -==== Configuration options - -You can specify the following options in the `file` section of the +{beatname_lc}.yml+ config file: - -===== `enabled` - -The enabled config is a boolean setting to enable or disable the output. If set -to false, the output is disabled. - -The default value is true. - -[[path]] -===== `path` - -The path to the directory where the generated files will be saved. This option is -mandatory. - -===== `filename` - -The name of the generated files. The default is set to the Beat name. For example, the files -generated by default for {beatname_uc} would be "{beatname_lc}", "{beatname_lc}.1", "{beatname_lc}.2", and so on. - -===== `rotate_every_kb` - -The maximum size in kilobytes of each file. When this size is reached, the files are -rotated. The default value is 10240 KB. - -===== `number_of_files` - -The maximum number of files to save under <>. When this number of files is reached, the -oldest file is deleted, and the rest of the files are shifted from last to first. -The number of files must be between 2 and 1024. The default is 7. - -===== `permissions` - -Permissions to use for file creation. The default is 0600. - -===== `codec` - -Output codec configuration. If the `codec` section is missing, events will be json encoded. - -See <> for more information. - -[[console-output]] -=== Configure the Console output - -++++ -Console -++++ - -The Console output writes events in JSON format to stdout. - -[source,yaml] ------------------------------------------------------------------------------- -output.console: - pretty: true ------------------------------------------------------------------------------- - -==== Configuration options - -You can specify the following options in the `console` section of the +{beatname_lc}.yml+ config file: - -===== `pretty` - -If `pretty` is set to true, events written to stdout will be nicely formatted. The default is false. - -===== `codec` +include::outputs/output-elasticsearch.asciidoc[] -Output codec configuration. If the `codec` section is missing, events will be json encoded using the `pretty` option. +include::outputs/output-logstash.asciidoc[] -See <> for more information. +include::outputs/output-kafka.asciidoc[] +include::outputs/output-redis.asciidoc[] -===== `enabled` +include::outputs/output-file.asciidoc[] -The enabled config is a boolean setting to enable or disable the output. If set -to false, the output is disabled. +include::outputs/output-console.asciidoc[] -The default value is true. +include::outputs/output-cloud.asciidoc[] -===== `bulk_max_size` - -The maximum number of events to buffer internally during publishing. The default is 2048. - -Specifying a larger batch size may add some latency and buffering during publishing. However, for Console output, this -setting does not affect how events are published. - -Setting `bulk_max_size` to values less than or equal to 0 disables the -splitting of batches. When splitting is disabled, the queue decides on the -number of events to be contained in a batch. - -//end outer exclude for ls, kafka, redis, file, and console -endif::[] - -[[configure-cloud-id]] -=== Configure the output for the Elastic Cloud - -++++ -Cloud -++++ - -ifdef::apm-server[] -NOTE: This page refers to using a separate instance of APM Server with an existing -https://www.elastic.co/cloud/elasticsearch-service[Elasticsearch Service deployment]. -If you want to use APM on Elastic Cloud, see the cloud docs: -{cloud}/ec-create-deployment.html[Create your deployment] or -{cloud}/ec-manage-apm-settings.html[Add APM user settings]. -endif::apm-server[] - -{beatname_uc} comes with two settings that simplify the output configuration -when used together with https://cloud.elastic.co/[Elastic Cloud]. When defined, -these setting overwrite settings from other parts in the configuration. - -Example: - -["source","yaml",subs="attributes"] ------------------------------------------------------------------------------- -cloud.id: "staging:dXMtZWFzdC0xLmF3cy5mb3VuZC5pbyRjZWM2ZjI2MWE3NGJmMjRjZTMzYmI4ODExYjg0Mjk0ZiRjNmMyY2E2ZDA0MjI0OWFmMGNjN2Q3YTllOTYyNTc0Mw==" -cloud.auth: "elastic:{pwd}" ------------------------------------------------------------------------------- - -These settings can be also specified at the command line, like this: - - -["source","sh",subs="attributes"] ------------------------------------------------------------------------------- -{beatname_lc} -e -E cloud.id="" -E cloud.auth="" ------------------------------------------------------------------------------- - - -==== `cloud.id` - -The Cloud ID, which can be found in the Elastic Cloud web console, is used by -{beatname_uc} to resolve the Elasticsearch and Kibana URLs. This setting -overwrites the `output.elasticsearch.hosts` and `setup.kibana.host` settings. - -==== `cloud.auth` - -When specified, the `cloud.auth` overwrites the `output.elasticsearch.username` and -`output.elasticsearch.password` settings. Because the Kibana settings inherit -the username and password from the Elasticsearch output, this can also be used -to set the `setup.kibana.username` and `setup.kibana.password` options. - -//begin exclude for output codec -ifndef::only-elasticsearch[] - -[[configuration-output-codec]] -=== Change the output codec - -For outputs that do not require a specific encoding, you can change the encoding -by using the codec configuration. You can specify either the `json` or `format` -codec. By default the `json` codec is used. - -*`json.pretty`*: If `pretty` is set to true, events will be nicely formatted. The default is false. - -*`json.escape_html`*: If `escape_html` is set to true, html symbols will be escaped in strings. The default is false. - -Example configuration that uses the `json` codec with pretty printing enabled to write events to the console: - -[source,yaml] ------------------------------------------------------------------------------- -output.console: - codec.json: - pretty: true - escape_html: false ------------------------------------------------------------------------------- - -*`format.string`*: Configurable format string used to create a custom formatted message. - -Example configurable that uses the `format` codec to print the events timestamp and message field to console: - -[source,yaml] ------------------------------------------------------------------------------- -output.console: - codec.format: - string: '%{[@timestamp]} %{[message]}' ------------------------------------------------------------------------------- - -//end exclude for output codec -endif::[] +include::outputs/change-output-codec.asciidoc[] diff --git a/docs/copied-from-beats/outputs/change-output-codec.asciidoc b/docs/copied-from-beats/outputs/change-output-codec.asciidoc new file mode 100644 index 00000000000..64068cbb971 --- /dev/null +++ b/docs/copied-from-beats/outputs/change-output-codec.asciidoc @@ -0,0 +1,31 @@ +[[configuration-output-codec]] +=== Change the output codec + +For outputs that do not require a specific encoding, you can change the encoding +by using the codec configuration. You can specify either the `json` or `format` +codec. By default the `json` codec is used. + +*`json.pretty`*: If `pretty` is set to true, events will be nicely formatted. The default is false. + +*`json.escape_html`*: If `escape_html` is set to true, html symbols will be escaped in strings. The default is false. + +Example configuration that uses the `json` codec with pretty printing enabled to write events to the console: + +[source,yaml] +------------------------------------------------------------------------------ +output.console: + codec.json: + pretty: true + escape_html: false +------------------------------------------------------------------------------ + +*`format.string`*: Configurable format string used to create a custom formatted message. + +Example configurable that uses the `format` codec to print the events timestamp and message field to console: + +[source,yaml] +------------------------------------------------------------------------------ +output.console: + codec.format: + string: '%{[@timestamp]} %{[message]}' +------------------------------------------------------------------------------ diff --git a/docs/copied-from-beats/outputs/output-cloud.asciidoc b/docs/copied-from-beats/outputs/output-cloud.asciidoc new file mode 100644 index 00000000000..d8347ef8770 --- /dev/null +++ b/docs/copied-from-beats/outputs/output-cloud.asciidoc @@ -0,0 +1,49 @@ +[[configure-cloud-id]] +=== Configure the output for the Elastic Cloud + +++++ +Cloud +++++ + +ifdef::apm-server[] +NOTE: This page refers to using a separate instance of APM Server with an existing +https://www.elastic.co/cloud/elasticsearch-service[Elasticsearch Service deployment]. +If you want to use APM on Elastic Cloud, see the cloud docs: +{cloud}/ec-create-deployment.html[Create your deployment] or +{cloud}/ec-manage-apm-settings.html[Add APM user settings]. +endif::apm-server[] + +{beatname_uc} comes with two settings that simplify the output configuration +when used together with https://cloud.elastic.co/[Elastic Cloud]. When defined, +these setting overwrite settings from other parts in the configuration. + +Example: + +["source","yaml",subs="attributes"] +------------------------------------------------------------------------------ +cloud.id: "staging:dXMtZWFzdC0xLmF3cy5mb3VuZC5pbyRjZWM2ZjI2MWE3NGJmMjRjZTMzYmI4ODExYjg0Mjk0ZiRjNmMyY2E2ZDA0MjI0OWFmMGNjN2Q3YTllOTYyNTc0Mw==" +cloud.auth: "elastic:{pwd}" +------------------------------------------------------------------------------ + +These settings can be also specified at the command line, like this: + + +["source","sh",subs="attributes"] +------------------------------------------------------------------------------ +{beatname_lc} -e -E cloud.id="" -E cloud.auth="" +------------------------------------------------------------------------------ + + +==== `cloud.id` + +The Cloud ID, which can be found in the Elastic Cloud web console, is used by +{beatname_uc} to resolve the Elasticsearch and Kibana URLs. This setting +overwrites the `output.elasticsearch.hosts` and `setup.kibana.host` settings. + +==== `cloud.auth` + +When specified, the `cloud.auth` overwrites the `output.elasticsearch.username` and +`output.elasticsearch.password` settings. Because the Kibana settings inherit +the username and password from the Elasticsearch output, this can also be used +to set the `setup.kibana.username` and `setup.kibana.password` options. + diff --git a/docs/copied-from-beats/outputs/output-console.asciidoc b/docs/copied-from-beats/outputs/output-console.asciidoc new file mode 100644 index 00000000000..102eb13c7c8 --- /dev/null +++ b/docs/copied-from-beats/outputs/output-console.asciidoc @@ -0,0 +1,47 @@ +[[console-output]] +=== Configure the Console output + +++++ +Console +++++ + +The Console output writes events in JSON format to stdout. + +[source,yaml] +------------------------------------------------------------------------------ +output.console: + pretty: true +------------------------------------------------------------------------------ + +==== Configuration options + +You can specify the following options in the `console` section of the +{beatname_lc}.yml+ config file: + +===== `pretty` + +If `pretty` is set to true, events written to stdout will be nicely formatted. The default is false. + +===== `codec` + +Output codec configuration. If the `codec` section is missing, events will be json encoded using the `pretty` option. + +See <> for more information. + + +===== `enabled` + +The enabled config is a boolean setting to enable or disable the output. If set +to false, the output is disabled. + +The default value is true. + +===== `bulk_max_size` + +The maximum number of events to buffer internally during publishing. The default is 2048. + +Specifying a larger batch size may add some latency and buffering during publishing. However, for Console output, this +setting does not affect how events are published. + +Setting `bulk_max_size` to values less than or equal to 0 disables the +splitting of batches. When splitting is disabled, the queue decides on the +number of events to be contained in a batch. diff --git a/docs/copied-from-beats/outputs/output-elasticsearch.asciidoc b/docs/copied-from-beats/outputs/output-elasticsearch.asciidoc new file mode 100644 index 00000000000..1296133c982 --- /dev/null +++ b/docs/copied-from-beats/outputs/output-elasticsearch.asciidoc @@ -0,0 +1,657 @@ +[[elasticsearch-output]] +=== Configure the Elasticsearch output + +++++ +Elasticsearch +++++ + +When you specify Elasticsearch for the output, {beatname_uc} sends the transactions directly to Elasticsearch by using the Elasticsearch HTTP API. + +Example configuration: + +["source","yaml",subs="attributes"] +------------------------------------------------------------------------------ + +output.elasticsearch: + hosts: ["https://localhost:9200"] + index: "{beat_default_index_prefix}-%{[{beat_version_key}]}-%{+yyyy.MM.dd}" + ssl.certificate_authorities: ["/etc/pki/root/ca.pem"] + ssl.certificate: "/etc/pki/client/cert.pem" + ssl.key: "/etc/pki/client/cert.key" +------------------------------------------------------------------------------ + +To enable SSL, just add `https` to all URLs defined under __hosts__. + +["source","yaml",subs="attributes,callouts"] +------------------------------------------------------------------------------ + +output.elasticsearch: + hosts: ["https://localhost:9200"] + username: "{beatname_lc}_internal" + password: "{pwd}" +------------------------------------------------------------------------------ + +If the Elasticsearch nodes are defined by `IP:PORT`, then add `protocol: https` to the yaml file. + +["source","yaml",subs="attributes,callouts"] +------------------------------------------------------------------------------ +output.elasticsearch: + hosts: ["localhost"] + protocol: "https" + username: "{beatname_lc}_internal" + password: "{pwd}" +------------------------------------------------------------------------------ + + +For more information about securing {beatname_uc}, see +<>. + +ifndef::no_ilm[] +If you are indexing large amounts of time-series data, you might also want to +configure {beatname_uc} to use index lifecycle management. For more information +about configuring and using index lifecycle management with {beatname_uc}, see +<>. +endif::no_ilm[] + +==== Compatibility + +This output works with all compatible versions of Elasticsearch. See the +https://www.elastic.co/support/matrix#matrix_compatibility[Elastic Support +Matrix]. + +==== Configuration options + +You can specify the following options in the `elasticsearch` section of the +{beatname_lc}.yml+ config file: + +===== `enabled` + +The enabled config is a boolean setting to enable or disable the output. If set +to false, the output is disabled. + +The default value is true. + + +[[hosts-option]] +===== `hosts` + +The list of Elasticsearch nodes to connect to. The events are distributed to +these nodes in round robin order. If one node becomes unreachable, the event is +automatically sent to another node. Each Elasticsearch node can be defined as a `URL` or `IP:PORT`. +For example: `http://192.15.3.2`, `https://es.found.io:9230` or `192.24.3.2:9300`. +If no port is specified, `9200` is used. + +NOTE: When a node is defined as an `IP:PORT`, the _scheme_ and _path_ are taken from the +<> and <> config options. + +[source,yaml] +------------------------------------------------------------------------------ +output.elasticsearch: + hosts: ["10.45.3.2:9220", "10.45.3.1:9230"] + protocol: https + path: /elasticsearch +------------------------------------------------------------------------------ + +In the previous example, the Elasticsearch nodes are available at `https://10.45.3.2:9220/elasticsearch` and +`https://10.45.3.1:9230/elasticsearch`. + +===== `compression_level` + +The gzip compression level. Setting this value to 0 disables compression. +The compression level must be in the range of 1 (best speed) to 9 (best compression). + +Increasing the compression level will reduce the network usage but will increase the cpu usage. + +The default value is 0. + +===== `escape_html` + +Configure escaping of HTML in strings. Set to `true` to enable escaping. + +The default value is `false`. + + +===== `worker` + +The number of workers per configured host publishing events to Elasticsearch. This +is best used with load balancing mode enabled. Example: If you have 2 hosts and +3 workers, in total 6 workers are started (3 for each host). + +The default value is 1. + +===== `username` + +The basic authentication username for connecting to Elasticsearch. + +===== `password` + +The basic authentication password for connecting to Elasticsearch. + +===== `parameters` + +Dictionary of HTTP parameters to pass within the url with index operations. + +[[protocol-option]] +===== `protocol` + +The name of the protocol Elasticsearch is reachable on. The options are: +`http` or `https`. The default is `http`. However, if you specify a URL for +<>, the value of `protocol` is overridden by whatever scheme you +specify in the URL. + +[[path-option]] +===== `path` + +An HTTP path prefix that is prepended to the HTTP API calls. This is useful for +the cases where Elasticsearch listens behind an HTTP reverse proxy that exports +the API under a custom prefix. + +===== `headers` + +Custom HTTP headers to add to each request created by the Elasticsearch output. +Example: + +[source,yaml] +------------------------------------------------------------------------------ +output.elasticsearch.headers: + X-My-Header: Header contents +------------------------------------------------------------------------------ + +It is generally possible to specify multiple header values for the same header +name by separating them with a comma. + +===== `proxy_url` + +The URL of the proxy to use when connecting to the Elasticsearch servers. The +value may be either a complete URL or a "host[:port]", in which case the "http" +scheme is assumed. If a value is not specified through the configuration file +then proxy environment variables are used. See the +https://golang.org/pkg/net/http/#ProxyFromEnvironment[Go documentation] +for more information about the environment variables. + +[[index-option-es]] +===== `index` + +ifndef::apm-server[] +The index name to write events to when you're using daily indices. The default is ++"{beatname_lc}-%{[{beat_version_key}]}-%{+yyyy.MM.dd}"+ (for example, ++"{beatname_lc}-{version}-{localdate}"+). If you change this setting, you also +need to configure the `setup.template.name` and `setup.template.pattern` options +(see <>). +endif::apm-server[] + +ifdef::apm-server[] +The index name to write events to. The default is ++"apm-%{[{beat_version_key}]}-{type}-%{+yyyy.MM.dd}"+ (for example, ++"apm-{version}-transaction-{localdate}"+). See +<> for more information on +default index configuration. + +IMPORTANT: If you change this setting, +you need to configure the `setup.template.name` and `setup.template.pattern` options +(see <>). You also must set the default index configuration +in the `apm-server.yml` file. + +NOTE: +{beat_version_key}+ is a field managed by Beats that is added to every document. +It holds the current version of APM Server. +endif::apm-server[] + +ifndef::no_dashboards[] +If you are using the pre-built Kibana +dashboards, you also need to set the `setup.dashboards.index` option (see +<>). +endif::no_dashboards[] + +ifndef::apm-server[] +ifndef::no_ilm[] +The `index` setting is ignored when index lifecycle management is enabled. If +you’re sending events to a cluster that supports index lifecycle management, see +<> to learn how to change the index name. +endif::no_ilm[] + +You can set the index dynamically by using a format string to access any event +field. For example, this configuration uses a custom field, `fields.log_type`, +to set the index: + +["source","yaml",subs="attributes"] +------------------------------------------------------------------------------ +output.elasticsearch: + hosts: ["http://localhost:9200"] + index: "%{[fields.log_type]}-%{[{beat_version_key}]}-%{+yyyy.MM.dd}" <1> +------------------------------------------------------------------------------ + +<1> We recommend including +{beat_version_key}+ in the name to avoid mapping issues +when you upgrade. + +With this configuration, all events with `log_type: normal` are sent to an +index named +normal-{version}-{localdate}+, and all events with +`log_type: critical` are sent to an index named ++critical-{version}-{localdate}+. +endif::apm-server[] + +ifdef::apm-server[] +You can set the index dynamically by using a format string to access any event +field. For example, this configuration uses the field, `processor.event`, +to set the index: + +["source","yaml",subs="attributes"] +------------------------------------------------------------------------------ +output.elasticsearch: + hosts: ["http://localhost:9200"] + index: "apm-%{[observer.version]}-%{[processor.event]}-%{+yyyy.MM.dd}\" <1> +------------------------------------------------------------------------------ + +<1> `observer` refers to {beatname_uc}. We recommend including ++{beat_version_key}+ in the name to avoid mapping issues when you upgrade +{beatname_uc}. + +With this configuration, +all events are separated by their `processor.event` into different indices. +endif::apm-server[] + +TIP: To learn how to add custom fields to events, see the +<> option. + +See the <> setting for other ways to set the index +dynamically. + + +[[indices-option-es]] +===== `indices` + +An array of index selector rules. Each rule specifies the index to use for +events that match the rule. During publishing, {beatname_uc} uses the first +matching rule in the array. Rules can contain conditionals, format string-based +fields, and name mappings. If the `indices` setting is missing or no rule +matches, the <> setting is used. + +Rule settings: + +*`index`*:: The index format string to use. If this string contains field +references, such as `%{[fields.name]}`, the fields must exist, or the rule fails. + +*`mappings`*:: A dictionary that takes the value returned by `index` and maps it +to a new name. + +*`default`*:: The default string value to use if `mappings` does not find a +match. + +*`when`*:: A condition that must succeed in order to execute the current rule. +ifndef::no-processors[] +All the <> supported by processors are also supported +here. +endif::no-processors[] + +ifndef::apm-server[] +The following example sets the index based on whether the `message` field +contains the specified string: + +["source","yaml",subs="attributes"] +------------------------------------------------------------------------------ +output.elasticsearch: + hosts: ["http://localhost:9200"] + indices: + - index: "warning-%{[{beat_version_key}]}-%{+yyyy.MM.dd}" + when.contains: + message: "WARN" + - index: "error-%{[{beat_version_key}]}-%{+yyyy.MM.dd}" + when.contains: + message: "ERR" +------------------------------------------------------------------------------ + + +This configuration results in indices named +warning-{version}-{localdate}+ +and +error-{version}-{localdate}+ (plus the default index if no matches are +found). + +The following example sets the index by taking the name returned by the `index` +format string and mapping it to a new name that's used for the index: + +["source","yaml"] +------------------------------------------------------------------------------ +output.elasticsearch: + hosts: ["http://localhost:9200"] + indices: + - index: "%{[fields.log_type]}" + mappings: + critical: "sev1" + normal: "sev2" + default: "sev3" +------------------------------------------------------------------------------ + + +This configuration results in indices named `sev1`, `sev2`, and `sev3`. + +The `mappings` setting simplifies the configuration, but is limited to string +values. You cannot specify format strings within the mapping pairs. +endif::apm-server[] + +ifdef::apm-server[] +The following example sets the index based on whether the `processor.event` field +contains the specified string: + +["source","yaml",subs="attributes"] +------------------------------------------------------------------------------ +output.elasticsearch: + hosts: ["http://localhost:9200"] + indices: + - index: "apm-%{[observer.version]}-sourcemap" + when.contains: + processor.event: "sourcemap" + + - index: "apm-%{[observer.version]}-error-%{+yyyy.MM.dd}" + when.contains: + processor.event: "error" + + - index: "apm-%{[observer.version]}-transaction-%{+yyyy.MM.dd}" + when.contains: + processor.event: "transaction" + + - index: "apm-%{[observer.version]}-span-%{+yyyy.MM.dd}" + when.contains: + processor.event: "span" + + - index: "apm-%{[observer.version]}-metric-%{+yyyy.MM.dd}" + when.contains: + processor.event: "metric" + + - index: "apm-%{[observer.version]}-onboarding-%{+yyyy.MM.dd}" + when.contains: + processor.event: "onboarding" +------------------------------------------------------------------------------ + +NOTE: `observer` refers to {beatname_uc}. We recommend including ++{beat_version_key}+ in the name to avoid mapping issues when you upgrade +{beatname_uc}. + +This is the default configuration for {beatname_uc} and results in indices +named in the following format: +"apm-%{[{beat_version_key}]}-{type}-%{+yyyy.MM.dd}"+ +For example: +"apm-{version}-transaction-{localdate}"+. + +The following example sets the index by taking the name returned by the `index` +format string and mapping it to a new name that's used for the index: + +["source","yaml"] +------------------------------------------------------------------------------ +output.elasticsearch: + hosts: ["http://localhost:9200"] + indices: + - index: "%{[processor.event]}" + mappings: + sourcemap: "apm-sourcemap" + error: "apm-error" + transaction: "apm-transaction" + span: "apm-span" + metric: "apm-metric" + onboarding: "apm-onboarding" + default: "apm" +------------------------------------------------------------------------------ + +This configuration results in indices named `apm-sourcemap`, `apm-error`, etc. + +The `mappings` setting simplifies the configuration, but is limited to string +values. You cannot specify format strings within the mapping pairs. +endif::apm-server[] + +//TODO: MOVE ILM OPTIONS TO APPEAR LOGICALLY BASED ON LOCATION IN THE YAML FILE. + +ifndef::no_ilm[] +[[ilm-es]] +===== `ilm` + +Configuration options for index lifecycle management. + +See <> for more information. +endif::no_ilm[] + +ifndef::no-pipeline[] +[[pipeline-option-es]] +===== `pipeline` + +A format string value that specifies the ingest node pipeline to write events to. + +["source","yaml"] +------------------------------------------------------------------------------ +output.elasticsearch: + hosts: ["http://localhost:9200"] + pipeline: my_pipeline_id +------------------------------------------------------------------------------ + +For more information, see <>. + +ifndef::apm-server[] +You can set the ingest node pipeline dynamically by using a format string to +access any event field. For example, this configuration uses a custom field, +`fields.log_type`, to set the pipeline for each event: + +["source","yaml",subs="attributes"] +------------------------------------------------------------------------------ +output.elasticsearch: + hosts: ["http://localhost:9200"] + pipeline: "%{[fields.log_type]}_pipeline" +------------------------------------------------------------------------------ + + +With this configuration, all events with `log_type: normal` are sent to a pipeline +named `normal_pipeline`, and all events with `log_type: critical` are sent to a +pipeline named `critical_pipeline`. +endif::apm-server[] + +ifdef::apm-server[] +You can set the ingest node pipeline dynamically by using a format string to +access any event field. For example, this configuration uses the field, +`processor.event`, to set the pipeline for each event: + +["source","yaml",subs="attributes"] +------------------------------------------------------------------------------ +output.elasticsearch: + hosts: ["http://localhost:9200"] + pipeline: "%{[processor.event]}_pipeline" +------------------------------------------------------------------------------ + + +With this configuration, all events with `processor.event: transaction` are sent to a pipeline +named `transaction_pipeline`. Similarly, all events with `processor.event: error` are sent to a +pipeline named `error_pipeline`. + +The default pipeline is `apm`. It adds user agent and geo ip information to events. +To disable this, or any other pipeline, set `output.elasticsearch.pipeline: _none`. +endif::apm-server[] + +TIP: To learn how to add custom fields to events, see the +<> option. + +See the <> setting for other ways to set the +ingest node pipeline dynamically. + +[[pipelines-option-es]] +===== `pipelines` + +An array of pipeline selector rules. Each rule specifies the ingest node +pipeline to use for events that match the rule. During publishing, {beatname_uc} +uses the first matching rule in the array. Rules can contain conditionals, +format string-based fields, and name mappings. If the `pipelines` setting is +missing or no rule matches, the <> setting is +used. + +Rule settings: + +*`pipeline`*:: The pipeline format string to use. If this string contains field +references, such as `%{[fields.name]}`, the fields must exist, or the rule +fails. + +*`mappings`*:: A dictionary that takes the value returned by `pipeline` and maps +it to a new name. + +*`default`*:: The default string value to use if `mappings` does not find a +match. + +*`when`*:: A condition that must succeed in order to execute the current rule. +ifndef::no-processors[] +All the <> supported by processors are also supported +here. +endif::no-processors[] + +ifndef::apm-server[] +The following example sends events to a specific pipeline based on whether the +`message` field contains the specified string: + +["source","yaml"] +------------------------------------------------------------------------------ +output.elasticsearch: + hosts: ["http://localhost:9200"] + pipelines: + - pipeline: "warning_pipeline" + when.contains: + message: "WARN" + - pipeline: "error_pipeline" + when.contains: + message: "ERR" +------------------------------------------------------------------------------ + + +The following example sets the pipeline by taking the name returned by the +`pipeline` format string and mapping it to a new name that's used for the +pipeline: + +["source","yaml"] +------------------------------------------------------------------------------ +output.elasticsearch: + hosts: ["http://localhost:9200"] + pipelines: + - pipeline: "%{[fields.log_type]}" + mappings: + critical: "sev1_pipeline" + normal: "sev2_pipeline" + default: "sev3_pipeline" +------------------------------------------------------------------------------ + + +With this configuration, all events with `log_type: critical` are sent to +`sev1_pipeline`, all events with `log_type: normal` are sent to a +`sev2_pipeline`, and all other events are sent to `sev3_pipeline`. +endif::apm-server[] + +ifdef::apm-server[] +The following example sends events to a specific pipeline based on whether the +`processor.event` field contains the specified string: + +["source","yaml"] +------------------------------------------------------------------------------ +output.elasticsearch: + hosts: ["http://localhost:9200"] + pipelines: + - pipeline: "sourcemap_pipeline" + when.contains: + processor.event: "sourcemap" + + - pipeline: "error_pipeline" + when.contains: + processor.event: "error" + + - pipeline: "transaction_pipeline" + when.contains: + processor.event: "transaction" + + - pipeline: "span_pipeline" + when.contains: + processor.event: "span" + + - pipeline: "metric_pipeline" + when.contains: + processor.event: "metric" + + - pipeline: "onboarding_pipeline" + when.contains: + processor.event: "onboarding" +------------------------------------------------------------------------------ + + +The following example sets the pipeline by taking the name returned by the +`pipeline` format string and mapping it to a new name that's used for the +pipeline: + +["source","yaml"] +------------------------------------------------------------------------------ +output.elasticsearch: + hosts: ["http://localhost:9200"] + pipelines: + - pipeline: "%{[processor.event]}" + mappings: + sourcemap: "sourcemap_pipeline" + error: "error_pipeline" + transaction: "transaction_pipeline" + span: "span_pipeline" + metric: "metric_pipeline" + onboarding: "onboarding_pipeline" + default: "apm_pipeline" +------------------------------------------------------------------------------ + +With this configuration, all events with `processor.event: transaction` are sent to a pipeline +named `transaction_pipeline`, all events with `processor.event: error` are sent to a +pipeline named `error_pipeline`, etc. + +NOTE: Defining any pipeline will deactivate the default `apm` pipeline. +endif::apm-server[] + +For more information about ingest node pipelines, see +<>. + +endif::[] + +===== `max_retries` + +ifdef::ignores_max_retries[] +{beatname_uc} ignores the `max_retries` setting and retries indefinitely. +endif::[] + +ifndef::ignores_max_retries[] +The number of times to retry publishing an event after a publishing failure. +After the specified number of retries, the events are typically dropped. + +Set `max_retries` to a value less than 0 to retry until all events are published. + +The default is 3. +endif::[] + + +===== `bulk_max_size` + +The maximum number of events to bulk in a single Elasticsearch bulk API index request. The default is 50. + +Events can be collected into batches. {beatname_uc} will split batches larger than `bulk_max_size` +into multiple batches. + +Specifying a larger batch size can improve performance by lowering the overhead of sending events. +However big batch sizes can also increase processing times, which might result in +API errors, killed connections, timed-out publishing requests, and, ultimately, lower +throughput. + +Setting `bulk_max_size` to values less than or equal to 0 disables the +splitting of batches. When splitting is disabled, the queue decides on the +number of events to be contained in a batch. + +===== `backoff.init` + +The number of seconds to wait before trying to reconnect to Elasticsearch after +a network error. After waiting `backoff.init` seconds, {beatname_uc} tries to +reconnect. If the attempt fails, the backoff timer is increased exponentially up +to `backoff.max`. After a successful connection, the backoff timer is reset. The +default is 1s. + + +===== `backoff.max` + +The maximum number of seconds to wait before attempting to connect to +Elasticsearch after a network error. The default is 60s. + +===== `timeout` + +The http request timeout in seconds for the Elasticsearch request. The default is 90. + +===== `ssl` + +Configuration options for SSL parameters like the certificate authority to use +for HTTPS-based connections. If the `ssl` section is missing, the host CAs are used for HTTPS connections to +Elasticsearch. + +See <> for more information. diff --git a/docs/copied-from-beats/outputs/output-file.asciidoc b/docs/copied-from-beats/outputs/output-file.asciidoc new file mode 100644 index 00000000000..aa0af27bb66 --- /dev/null +++ b/docs/copied-from-beats/outputs/output-file.asciidoc @@ -0,0 +1,63 @@ +[[file-output]] +=== Configure the File output + +++++ +File +++++ + +The File output dumps the transactions into a file where each transaction is in a JSON format. +Currently, this output is used for testing, but it can be used as input for +Logstash. + +["source","yaml",subs="attributes"] +------------------------------------------------------------------------------ +output.file: + path: "/tmp/{beatname_lc}" + filename: {beatname_lc} + #rotate_every_kb: 10000 + #number_of_files: 7 + #permissions: 0600 +------------------------------------------------------------------------------ + +==== Configuration options + +You can specify the following options in the `file` section of the +{beatname_lc}.yml+ config file: + +===== `enabled` + +The enabled config is a boolean setting to enable or disable the output. If set +to false, the output is disabled. + +The default value is true. + +[[path]] +===== `path` + +The path to the directory where the generated files will be saved. This option is +mandatory. + +===== `filename` + +The name of the generated files. The default is set to the Beat name. For example, the files +generated by default for {beatname_uc} would be "{beatname_lc}", "{beatname_lc}.1", "{beatname_lc}.2", and so on. + +===== `rotate_every_kb` + +The maximum size in kilobytes of each file. When this size is reached, the files are +rotated. The default value is 10240 KB. + +===== `number_of_files` + +The maximum number of files to save under <>. When this number of files is reached, the +oldest file is deleted, and the rest of the files are shifted from last to first. +The number of files must be between 2 and 1024. The default is 7. + +===== `permissions` + +Permissions to use for file creation. The default is 0600. + +===== `codec` + +Output codec configuration. If the `codec` section is missing, events will be json encoded. + +See <> for more information. diff --git a/docs/copied-from-beats/outputs/output-kafka.asciidoc b/docs/copied-from-beats/outputs/output-kafka.asciidoc new file mode 100644 index 00000000000..4006fcb1e6e --- /dev/null +++ b/docs/copied-from-beats/outputs/output-kafka.asciidoc @@ -0,0 +1,269 @@ +[[kafka-output]] +=== Configure the Kafka output + +++++ +Kafka +++++ + +The Kafka output sends the events to Apache Kafka. + +Example configuration: + +[source,yaml] +------------------------------------------------------------------------------ +output.kafka: + # initial brokers for reading cluster metadata + hosts: ["kafka1:9092", "kafka2:9092", "kafka3:9092"] + + # message topic selection + partitioning + topic: '%{[fields.log_topic]}' + partition.round_robin: + reachable_only: false + + required_acks: 1 + compression: gzip + max_message_bytes: 1000000 +------------------------------------------------------------------------------ + +NOTE: Events bigger than <> will be dropped. To avoid this problem, make sure {beatname_uc} does not generate events bigger than <>. + +[[kafka-compatibility]] +==== Compatibility + +This output works with all Kafka versions in between 0.11 and 2.1.0. Older versions +might work as well, but are not supported. + +==== Configuration options + +You can specify the following options in the `kafka` section of the +{beatname_lc}.yml+ config file: + +===== `enabled` + +The `enabled` config is a boolean setting to enable or disable the output. If set +to false, the output is disabled. + +The default value is true. + +===== `hosts` + +The list of Kafka broker addresses from where to fetch the cluster metadata. +The cluster metadata contain the actual Kafka brokers events are published to. + +===== `version` + +Kafka version {beatname_lc} is assumed to run against. Defaults to 1.0.0. + +Event timestamps will be added, if version 0.10.0.0+ is enabled. + +Valid values are all kafka releases in between `0.8.2.0` and `2.0.0`. + +See <> for information on supported versions. + +===== `username` + +The username for connecting to Kafka. If username is configured, the password +must be configured as well. Only SASL/PLAIN is supported. + +===== `password` + +The password for connecting to Kafka. + +[[topic-option-kafka]] +===== `topic` + +The Kafka topic used for produced events. + +You can set the topic dynamically by using a format string to access any +event field. For example, this configuration uses a custom field, +`fields.log_topic`, to set the topic for each event: + +[source,yaml] +----- +topic: '%{[fields.log_topic]}' +----- + +TIP: To learn how to add custom fields to events, see the +<> option. + +See the <> setting for other ways to set the +topic dynamically. + +[[topics-option-kafka]] +===== `topics` + +An array of topic selector rules. Each rule specifies the `topic` to use for +events that match the rule. During publishing, {beatname_uc} sets the `topic` +for each event based on the first matching rule in the array. Rules +can contain conditionals, format string-based fields, and name mappings. If the +`topics` setting is missing or no rule matches, the +<> field is used. + +Rule settings: + +*`topic`*:: The topic format string to use. If this string contains field +references, such as `%{[fields.name]}`, the fields must exist, or the rule +fails. + +*`mappings`*:: A dictionary that takes the value returned by `topic` and maps it +to a new name. + +*`default`*:: The default string value to use if `mappings` does not find a +match. + +*`when`*:: A condition that must succeed in order to execute the current rule. +ifndef::no-processors[] +All the <> supported by processors are also supported +here. +endif::no-processors[] + +The following example sets the topic based on whether the message field contains +the specified string: + +["source","yaml",subs="attributes"] +------------------------------------------------------------------------------ +output.kafka: + hosts: ["localhost:9092"] + topic: "logs-%{[beat.version]}" + topics: + - topic: "critical-%{[beat.version]}" + when.contains: + message: "CRITICAL" + - topic: "error-%{[beat.version]}" + when.contains: + message: "ERR" +------------------------------------------------------------------------------ + + +This configuration results in topics named +critical-{version}+, ++error-{version}+, and +logs-{version}+. + +===== `key` + +Optional Kafka event key. If configured, the event key must be unique and can be extracted from the event using a format string. + +===== `partition` + +Kafka output broker event partitioning strategy. Must be one of `random`, +`round_robin`, or `hash`. By default the `hash` partitioner is used. + +*`random.group_events`*: Sets the number of events to be published to the same + partition, before the partitioner selects a new partition by random. The + default value is 1 meaning after each event a new partition is picked randomly. + +*`round_robin.group_events`*: Sets the number of events to be published to the + same partition, before the partitioner selects the next partition. The default + value is 1 meaning after each event the next partition will be selected. + +*`hash.hash`*: List of fields used to compute the partitioning hash value from. + If no field is configured, the events `key` value will be used. + +*`hash.random`*: Randomly distribute events if no hash or key value can be computed. + +All partitioners will try to publish events to all partitions by default. If a +partition's leader becomes unreachable for the beat, the output might block. All +partitioners support setting `reachable_only` to overwrite this +behavior. If `reachable_only` is set to `true`, events will be published to +available partitions only. + +NOTE: Publishing to a subset of available partitions potentially increases resource usage because events may become unevenly distributed. + +===== `client_id` + +The configurable ClientID used for logging, debugging, and auditing purposes. The default is "beats". + +===== `worker` + +The number of concurrent load-balanced Kafka output workers. + +===== `codec` + +Output codec configuration. If the `codec` section is missing, events will be json encoded. + +See <> for more information. + +===== `metadata` + +Kafka metadata update settings. The metadata do contain information about +brokers, topics, partition, and active leaders to use for publishing. + +*`refresh_frequency`*:: Metadata refresh interval. Defaults to 10 minutes. + +*`full`*:: Strategy to use when fetching metadata, when this option is `true`, the client will maintain +a full set of metadata for all the available topics, if the this option is set to `false` it will only refresh the +metadata for the configured topics. The default is false. + +*`retry.max`*:: Total number of metadata update retries when cluster is in middle of leader election. The default is 3. + +*`retry.backoff`*:: Waiting time between retries during leader elections. Default is 250ms. + +===== `max_retries` + +ifdef::ignores_max_retries[] +{beatname_uc} ignores the `max_retries` setting and retries indefinitely. +endif::[] + +ifndef::ignores_max_retries[] +The number of times to retry publishing an event after a publishing failure. +After the specified number of retries, the events are typically dropped. + +Set `max_retries` to a value less than 0 to retry until all events are published. + +The default is 3. +endif::[] + +===== `bulk_max_size` + +The maximum number of events to bulk in a single Kafka request. The default is 2048. + +===== `bulk_flush_frequency` + +Duration to wait before sending bulk Kafka request. 0 is no delay. The default is 0. + +===== `timeout` + +The number of seconds to wait for responses from the Kafka brokers before timing +out. The default is 30 (seconds). + +===== `broker_timeout` + +The maximum duration a broker will wait for number of required ACKs. The default is 10s. + +===== `channel_buffer_size` + +Per Kafka broker number of messages buffered in output pipeline. The default is 256. + +===== `keep_alive` + +The keep-alive period for an active network connection. If 0s, keep-alives are disabled. The default is 0 seconds. + +===== `compression` + +Sets the output compression codec. Must be one of `none`, `snappy`, `lz4` and `gzip`. The default is `gzip`. + +===== `compression_level` + +Sets the compression level used by gzip. Setting this value to 0 disables compression. +The compression level must be in the range of 1 (best speed) to 9 (best compression). + +Increasing the compression level will reduce the network usage but will increase the cpu usage. + +The default value is 4. + +[[kafka-max_message_bytes]] +===== `max_message_bytes` + +The maximum permitted size of JSON-encoded messages. Bigger messages will be dropped. The default value is 1000000 (bytes). This value should be equal to or less than the broker's `message.max.bytes`. + +===== `required_acks` + +The ACK reliability level required from broker. 0=no response, 1=wait for local commit, -1=wait for all replicas to commit. The default is 1. + +Note: If set to 0, no ACKs are returned by Kafka. Messages might be lost silently on error. + +===== `ssl` + +Configuration options for SSL parameters like the root CA for Kafka connections. + The Kafka host keystore should be created with the +`-keyalg RSA` argument to ensure it uses a cipher supported by +https://github.com/Shopify/sarama/wiki/Frequently-Asked-Questions#why-cant-sarama-connect-to-my-kafka-cluster-using-ssl[Filebeat's Kafka library]. +See <> for more information. diff --git a/docs/copied-from-beats/outputs/output-logstash.asciidoc b/docs/copied-from-beats/outputs/output-logstash.asciidoc new file mode 100644 index 00000000000..76ad6e27073 --- /dev/null +++ b/docs/copied-from-beats/outputs/output-logstash.asciidoc @@ -0,0 +1,397 @@ +[[logstash-output]] +=== Configure the Logstash output + +++++ +Logstash +++++ + +The Logstash output sends events directly to Logstash by using the lumberjack +protocol, which runs over TCP. Logstash allows for additional processing and routing of +generated events. + +// tag::shared-logstash-config[] + +[IMPORTANT] +.Prerequisite +To send events to {ls}, you also need to create a {ls} configuration pipeline +that listens for incoming Beats connections and indexes the received events into +{es}. For more information, see the section about +{stack-gs}/get-started-elastic-stack.html#logstash-setup[configuring {ls}] in +the {stack} getting started tutorial. Also see the documentation for the +{logstash-ref}/plugins-inputs-beats.html[{beats} input] and +{logstash-ref}/plugins-outputs-elasticsearch.html[{es} output] plugins. + +If you want to use {ls} to perform additional processing on the data collected by +{beatname_uc}, you need to configure {beatname_uc} to use {ls}. + +To do this, you edit the {beatname_uc} configuration file to disable the {es} +output by commenting it out and enable the {ls} output by uncommenting the +logstash section: + +[source,yaml] +------------------------------------------------------------------------------ +#----------------------------- Logstash output -------------------------------- +output.logstash: + hosts: ["127.0.0.1:5044"] +------------------------------------------------------------------------------ + +The `hosts` option specifies the {ls} server and the port (`5044`) where {ls} is configured to listen for incoming +Beats connections. + +For this configuration, you must <> +because the options for auto loading the template are only available for the {es} output. + +ifeval::["{beatname_lc}"=="filebeat"] +Want to use <> with {ls}? You need to do +some extra setup. For more information, see +{logstash-ref}/filebeat-modules.html[Working with {beatname_uc} modules]. +endif::[] + +ifndef::win-only[] +ifndef::apm-server[] + +include::{libbeat-dir}/docs/step-test-config.asciidoc[] + +endif::apm-server[] +ifdef::apm-server[] + +include::./../step-test-config.asciidoc[] + +endif::apm-server[] +endif::win-only[] + +ifdef::win-only[] + +TIP: To test your configuration file, change to the directory where the {beatname_uc} +binary is installed, and run {beatname_uc} in the foreground with the following +options specified: +.\winlogbeat.exe test config -c .\winlogbeat.yml -e+. + +endif::win-only[] + +// end::shared-logstash-config[] + +==== Accessing metadata fields + +Every event sent to Logstash contains the following metadata fields that you can +use in Logstash for indexing and filtering: + +ifndef::apm-server[] +["source","json",subs="attributes"] +------------------------------------------------------------------------------ +{ + ... + "@metadata": { <1> + "beat": "{beat_default_index_prefix}", <2> + "version": "{stack-version}" <3> + } +} +------------------------------------------------------------------------------ +<1> {beatname_uc} uses the `@metadata` field to send metadata to Logstash. See the +{logstash-ref}/event-dependent-configuration.html#metadata[Logstash documentation] +for more about the `@metadata` field. +<2> The default is {beat_default_index_prefix}. To change this value, set the +<> option in the {beatname_uc} config file. +<3> The current version of {beatname_uc}. +endif::[] + +ifdef::apm-server[] +["source","json",subs="attributes"] +------------------------------------------------------------------------------ +{ + ... + "@metadata": { <1> + "beat": "{beat_default_index_prefix}", <2> + "pipeline":"apm", <3> + "version": "{stack-version}" <4> + } +} +------------------------------------------------------------------------------ +<1> {beatname_uc} uses the `@metadata` field to send metadata to Logstash. See the +{logstash-ref}/event-dependent-configuration.html#metadata[Logstash documentation] +for more about the `@metadata` field. +<2> The default is {beat_default_index_prefix}. To change this value, set the +<> option in the {beatname_uc} config file. +<3> The default pipeline configuration: `apm`. Additional pipelines can be enabled +with a {logstash-ref}/use-ingest-pipelines.html[Logstash pipeline config]. +<4> The current version of {beatname_uc}. +endif::[] + +You can access this metadata from within the Logstash config file to set values +dynamically based on the contents of the metadata. + +For example, the following Logstash configuration file tells +Logstash to use the index reported by {beatname_uc} for indexing events +into Elasticsearch: + +ifndef::apm-server[] +[source,logstash] +------------------------------------------------------------------------------ + +input { + beats { + port => 5044 + } +} + +output { + elasticsearch { + hosts => ["http://localhost:9200"] + index => "%{[@metadata][beat]}-%{[@metadata][version]}" <1> + } +} +------------------------------------------------------------------------------ +<1> `%{[@metadata][beat]}` sets the first part of the index name to the value +of the `beat` metadata field and `%{[@metadata][version]}` sets the second part to +the Beat's version. For example: ++{beat_default_index_prefix}-{version}+. +endif::[] + +ifdef::apm-server[] +[source,logstash] +------ +input { + beats { + port => 5044 + } +} + +filter { + if [@metadata][beat] == "apm" { + if [processor][event] == "sourcemap" { + mutate { + add_field => { "[@metadata][index]" => "%{[@metadata][beat]}-%{[@metadata][version]}-%{[processor][event]}" } <1> + } + } else { + mutate { + add_field => { "[@metadata][index]" => "%{[@metadata][beat]}-%{[@metadata][version]}-%{[processor][event]}-%{+yyyy.MM.dd}" } <2> + } + } + } +} + +output { + elasticsearch { + hosts => ["http://localhost:9200"] + index => "%{[@metadata][index]}" + } +} +------ +<1> Creates a new field named `@metadata.index`. +`%{[@metadata][beat]}` sets the first part of the index name to the value of the `beat` metadata field. +`%{[@metadata][version]}` sets the second part to {beatname_uc}'s version. +`%{[processor][event]}` sets the final part based on the APM event type. +For example: +{beat_default_index_prefix}-{version}-sourcemap+. +<2> In addition to the above rules, this pattern appends a date to the `index` name so Logstash creates a new index each day. +For example: +{beat_default_index_prefix}-{version}-transaction-{sample_date_0}+. +endif::[] + +Events indexed into Elasticsearch with the Logstash configuration shown here +will be similar to events directly indexed by {beatname_uc} into Elasticsearch. + +ifndef::apm-server[] +NOTE: If ILM is not being used, set `index` to `%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}` instead so Logstash creates an index per day, based on the `@timestamp` value of the events coming from Beats. +endif::[] + +ifdef::apm-server[] +==== Logstash and ILM + +When used with {apm-server-ref}/manual-ilm-setup.html[Index lifecycle management], Logstash does not need to create a new index each day. +Here's a sample Logstash configuration file that would accomplish this: + +[source,logstash] +------ +input { + beats { + port => 5044 + } +} + +output { + elasticsearch { + hosts => ["http://localhost:9200"] + index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{[processor][event]}" + } +} +------ +endif::[] + +==== Compatibility + +This output works with all compatible versions of Logstash. See the +https://www.elastic.co/support/matrix#matrix_compatibility[Elastic Support +Matrix]. + +==== Configuration options + +You can specify the following options in the `logstash` section of the ++{beatname_lc}.yml+ config file: + +===== `enabled` + +The enabled config is a boolean setting to enable or disable the output. If set +to false, the output is disabled. + +The default value is `true`. + +[[hosts]] +===== `hosts` + +The list of known Logstash servers to connect to. If load balancing is disabled, but +multiple hosts are configured, one host is selected randomly (there is no precedence). +If one host becomes unreachable, another one is selected randomly. + +All entries in this list can contain a port number. The default port number 5044 will be used, if no number is given. + +===== `compression_level` + +The gzip compression level. Setting this value to 0 disables compression. +The compression level must be in the range of 1 (best speed) to 9 (best compression). + +Increasing the compression level will reduce the network usage but will increase the cpu usage. + +The default value is 3. + +===== `escape_html` + +Configure escaping of HTML in strings. Set to `true` to enable escaping. + +The default value is `false`. + +===== `worker` + +The number of workers per configured host publishing events to Logstash. This +is best used with load balancing mode enabled. Example: If you have 2 hosts and +3 workers, in total 6 workers are started (3 for each host). + +[[loadbalance]] +===== `loadbalance` + +If set to true and multiple Logstash hosts are configured, the output plugin +load balances published events onto all Logstash hosts. If set to false, +the output plugin sends all events to only one host (determined at random) and +will switch to another host if the selected one becomes unresponsive. The default value is false. + +["source","yaml",subs="attributes"] +------------------------------------------------------------------------------ +output.logstash: + hosts: ["localhost:5044", "localhost:5045"] + loadbalance: true + index: {beatname_lc} +------------------------------------------------------------------------------ + +===== `ttl` + +Time to live for a connection to Logstash after which the connection will be re-established. +Useful when Logstash hosts represent load balancers. Since the connections to Logstash hosts +are sticky, operating behind load balancers can lead to uneven load distribution between the instances. +Specifying a TTL on the connection allows to achieve equal connection distribution between the +instances. Specifying a TTL of 0 will disable this feature. + +The default value is 0. + +NOTE: The "ttl" option is not yet supported on an async Logstash client (one with the "pipelining" option set). + +===== `pipelining` + +Configures number of batches to be sent asynchronously to logstash while waiting +for ACK from logstash. Output only becomes blocking once number of `pipelining` +batches have been written. Pipelining is disabled if a values of 0 is +configured. The default value is 2. + +===== `proxy_url` + +The URL of the SOCKS5 proxy to use when connecting to the Logstash servers. The +value must be a URL with a scheme of `socks5://`. The protocol used to +communicate to Logstash is not based on HTTP so a web-proxy cannot be used. + +If the SOCKS5 proxy server requires client authentication, then a username and +password can be embedded in the URL as shown in the example. + +When using a proxy, hostnames are resolved on the proxy server instead of on the +client. You can change this behavior by setting the +<> option. + +["source","yaml",subs="attributes"] +------------------------------------------------------------------------------ +output.logstash: + hosts: ["remote-host:5044"] + proxy_url: socks5://user:password@socks5-proxy:2233 +------------------------------------------------------------------------------ + +[[logstash-proxy-use-local-resolver]] +===== `proxy_use_local_resolver` + +The `proxy_use_local_resolver` option determines if Logstash hostnames are +resolved locally when using a proxy. The default value is false which means +that when a proxy is used the name resolution occurs on the proxy server. + +[[logstash-index]] +===== `index` + +The index root name to write events to. The default is the Beat name. For +example +"{beat_default_index_prefix}"+ generates +"[{beat_default_index_prefix}-]{version}-YYYY.MM.DD"+ +indices (for example, +"{beat_default_index_prefix}-{version}-2017.04.26"+). + +===== `ssl` + +Configuration options for SSL parameters like the root CA for Logstash connections. See +<> for more information. To use SSL, you must also configure the +https://www.elastic.co/guide/en/logstash/current/plugins-inputs-beats.html[Beats input plugin for Logstash] to use SSL/TLS. + +===== `timeout` + +The number of seconds to wait for responses from the Logstash server before timing out. The default is 30 (seconds). + +===== `max_retries` + +ifdef::ignores_max_retries[] +{beatname_uc} ignores the `max_retries` setting and retries indefinitely. +endif::[] + +ifndef::ignores_max_retries[] +The number of times to retry publishing an event after a publishing failure. +After the specified number of retries, the events are typically dropped. + +Set `max_retries` to a value less than 0 to retry until all events are published. + +The default is 3. +endif::[] + +===== `bulk_max_size` + +The maximum number of events to bulk in a single Logstash request. The default is 2048. + +If the Beat sends single events, the events are collected into batches. If the Beat publishes +a large batch of events (larger than the value specified by `bulk_max_size`), the batch is +split. + +Specifying a larger batch size can improve performance by lowering the overhead of sending events. +However big batch sizes can also increase processing times, which might result in +API errors, killed connections, timed-out publishing requests, and, ultimately, lower +throughput. + +Setting `bulk_max_size` to values less than or equal to 0 disables the +splitting of batches. When splitting is disabled, the queue decides on the +number of events to be contained in a batch. + + +===== `slow_start` + +If enabled only a subset of events in a batch of events is transferred per transaction. +The number of events to be sent increases up to `bulk_max_size` if no error is encountered. +On error the number of events per transaction is reduced again. + +The default is `false`. + +===== `backoff.init` + +The number of seconds to wait before trying to reconnect to Logstash after +a network error. After waiting `backoff.init` seconds, {beatname_uc} tries to +reconnect. If the attempt fails, the backoff timer is increased exponentially up +to `backoff.max`. After a successful connection, the backoff timer is reset. The +default is 1s. + +===== `backoff.max` + +The maximum number of seconds to wait before attempting to connect to +Logstash after a network error. The default is 60s. diff --git a/docs/copied-from-beats/outputs/output-redis.asciidoc b/docs/copied-from-beats/outputs/output-redis.asciidoc new file mode 100644 index 00000000000..3123a9fa9f3 --- /dev/null +++ b/docs/copied-from-beats/outputs/output-redis.asciidoc @@ -0,0 +1,251 @@ +//begin inner exclude for redis +ifndef::no-redis-output[] +[[redis-output]] +=== Configure the Redis output + +++++ +Redis +++++ + +The Redis output inserts the events into a Redis list or a Redis channel. +This output plugin is compatible with +the https://www.elastic.co/guide/en/logstash/current/plugins-inputs-redis.html[Redis input plugin] for Logstash. + +Example configuration: + +["source","yaml",subs="attributes"] +------------------------------------------------------------------------------ +output.redis: + hosts: ["localhost"] + password: "my_password" + key: "{beatname_lc}" + db: 0 + timeout: 5 +------------------------------------------------------------------------------ + +==== Compatibility + +This output works with Redis 3.2.4. + +==== Configuration options + +You can specify the following options in the `redis` section of the +{beatname_lc}.yml+ config file: + +===== `enabled` + +The enabled config is a boolean setting to enable or disable the output. If set +to false, the output is disabled. + +The default value is true. + +===== `hosts` + +The list of Redis servers to connect to. If load balancing is enabled, the events are +distributed to the servers in the list. If one server becomes unreachable, the events are +distributed to the reachable servers only. You can define each Redis server by specifying +`HOST` or `HOST:PORT`. For example: `"192.15.3.2"` or `"test.redis.io:12345"`. If you +don't specify a port number, the value configured by `port` is used. + +===== `index` + +The index name added to the events metadata for use by Logstash. The default is "{beatname_lc}". + +[[key-option-redis]] +===== `key` + +The name of the Redis list or channel the events are published to. If not +configured, the value of the `index` setting is used. + +You can set the key dynamically by using a format string to access any event +field. For example, this configuration uses a custom field, `fields.list`, to +set the Redis list key. If `fields.list` is missing, `fallback` is used: + +["source","yaml"] +------------------------------------------------------------------------------ +output.redis: + hosts: ["localhost"] + key: "%{[fields.list]:fallback}" +------------------------------------------------------------------------------ + + +TIP: To learn how to add custom fields to events, see the +<> option. + +See the <> setting for other ways to set the key +dynamically. + +[[keys-option-redis]] +===== `keys` + +An array of key selector rules. Each rule specifies the `key` to use for events +that match the rule. During publishing, {beatname_uc} uses the first matching +rule in the array. Rules can contain conditionals, format string-based fields, +and name mappings. If the `keys` setting is missing or no rule matches, the +<> setting is used. + +Rule settings: + +*`index`*:: The key format string to use. If this string contains field +references, such as `%{[fields.name]}`, the fields must exist, or the rule +fails. + +*`mappings`*:: A dictionary that takes the value returned by `key` and maps it to +a new name. + +*`default`*:: The default string value to use if `mappings` does not find a match. + +*`when`*:: A condition that must succeed in order to execute the current rule. +ifndef::no-processors[] +All the <> supported by processors are also supported +here. +endif::no-processors[] + +Example `keys` settings: + +["source","yaml"] +------------------------------------------------------------------------------ +output.redis: + hosts: ["localhost"] + key: "default_list" + keys: + - key: "info_list" # send to info_list if `message` field contains INFO + when.contains: + message: "INFO" + - key: "debug_list" # send to debug_list if `message` field contains DEBUG + when.contains: + message: "DEBUG" + - key: "%{[fields.list]}" + mappings: + http: "frontend_list" + nginx: "frontend_list" + mysql: "backend_list" +------------------------------------------------------------------------------ + +===== `password` + +The password to authenticate with. The default is no authentication. + +===== `db` + +The Redis database number where the events are published. The default is 0. + +===== `datatype` + +The Redis data type to use for publishing events.If the data type is `list`, the +Redis RPUSH command is used and all events are added to the list with the key defined under `key`. +If the data type `channel` is used, the Redis `PUBLISH` command is used and means that all events +are pushed to the pub/sub mechanism of Redis. The name of the channel is the one defined under `key`. +The default value is `list`. + +===== `codec` + +Output codec configuration. If the `codec` section is missing, events will be json encoded. + +See <> for more information. + +===== `host_topology` + +deprecated:[5.0.0] + +The Redis host to connect to when using topology map support. Topology map support is disabled if this option is not set. + +===== `password_topology` + +deprecated:[5.0.0] + +The password to use for authenticating with the Redis topology server. The default is no authentication. + +===== `db_topology` + +deprecated:[5.0.0] + +The Redis database number where the topology information is stored. The default is 1. + +===== `worker` + +The number of workers to use for each host configured to publish events to Redis. Use this setting along with the +`loadbalance` option. For example, if you have 2 hosts and 3 workers, in total 6 workers are started (3 for each host). + +===== `loadbalance` + +If set to true and multiple hosts or workers are configured, the output plugin load balances published events onto all +Redis hosts. If set to false, the output plugin sends all events to only one host (determined at random) and will switch +to another host if the currently selected one becomes unreachable. The default value is true. + +===== `timeout` + +The Redis connection timeout in seconds. The default is 5 seconds. + +===== `backoff.init` + +The number of seconds to wait before trying to reconnect to Redis after +a network error. After waiting `backoff.init` seconds, {beatname_uc} tries to +reconnect. If the attempt fails, the backoff timer is increased exponentially up +to `backoff.max`. After a successful connection, the backoff timer is reset. The +default is 1s. + +===== `backoff.max` + +The maximum number of seconds to wait before attempting to connect to +Redis after a network error. The default is 60s. + +===== `max_retries` + +ifdef::ignores_max_retries[] +{beatname_uc} ignores the `max_retries` setting and retries indefinitely. +endif::[] + +ifndef::ignores_max_retries[] +The number of times to retry publishing an event after a publishing failure. +After the specified number of retries, the events are typically dropped. + +Set `max_retries` to a value less than 0 to retry until all events are published. + +The default is 3. +endif::[] + + +===== `bulk_max_size` + +The maximum number of events to bulk in a single Redis request or pipeline. The default is 2048. + +If the Beat sends single events, the events are collected into batches. If the +Beat publishes a large batch of events (larger than the value specified by +`bulk_max_size`), the batch is split. + +Specifying a larger batch size can improve performance by lowering the overhead +of sending events. However big batch sizes can also increase processing times, +which might result in API errors, killed connections, timed-out publishing +requests, and, ultimately, lower throughput. + +Setting `bulk_max_size` to values less than or equal to 0 disables the +splitting of batches. When splitting is disabled, the queue decides on the +number of events to be contained in a batch. + +===== `ssl` + +Configuration options for SSL parameters like the root CA for Redis connections +guarded by SSL proxies (for example https://www.stunnel.org[stunnel]). See +<> for more information. + +===== `proxy_url` + +The URL of the SOCKS5 proxy to use when connecting to the Redis servers. The +value must be a URL with a scheme of `socks5://`. You cannot use a web proxy +because the protocol used to communicate with Redis is not based on HTTP. + +If the SOCKS5 proxy server requires client authentication, you can embed +a username and password in the URL. + +When using a proxy, hostnames are resolved on the proxy server instead of on the +client. You can change this behavior by setting the +<> option. + +[[redis-proxy-use-local-resolver]] +===== `proxy_use_local_resolver` + +This option determines whether Redis hostnames are resolved locally when using a proxy. +The default value is false, which means that name resolution occurs on the proxy server. + +//end inner exclude for redis +endif::[] diff --git a/docs/copied-from-beats/shared-securing-beat.asciidoc b/docs/copied-from-beats/shared-securing-beat.asciidoc index d8f2b76383c..a46882a7c5c 100644 --- a/docs/copied-from-beats/shared-securing-beat.asciidoc +++ b/docs/copied-from-beats/shared-securing-beat.asciidoc @@ -12,13 +12,13 @@ ifdef::apm-server[] * <> endif::[] * <> -ifndef::only-elasticsearch[] +ifndef::no_logstash_output[] * <> -endif::only-elasticsearch[] +endif::[] * <> -ifndef::only-elasticsearch[] +ifndef::serverless[] * <> -endif::only-elasticsearch[] +endif::[] ifdef::beat-specific-security[] include::{beat-specific-security}[] @@ -39,15 +39,15 @@ include::./https.asciidoc[] //sets block macro for shared-ssl-logstash-config.asciidoc included in next section -ifndef::only-elasticsearch[] +ifndef::no_logstash_output[] [[configuring-ssl-logstash]] == Secure communication with Logstash include::./shared-ssl-logstash-config.asciidoc[] -endif::only-elasticsearch[] +endif::[] include::./security/securing-beats.asciidoc[] -ifndef::only-elasticsearch[] +ifndef::serverless[] include::./security/linux-seccomp.asciidoc[] -endif::only-elasticsearch[] +endif::[] diff --git a/docs/copied-from-beats/shared-ssl-config.asciidoc b/docs/copied-from-beats/shared-ssl-config.asciidoc index c835e936dc2..329cfb23aba 100644 --- a/docs/copied-from-beats/shared-ssl-config.asciidoc +++ b/docs/copied-from-beats/shared-ssl-config.asciidoc @@ -31,7 +31,7 @@ output.elasticsearch.ssl.certificate: "/etc/pki/client/cert.pem" output.elasticsearch.ssl.key: "/etc/pki/client/cert.key" ---- -ifndef::only-elasticsearch[] +ifndef::no_logstash_output[] Also see <>. endif::[] diff --git a/docs/copied-from-beats/shared-template-load.asciidoc b/docs/copied-from-beats/shared-template-load.asciidoc index d044b4e500c..726df9a597e 100644 --- a/docs/copied-from-beats/shared-template-load.asciidoc +++ b/docs/copied-from-beats/shared-template-load.asciidoc @@ -10,11 +10,11 @@ //// This content must be embedded underneath a level 3 heading. ////////////////////////////////////////////////////////////////////////// -ifndef::only-elasticsearch[] +ifndef::no_logstash_output[] NOTE: A connection to Elasticsearch is required to load the index template. If the output is not Elasticsearch, you must <>. -endif::only-elasticsearch[] +endif::[] In Elasticsearch, {ref}/indices-templates.html[index templates] are used to define settings and mappings that determine how fields @@ -107,13 +107,14 @@ See <> for the full list of configuration options. ==== Load the template manually To load the template manually, run the <> command. A -connection to Elasticsearch is required. -ifndef::only-elasticsearch[] -If another output is enabled, you need -to temporarily disable that output and enable Elasticsearch by using the -`-E` option. The examples here assume that Logstash output is enabled. You can -omit the `-E` flags if Elasticsearch output is already enabled. -endif::only-elasticsearch[] +connection to Elasticsearch is required. If another output is enabled, you need +to temporarily disable that output and enable Elasticsearch by using the `-E` +option. +ifndef::no_logstash_output[] +The examples here assume that Logstash output is enabled. +endif::[] +You can omit the `-E` flags if Elasticsearch output is already enabled. + If you are connecting to a secured Elasticsearch cluster, make sure you've configured credentials as described in <<{beatname_lc}-configuration>>. @@ -129,13 +130,13 @@ ifdef::win_only[] To load the template: endif::win_only[] -ifndef::only-elasticsearch[] +ifndef::no_logstash_output[] :disable_logstash: {sp}-E output.logstash.enabled=false -endif::only-elasticsearch[] +endif::[] -ifdef::only-elasticsearch[] +ifdef::no_logstash_output[] :disable_logstash: -endif::only-elasticsearch[] +endif::[] ifdef::requires-sudo[] include::./shared-note-sudo.asciidoc[] diff --git a/docs/copied-from-beats/template-config.asciidoc b/docs/copied-from-beats/template-config.asciidoc index 2a99f1fe618..ac5bf867d51 100644 --- a/docs/copied-from-beats/template-config.asciidoc +++ b/docs/copied-from-beats/template-config.asciidoc @@ -8,7 +8,7 @@ mappings in Elasticsearch. If template loading is enabled (the default), {beatname_uc} loads the index template automatically after successfully connecting to Elasticsearch. -ifndef::only-elasticsearch[] +ifndef::no_logstash_output[] NOTE: A connection to Elasticsearch is required to load the index template. If the output is Logstash, you must <