diff --git a/libbeat/docs/command-reference.asciidoc b/libbeat/docs/command-reference.asciidoc index c59b30d507be..e3bb27cbd61d 100644 --- a/libbeat/docs/command-reference.asciidoc +++ b/libbeat/docs/command-reference.asciidoc @@ -11,6 +11,7 @@ // These attributes are used to resolve short descriptions +tag::attributes[] :global-flags: Also see <>. @@ -53,6 +54,8 @@ endif::[] :test-command-short-desc: Tests the configuration :version-command-short-desc: Shows information about the current version +end::attributes[] + [[command-line-options]] === {beatname_uc} command reference @@ -634,7 +637,7 @@ endif::[] If index lifecycle management is enabled it also ensures that the defined ILM policy and write alias are connected to the indices matching the index template. The ILM policy takes care of the lifecycle of an index, when to do a rollover, -when to move an index from the hot phase to the next phase etc. +when to move an index from the hot phase to the next phase, etc. ifndef::no_dashboards[] * The {kib} dashboards make it easier for you to visualize {beatname_uc} data @@ -646,11 +649,12 @@ ifdef::has_ml_jobs[] necessary to analyze data for anomalies. endif::[] -Use this command if you want to set up the environment without actually running +This command sets up the environment without actually running {beatname_uc} and ingesting data. *SYNOPSIS* +tag::setup-command-tag[] ["source","sh",subs="attributes"] ---- {beatname_lc} setup [FLAGS] @@ -691,6 +695,11 @@ endif::[] Sets up components related to Elasticsearch index management including template, ILM policy, and write alias (if supported and configured). +ifdef::apm-server[] +*`--pipelines`*:: +Registers the <> definitions set in `ingest/pipeline/definition.json`. +endif::apm-server[] + *`--template`*:: deprecated:[7.2] Sets up the index template only. @@ -729,6 +738,8 @@ ifndef::no_dashboards[] {beatname_lc} setup --index-management ----- endif::no_dashboards[] + +ifndef::apm-server[] ifdef::no_dashboards[] ["source","sh",subs="attributes"] ----- @@ -736,8 +747,18 @@ ifdef::no_dashboards[] {beatname_lc} setup --index-management ----- endif::no_dashboards[] +endif::apm-server[] + +ifdef::apm-server[] +["source","sh",subs="attributes"] +----- +{beatname_lc} setup --index-management +{beatname_lc} setup --pipelines +----- +endif::apm-server[] endif::[] +end::setup-command-tag[] [[test-command]] ==== `test` command diff --git a/libbeat/docs/https.asciidoc b/libbeat/docs/https.asciidoc index f8eab291d493..06915d9776cd 100644 --- a/libbeat/docs/https.asciidoc +++ b/libbeat/docs/https.asciidoc @@ -41,7 +41,7 @@ this document. By default {beatname_uc} uses the list of trusted certificate authorities from the operating system where {beatname_uc} is running. You can configure {beatname_uc} to use a specific list of CA certificates instead of the list from the OS. You can also configure it to use client authentication -by specifying the certificate and key to use when the server requires the Beat to authenticate. Here is an example +by specifying the certificate and key to use when the server requires the {beatname_uc} to authenticate. Here is an example configuration: ["source","yaml",subs="attributes,callouts"] diff --git a/libbeat/docs/outputconfig.asciidoc b/libbeat/docs/outputconfig.asciidoc index 802caee69617..56a89c50ae3d 100644 --- a/libbeat/docs/outputconfig.asciidoc +++ b/libbeat/docs/outputconfig.asciidoc @@ -723,6 +723,7 @@ include::./shared-logstash-config.asciidoc[] Every event sent to Logstash contains the following metadata fields that you can use in Logstash for indexing and filtering: +ifndef::apm-server[] ["source","json",subs="attributes"] ------------------------------------------------------------------------------ { @@ -730,7 +731,6 @@ use in Logstash for indexing and filtering: "@metadata": { <1> "beat": "{beat_default_index_prefix}", <2> "version": "{stack-version}" <3> - "type": "doc" <4> } } ------------------------------------------------------------------------------ @@ -739,21 +739,39 @@ use in Logstash for indexing and filtering: for more about the `@metadata` field. <2> The default is {beat_default_index_prefix}. To change this value, set the <> option in the {beatname_uc} config file. -<3> The beats current version. -<4> The value of `type` is currently hardcoded to `doc`. It was used by previous -Logstash configs to set the type of the document in Elasticsearch. - +<3> The current version of {beatname_uc}. +endif::[] -WARNING: The `@metadata.type` field, added by the Logstash output, is -deprecated, hardcoded to `doc`, and will be removed in {beatname_uc} 7.0. +ifdef::apm-server[] +["source","json",subs="attributes"] +------------------------------------------------------------------------------ +{ + ... + "@metadata": { <1> + "beat": "{beat_default_index_prefix}", <2> + "pipeline":"apm", <3> + "version": "{stack-version}" <4> + } +} +------------------------------------------------------------------------------ +<1> {beatname_uc} uses the `@metadata` field to send metadata to Logstash. See the +{logstash-ref}/event-dependent-configuration.html#metadata[Logstash documentation] +for more about the `@metadata` field. +<2> The default is {beat_default_index_prefix}. To change this value, set the +<> option in the {beatname_uc} config file. +<3> The default pipeline configuration: `apm`. Additional pipelines can be enabled +with a {logstash-ref}/use-ingest-pipelines.html[Logstash pipeline config]. +<4> The current version of {beatname_uc}. +endif::[] You can access this metadata from within the Logstash config file to set values dynamically based on the contents of the metadata. -For example, the following Logstash configuration file for versions 2.x and -5.x sets Logstash to use the index and document type reported by Beats for -indexing events into Elasticsearch: +For example, the following Logstash configuration file tells +Logstash to use the index reported by {beatname_uc} for indexing events +into Elasticsearch: +ifndef::apm-server[] [source,logstash] ------------------------------------------------------------------------------ @@ -774,11 +792,76 @@ output { of the `beat` metadata field and `%{[@metadata][version]}` sets the second part to the Beat's version. For example: +{beat_default_index_prefix}-{version}+. +endif::[] + +ifdef::apm-server[] +[source,logstash] +------ +input { + beats { + port => 5044 + } +} + +filter { + if [@metadata][beat] == "apm" { + if [processor][event] == "sourcemap" { + mutate { + add_field => { "[@metadata][index]" => "%{[@metadata][beat]}-%{[@metadata][version]}-%{[processor][event]}" } <1> + } + } else { + mutate { + add_field => { "[@metadata][index]" => "%{[@metadata][beat]}-%{[@metadata][version]}-%{[processor][event]}-%{+yyyy.MM.dd}" } <2> + } + } + } +} + +output { + elasticsearch { + hosts => ["http://localhost:9200"] + index => "%{[@metadata][index]}" + } +} +------ +<1> Creates a new field named `@metadata.index`. +`%{[@metadata][beat]}` sets the first part of the index name to the value of the `beat` metadata field. +`%{[@metadata][version]}` sets the second part to {beatname_uc}'s version. +`%{[processor][event]}` sets the final part based on the APM event type. +For example: +{beat_default_index_prefix}-{version}-sourcemap+. +<2> In addition to the above rules, this pattern appends a date to the `index` name so Logstash creates a new index each day. +For example: +{beat_default_index_prefix}-{version}-transaction-{sample_date_0}+. +endif::[] Events indexed into Elasticsearch with the Logstash configuration shown here -will be similar to events directly indexed by Beats into Elasticsearch. +will be similar to events directly indexed by {beatname_uc} into Elasticsearch. +ifndef::apm-server[] NOTE: If ILM is not being used, set `index` to `%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}` instead so Logstash creates an index per day, based on the `@timestamp` value of the events coming from Beats. +endif::[] + +ifdef::apm-server[] +==== Logstash and ILM + +When used with {apm-server-ref}/manual-ilm-setup.html[Index lifecycle management], Logstash does not need to create a new index each day. +Here's a sample Logstash configuration file that would accomplish this: + +[source,logstash] +------ +input { + beats { + port => 5044 + } +} + +output { + elasticsearch { + hosts => ["http://localhost:9200"] + index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{[processor][event]}" + } +} +------ +endif::[] ==== Compatibility @@ -796,7 +879,7 @@ You can specify the following options in the `logstash` section of the The enabled config is a boolean setting to enable or disable the output. If set to false, the output is disabled. -The default value is true. +The default value is `true`. [[hosts]] ===== `hosts` diff --git a/libbeat/docs/shared-securing-beat.asciidoc b/libbeat/docs/shared-securing-beat.asciidoc index 7677da1a876d..d8f2b76383c8 100644 --- a/libbeat/docs/shared-securing-beat.asciidoc +++ b/libbeat/docs/shared-securing-beat.asciidoc @@ -8,6 +8,9 @@ The following topics provide information about securing the {beatname_uc} process and securing communication between {beatname_uc} and other products in the Elastic stack: +ifdef::apm-server[] +* <> +endif::[] * <> ifndef::only-elasticsearch[] * <> @@ -21,10 +24,14 @@ ifdef::beat-specific-security[] include::{beat-specific-security}[] endif::[] -//sets block macro for https.asciidoc included in next section - -- +ifdef::apm-server[] +include::../secure-communication-agents.asciidoc[] +endif::[] + +//sets block macro for https.asciidoc included in next section + [[securing-communication-elasticsearch]] == Secure communication with Elasticsearch @@ -34,7 +41,7 @@ include::./https.asciidoc[] ifndef::only-elasticsearch[] [[configuring-ssl-logstash]] -== Secure communication with Logstash by using SSL +== Secure communication with Logstash include::./shared-ssl-logstash-config.asciidoc[] endif::only-elasticsearch[] diff --git a/libbeat/docs/shared-ssl-config.asciidoc b/libbeat/docs/shared-ssl-config.asciidoc index 0f4d649fb9d8..c835e936dc21 100644 --- a/libbeat/docs/shared-ssl-config.asciidoc +++ b/libbeat/docs/shared-ssl-config.asciidoc @@ -1,18 +1,25 @@ [[configuration-ssl]] +ifndef::apm-server[] == Specify SSL settings +endif::apm-server[] +ifdef::apm-server[] +== SSL output settings +You can specify SSL options with any output that supports SSL, like {es}, {ls}, or Kafka. +endif::[] + +ifndef::apm-server[] You can specify SSL options when you configure: * <> that support SSL -ifeval::["{beatname_lc}"!="apm-server"] * the <> -endif::[] ifeval::["{beatname_lc}"=="heartbeat"] * <> that support SSL endif::[] ifeval::["{beatname_lc}"=="metricbeat"] * <> that define the host as an HTTP URL endif::[] +endif::[] Example output config with SSL enabled: diff --git a/libbeat/docs/shared-ssl-logstash-config.asciidoc b/libbeat/docs/shared-ssl-logstash-config.asciidoc index a7b28c7fe99a..090b34cfc338 100644 --- a/libbeat/docs/shared-ssl-logstash-config.asciidoc +++ b/libbeat/docs/shared-ssl-logstash-config.asciidoc @@ -126,7 +126,7 @@ curl: (51) SSL: certificate verification failed (result: 5) See the <> for info about resolving this issue. [float] -=== Test the Beats to Logstash connection +=== Test the {beatname_uc} to Logstash connection If you have {beatname_uc} running as a service, first stop the service. Then test your setup by running {beatname_uc} in the foreground so you can quickly see any errors that occur: