[docs][7.0] documentation backports

docs/common-problems.asciidoc

@@ -5,6 +5,7 @@ This section describes common problems you might encounter with APM Server.
 
 * <<no-data-indexed>>
 * <<bad-request>>
+* <<event-too-large>>
 * <<unauthorized>>
 * <<forbidden>>
 * <<queue-full>>
@@ -51,7 +52,7 @@ As a result, Elasticsearch must be configured to allow {ref}/docs-index_.html#in
 === HTTP 400: Data decoding error / Data validation error
 
 The most likely cause for this is that you are using incompatible versions of agent and APM Server.
-For instance, APM Server 6.2.0 changed the Intake API spec and requires a minimum version of each agent.
+For instance, APM Server 6.2 and 6.5 changed the Intake API spec and require a minimum version of each agent.
 
 View the {apm-overview-ref-v}/agent-server-compatibility.html[agent/server compatibility matrix] for more information.
 

docs/configuration-process.asciidoc

@@ -6,15 +6,31 @@ Example config file:
 ["source","yaml"]
 ----
 apm-server:
-  host: "localhost:8200"
+  hosts: ["localhost:8200"]
   rum:
     enabled: true
 
+output:
+  elasticsearch:
+    hosts: ElasticsearchAddress:9200
+
 queue.mem.events: 4096
 
 max_procs: 4
 ----
 
+NOTE: If you are using an X-Pack secured version of Elastic Stack,
+you need to specify credentials in the config file before you run the commands that set up and start APM Server.
+For example:
+
+[source,yaml]
+----
+output.elasticsearch:
+  hosts: ["ElasticsearchAddress:9200"]
+  username: "elastic"
+  password: "elastic"
+----
+
 [float]
 [[configuration-apm-server]]
 === Configuration options: `apm-server.*`

docs/copied-from-beats/command-reference.asciidoc

@@ -11,6 +11,7 @@
 
 
 // These attributes are used to resolve short descriptions
+tag::attributes[]
 
 :global-flags: Also see <<global-flags,Global flags>>.
 
@@ -47,6 +48,7 @@ endif::[]
 :test-command-short-desc: Tests the configuration
 :version-command-short-desc: Shows information about the current version
 
+end::attributes[]
 
 [[command-line-options]]
 === {beatname_uc} command reference
@@ -584,6 +586,10 @@ Or:
 {setup-command-short-desc}
 
 * The index template ensures that fields are mapped correctly in Elasticsearch.
+If index lifecycle management is enabled it also ensures that the defined ILM policy
+and write alias are connected to the indices matching the index template.
+The ILM policy takes care of the lifecycle of an index, when to do a rollover,
+when to move an index from the hot phase to the next phase, etc.
 
 ifndef::no_dashboards[]
 * The {kib} dashboards make it easier for you to visualize {beatname_uc} data
@@ -595,11 +601,12 @@ ifdef::has_ml_jobs[]
 necessary to analyze data for anomalies.
 endif::[]
 
-Use this command if you want to set up the environment without actually running
+This command sets up the environment without actually running
 {beatname_uc} and ingesting data.
 
 *SYNOPSIS*
 
+tag::setup-command-tag[]
 ["source","sh",subs="attributes"]
 ----
 {beatname_lc} setup [FLAGS]
@@ -636,6 +643,15 @@ enabled modules in the +{beatname_lc}.yml+ file. If you used the
 directory, also specify the `--modules` flag.
 endif::[]
 
+*`--index-management`*::
+Sets up components related to Elasticsearch index management including
+template, ilm policy, and write alias.
+
+ifdef::apm-server[]
+*`--pipelines`*::
+Registers the <<configuring-ingest-node,pipeline>> definitions set in `ingest/pipeline/definition.json`.
+end::apm-server[]
+
 *`--template`*::
 Sets up the index template only.
 
@@ -666,6 +682,27 @@ ifeval::["{beatname_lc}"!="filebeat"]
 -----
 endif::[]
 
+ifndef::apm-server[]
+ifdef::no_dashboards[]
+["source","sh",subs="attributes"]
+-----
+{beatname_lc} setup --machine-learning
+{beatname_lc} setup --index-management
+-----
+endif::no_dashboards[]
+endif::apm-server[]
+
+ifdef::apm-server[]
+["source","sh",subs="attributes"]
+-----
+{beatname_lc} setup --index-management
+{beatname_lc} setup --pipelines
+-----
+endif::apm-server[]
+
+endif::[]
+end::setup-command-tag[]
+
 [[test-command]]
 ==== `test` command
 

docs/copied-from-beats/outputconfig.asciidoc

@@ -712,14 +712,14 @@ include::./shared-logstash-config.asciidoc[]
 Every event sent to Logstash contains the following metadata fields that you can
 use in Logstash for indexing and filtering:
 
+ifndef::apm-server[]
 ["source","json",subs="attributes"]
 ------------------------------------------------------------------------------
 {
     ...
     "@metadata": { <1>
       "beat": "{beat_default_index_prefix}", <2>
       "version": "{stack-version}" <3>
-      "type": "doc" <4>
     }
 }
 ------------------------------------------------------------------------------
@@ -728,21 +728,39 @@ use in Logstash for indexing and filtering:
 for more about the `@metadata` field.
 <2> The default is {beat_default_index_prefix}. To change this value, set the
 <<logstash-index,`index`>> option in the {beatname_uc} config file.
-<3> The beats current version.
-<4> The value of `type` is currently hardcoded to `doc`. It was used by previous
-Logstash configs to set the type of the document in Elasticsearch.
-
+<3> The current version of {beatname_uc}.
+endif::[]
 
-WARNING: The `@metadata.type` field, added by the Logstash output, is
-deprecated, hardcoded to `doc`, and will be removed in {beatname_uc} 7.0.
+ifdef::apm-server[]
+["source","json",subs="attributes"]
+------------------------------------------------------------------------------
+{
+    ...
+    "@metadata": { <1>
+      "beat": "{beat_default_index_prefix}", <2>
+      "pipeline":"apm", <3>
+      "version": "{stack-version}" <4>
+    }
+}
+------------------------------------------------------------------------------
+<1> {beatname_uc} uses the `@metadata` field to send metadata to Logstash. See the
+{logstash-ref}/event-dependent-configuration.html#metadata[Logstash documentation]
+for more about the `@metadata` field.
+<2> The default is {beat_default_index_prefix}. To change this value, set the
+<<logstash-index,`index`>> option in the {beatname_uc} config file.
+<3> The default pipeline configuration: `apm`. Additional pipelines can be enabled
+with a {logstash-ref}/use-ingest-pipelines.html[Logstash pipeline config].
+<4> The current version of {beatname_uc}.
+endif::[]
 
 You can access this metadata from within the Logstash config file to set values
 dynamically based on the contents of the metadata.
 
-For example, the following Logstash configuration file for versions 2.x and
-5.x sets Logstash to use the index and document type reported by Beats for
-indexing events into Elasticsearch:
+For example, the following Logstash configuration file for version 7.x sets
+Logstash to use the index reported by {beatname_uc} for indexing events
+into Elasticsearch:
 
+ifndef::apm-server[]
 [source,logstash]
 ------------------------------------------------------------------------------
 
@@ -760,14 +778,79 @@ output {
 }
 ------------------------------------------------------------------------------
 <1> `%{[@metadata][beat]}` sets the first part of the index name to the value
-of the `beat` metadata field, `%{[@metadata][version]}` sets the second part to
-the Beat's version, and `%{+YYYY.MM.dd}` sets the third part of the
-name to a date based on the Logstash `@timestamp` field. For example:
-+{beat_default_index_prefix}-{version}-2017.03.29+.
+of the `beat` metadata field and `%{[@metadata][version]}` sets the second part to
+the Beat's version. For example:
++{beat_default_index_prefix}-{version}+.
+endif::[]
+
+ifdef::apm-server[]
+[source,logstash]
+------
+input {
+    beats {
+        port => 5044
+    }
+}
+
+filter {
+    if [@metadata][beat] == "apm" {
+        if [processor][event] == "sourcemap" {
+            mutate {
+                add_field => { "[@metadata][index]" => "%{[@metadata][beat]}-%{[@metadata][version]}-%{[processor][event]}" } <1>
+            }
+        } else {
+            mutate {
+                add_field => { "[@metadata][index]" => "%{[@metadata][beat]}-%{[@metadata][version]}-%{[processor][event]}-%{+yyyy.MM.dd}" } <2>
+            }
+        }
+    }
+}
+
+output {
+    elasticsearch {
+        hosts => ["http://localhost:9200"]
+        index => "%{[@metadata][index]}"
+    }
+}
+------
+<1> Creates a new field named `@metadata.index`.
+`%{[@metadata][beat]}` sets the first part of the index name to the value of the `beat` metadata field.
+`%{[@metadata][version]}` sets the second part to {beatname_uc}'s version.
+`%{[processor][event]}` sets the final part based on the APM event type.
+For example: +{beat_default_index_prefix}-{version}-sourcemap+.
+<2> In addition to the above rules, this pattern appends a date to the `index` name so Logstash creates a new index each day.
+For example: +{beat_default_index_prefix}-{version}-transaction-{sample_date_0}+.
+endif::[]
 
 Events indexed into Elasticsearch with the Logstash configuration shown here
-will be similar to events directly indexed by Beats into Elasticsearch.
+will be similar to events directly indexed by {beatname_uc} into Elasticsearch.
+
+ifndef::apm-server[]
+NOTE: If ILM is not being used, set `index` to `%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}` instead so Logstash creates an index per day, based on the `@timestamp` value of the events coming from Beats.
+endif::[]
 
+ifdef::apm-server[]
+==== Logstash and ILM
+
+When used with {apm-server-ref}/manual-ilm-setup.html[Index lifecycle management], Logstash does not need to create a new index each day.
+Here's a sample Logstash configuration file that would accomplish this:
+
+[source,logstash]
+------
+input {
+    beats {
+        port => 5044
+    }
+}
+
+output {
+    elasticsearch {
+        hosts => ["http://localhost:9200"]
+        index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{[processor][event]}"
+    }
+}
+------
+endif::[]
 
 ==== Compatibility
 
@@ -785,7 +868,7 @@ You can specify the following options in the `logstash` section of the
 The enabled config is a boolean setting to enable or disable the output. If set
 to false, the output is disabled.
 
-The default value is true.
+The default value is `true`.
 
 [[hosts]]
 ===== `hosts`

docs/data-ingestion.asciidoc

@@ -13,8 +13,6 @@ This section explains how to adapt data ingestion according to your needs.
 [[tune-apm-server]]
 == Tune APM Server
 
-Tuning topics:
-
 * <<tune-output-config>>
 * <<adjust-queue-size>>
 * <<add-apm-server-instances>>
@@ -87,15 +85,18 @@ Increasing the <<event_rate.limit,`event_rate.limit`>> default value will help a
 [[tune-es]]
 == Tune Elasticsearch
 
-Get insights about tuning the Elasticsearch ingestion rate,
-especially with regards to
+The Elasticsearch reference provides insight on tuning Elasticsearch.
+
+{ref}/tune-for-indexing-speed.html[Tune for indexing speed] provides information on:
 
-* refresh interval
-* disable swapping
-* optimizing filesystem cache
-* considerations regarding faster hardware
-* setting the indexing buffer size
+* Refresh interval
+* Disabling swapping
+* Optimizing filesystem cache
+* Considerations regarding faster hardware
+* Setting the indexing buffer size
 
-directly at the {ref}/tune-for-indexing-speed.html[Elasticsearch reference].
+{ref}/tune-for-disk-usage.html[Tune for disk usage] provides information on:
 
-You might also want to read up on how to {ref}/tune-for-disk-usage.html[tune for disk usage].
+* Disabling unneeded features
+* Shard size
+* Shrink index