[docs][7.x] documentation backports

docs/common-problems.asciidoc

@@ -5,6 +5,7 @@ This section describes common problems you might encounter with APM Server.
 
 * <<no-data-indexed>>
 * <<bad-request>>
+* <<event-too-large>>
 * <<unauthorized>>
 * <<forbidden>>
 * <<queue-full>>
@@ -51,7 +52,7 @@ As a result, Elasticsearch must be configured to allow {ref}/docs-index_.html#in
 === HTTP 400: Data decoding error / Data validation error
 
 The most likely cause for this is that you are using incompatible versions of agent and APM Server.
-For instance, APM Server 6.2.0 changed the Intake API spec and requires a minimum version of each agent.
+For instance, APM Server 6.2 and 6.5 changed the Intake API spec and require a minimum version of each agent.
 
 View the {apm-overview-ref-v}/agent-server-compatibility.html[agent/server compatibility matrix] for more information.
 

docs/configuration-process.asciidoc

@@ -6,15 +6,31 @@ Example config file:
 ["source","yaml"]
 ----
 apm-server:
-  host: "localhost:8200"
+  hosts: ["localhost:8200"]
   rum:
     enabled: true
 
+output:
+  elasticsearch:
+    hosts: ElasticsearchAddress:9200
+
 queue.mem.events: 4096
 
 max_procs: 4
 ----
 
+NOTE: If you are using an X-Pack secured version of Elastic Stack,
+you need to specify credentials in the config file before you run the commands that set up and start APM Server.
+For example:
+
+[source,yaml]
+----
+output.elasticsearch:
+  hosts: ["ElasticsearchAddress:9200"]
+  username: "elastic"
+  password: "elastic"
+----
+
 [float]
 [[configuration-apm-server]]
 === Configuration options: `apm-server.*`

docs/configuring.asciidoc

@@ -8,7 +8,7 @@ include::./copied-from-beats/shared-configuring.asciidoc[]
 * <<configuration-process>>
 * <<configuring-output>>
 * <<configuring-ingest-node>>
-* <<configuration-ssl>>
+* <<configuration-ssl-landing>>
 * <<configuration-template>>
 * <<ilm>>
 * <<configuration-logging>>
@@ -25,7 +25,28 @@ include::./copied-from-beats/outputconfig.asciidoc[]
 
 include::./configuring-ingest.asciidoc[]
 
+[[configuration-ssl-landing]]
+== SSL/TLS settings
+
+SSL/TLS is available for:
+
+* <<agent-server-ssl,APM Server **inputs**>> (APM Agents)
+* <<configuration-ssl,APM Server **outputs**>> that support SSL, like {es}, {ls}, or Kafka.
+
+Additional information on getting started with SSL/TLS is available in <<securing-apm-server>>.
+
+// The leveloffset attribute pushes all headings in the included document down by
+// the specified number of levels. It is required here because the shared Beats
+// documentation was created as a level 1 heading. In the APM book, this level
+// would break the DTD. Using leveloffset +1, we can include this file here.
+// It's important too reset the level heading after including the file.
+:leveloffset: +1
 include::./copied-from-beats/shared-ssl-config.asciidoc[]
+:leveloffset: -1
+
+include::ssl-input-settings.asciidoc[]
+
+See <<securing-apm-server,Securing APM Server>> for more information.
 
 include::./template-config.asciidoc[]
 

docs/copied-from-beats/command-reference.asciidoc

@@ -11,6 +11,7 @@
 
 
 // These attributes are used to resolve short descriptions
+tag::attributes[]
 
 :global-flags: Also see <<global-flags,Global flags>>.
 
@@ -47,6 +48,7 @@ endif::[]
 :test-command-short-desc: Tests the configuration
 :version-command-short-desc: Shows information about the current version
 
+end::attributes[]
 
 [[command-line-options]]
 === {beatname_uc} command reference
@@ -599,7 +601,7 @@ Or:
 If index lifecycle management is enabled it also ensures that the defined ILM policy
 and write alias are connected to the indices matching the index template.
 The ILM policy takes care of the lifecycle of an index, when to do a rollover,
-when to move an index from the hot phase to the next phase etc.
+when to move an index from the hot phase to the next phase, etc.
 
 ifndef::no_dashboards[]
 * The {kib} dashboards make it easier for you to visualize {beatname_uc} data
@@ -611,11 +613,12 @@ ifdef::has_ml_jobs[]
 necessary to analyze data for anomalies.
 endif::[]
 
-Use this command if you want to set up the environment without actually running
+This command sets up the environment without actually running
 {beatname_uc} and ingesting data.
 
 *SYNOPSIS*
 
+tag::setup-command-tag[]
 ["source","sh",subs="attributes"]
 ----
 {beatname_lc} setup [FLAGS]
@@ -656,6 +659,11 @@ endif::[]
 Sets up components related to Elasticsearch index management including
 template, ILM policy, and write alias (if supported and configured).
 
+ifdef::apm-server[]
+*`--pipelines`*::
+Registers the <<configuring-ingest-node,pipeline>> definitions set in `ingest/pipeline/definition.json`.
+end::apm-server[]
+
 *`--template`*::
 deprecated:[7.2]
 Sets up the index template only.
@@ -694,15 +702,27 @@ ifndef::no_dashboards[]
 {beatname_lc} setup --index-management
 -----
 endif::no_dashboards[]
+
+ifndef::apm-server[]
 ifdef::no_dashboards[]
 ["source","sh",subs="attributes"]
 -----
 {beatname_lc} setup --machine-learning
 {beatname_lc} setup --index-management
 -----
 endif::no_dashboards[]
+endif::apm-server[]
+
+ifdef::apm-server[]
+["source","sh",subs="attributes"]
+-----
+{beatname_lc} setup --index-management
+{beatname_lc} setup --pipelines
+-----
+endif::apm-server[]
 
 endif::[]
+end::setup-command-tag[]
 
 [[test-command]]
 ==== `test` command

docs/copied-from-beats/https.asciidoc

@@ -41,7 +41,7 @@ this document.
 By default {beatname_uc} uses the list of trusted certificate authorities from the
 operating system where {beatname_uc} is running. You can configure {beatname_uc} to use a specific list of
 CA certificates instead of the list from the OS. You can also configure it to use client authentication
-by specifying the certificate and key to use when the server requires the Beat to authenticate. Here is an example
+by specifying the certificate and key to use when the server requires the {beatname_uc} to authenticate. Here is an example
 configuration:
 
 ["source","yaml",subs="attributes,callouts"]

docs/copied-from-beats/outputconfig.asciidoc

@@ -723,14 +723,14 @@ include::./shared-logstash-config.asciidoc[]
 Every event sent to Logstash contains the following metadata fields that you can
 use in Logstash for indexing and filtering:
 
+ifndef::apm-server[]
 ["source","json",subs="attributes"]
 ------------------------------------------------------------------------------
 {
     ...
     "@metadata": { <1>
       "beat": "{beat_default_index_prefix}", <2>
       "version": "{stack-version}" <3>
-      "type": "doc" <4>
     }
 }
 ------------------------------------------------------------------------------
@@ -739,21 +739,39 @@ use in Logstash for indexing and filtering:
 for more about the `@metadata` field.
 <2> The default is {beat_default_index_prefix}. To change this value, set the
 <<logstash-index,`index`>> option in the {beatname_uc} config file.
-<3> The beats current version.
-<4> The value of `type` is currently hardcoded to `doc`. It was used by previous
-Logstash configs to set the type of the document in Elasticsearch.
-
+<3> The current version of {beatname_uc}.
+endif::[]
 
-WARNING: The `@metadata.type` field, added by the Logstash output, is
-deprecated, hardcoded to `doc`, and will be removed in {beatname_uc} 7.0.
+ifdef::apm-server[]
+["source","json",subs="attributes"]
+------------------------------------------------------------------------------
+{
+    ...
+    "@metadata": { <1>
+      "beat": "{beat_default_index_prefix}", <2>
+      "pipeline":"apm", <3>
+      "version": "{stack-version}" <4>
+    }
+}
+------------------------------------------------------------------------------
+<1> {beatname_uc} uses the `@metadata` field to send metadata to Logstash. See the
+{logstash-ref}/event-dependent-configuration.html#metadata[Logstash documentation]
+for more about the `@metadata` field.
+<2> The default is {beat_default_index_prefix}. To change this value, set the
+<<logstash-index,`index`>> option in the {beatname_uc} config file.
+<3> The default pipeline configuration: `apm`. Additional pipelines can be enabled
+with a {logstash-ref}/use-ingest-pipelines.html[Logstash pipeline config].
+<4> The current version of {beatname_uc}.
+endif::[]
 
 You can access this metadata from within the Logstash config file to set values
 dynamically based on the contents of the metadata.
 
-For example, the following Logstash configuration file for versions 2.x and
-5.x sets Logstash to use the index and document type reported by Beats for
-indexing events into Elasticsearch:
+For example, the following Logstash configuration file for version 7.x sets
+Logstash to use the index reported by {beatname_uc} for indexing events
+into Elasticsearch:
 
+ifndef::apm-server[]
 [source,logstash]
 ------------------------------------------------------------------------------
 
@@ -774,11 +792,76 @@ output {
 of the `beat` metadata field and `%{[@metadata][version]}` sets the second part to
 the Beat's version. For example:
 +{beat_default_index_prefix}-{version}+.
+endif::[]
+
+ifdef::apm-server[]
+[source,logstash]
+------
+input {
+    beats {
+        port => 5044
+    }
+}
+
+filter {
+    if [@metadata][beat] == "apm" {
+        if [processor][event] == "sourcemap" {
+            mutate {
+                add_field => { "[@metadata][index]" => "%{[@metadata][beat]}-%{[@metadata][version]}-%{[processor][event]}" } <1>
+            }
+        } else {
+            mutate {
+                add_field => { "[@metadata][index]" => "%{[@metadata][beat]}-%{[@metadata][version]}-%{[processor][event]}-%{+yyyy.MM.dd}" } <2>
+            }
+        }
+    }
+}
+
+output {
+    elasticsearch {
+        hosts => ["http://localhost:9200"]
+        index => "%{[@metadata][index]}"
+    }
+}
+------
+<1> Creates a new field named `@metadata.index`.
+`%{[@metadata][beat]}` sets the first part of the index name to the value of the `beat` metadata field.
+`%{[@metadata][version]}` sets the second part to {beatname_uc}'s version.
+`%{[processor][event]}` sets the final part based on the APM event type.
+For example: +{beat_default_index_prefix}-{version}-sourcemap+.
+<2> In addition to the above rules, this pattern appends a date to the `index` name so Logstash creates a new index each day.
+For example: +{beat_default_index_prefix}-{version}-transaction-{sample_date_0}+.
+endif::[]
 
 Events indexed into Elasticsearch with the Logstash configuration shown here
-will be similar to events directly indexed by Beats into Elasticsearch.
+will be similar to events directly indexed by {beatname_uc} into Elasticsearch.
 
+ifndef::apm-server[]
 NOTE: If ILM is not being used, set `index` to `%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}` instead so Logstash creates an index per day, based on the `@timestamp` value of the events coming from Beats.
+endif::[]
+
+ifdef::apm-server[]
+==== Logstash and ILM
+
+When used with {apm-server-ref}/manual-ilm-setup.html[Index lifecycle management], Logstash does not need to create a new index each day.
+Here's a sample Logstash configuration file that would accomplish this:
+
+[source,logstash]
+------
+input {
+    beats {
+        port => 5044
+    }
+}
+
+output {
+    elasticsearch {
+        hosts => ["http://localhost:9200"]
+        index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{[processor][event]}"
+    }
+}
+------
+endif::[]
 
 ==== Compatibility
 
@@ -796,7 +879,7 @@ You can specify the following options in the `logstash` section of the
 The enabled config is a boolean setting to enable or disable the output. If set
 to false, the output is disabled.
 
-The default value is true.
+The default value is `true`.
 
 [[hosts]]
 ===== `hosts`

docs/copied-from-beats/shared-securing-beat.asciidoc

@@ -8,6 +8,9 @@ The following topics provide information about securing the {beatname_uc}
 process and securing communication between {beatname_uc} and other products in
 the Elastic stack:
 
+ifdef::apm-server[]
+* <<secure-communication-agents>>
+endif::[]
 * <<securing-communication-elasticsearch>>
 ifndef::only-elasticsearch[]
 * <<configuring-ssl-logstash>>
@@ -21,10 +24,14 @@ ifdef::beat-specific-security[]
 include::{beat-specific-security}[]
 endif::[]
 
-//sets block macro for https.asciidoc included in next section
-
 --
 
+ifdef::apm-server[]
+include::../secure-communication-agents.asciidoc[]
+endif::[]
+
+//sets block macro for https.asciidoc included in next section
+
 [[securing-communication-elasticsearch]]
 == Secure communication with Elasticsearch
 
@@ -34,7 +41,7 @@ include::./https.asciidoc[]
 
 ifndef::only-elasticsearch[]
 [[configuring-ssl-logstash]]
-== Secure communication with Logstash by using SSL
+== Secure communication with Logstash
 
 include::./shared-ssl-logstash-config.asciidoc[]
 endif::only-elasticsearch[]

docs/copied-from-beats/shared-ssl-config.asciidoc

@@ -1,18 +1,25 @@
 [[configuration-ssl]]
+ifndef::apm-server[]
 == Specify SSL settings
+endif::apm-server[]
+ifdef::apm-server[]
+== SSL output settings
 
+You can specify SSL options with any output that supports SSL, like {es}, {ls}, or Kafka.
+endif::[]
+
+ifndef::apm-server[]
 You can specify SSL options when you configure:
 
 * <<configuring-output,outputs>> that support SSL
-ifeval::["{beatname_lc}"!="apm-server"]
 * the <<setup-kibana-endpoint,Kibana endpoint>>
-endif::[]
 ifeval::["{beatname_lc}"=="heartbeat"]
 * <<configuration-heartbeat-options,{beatname_uc} monitors>> that support SSL
 endif::[]
 ifeval::["{beatname_lc}"=="metricbeat"]
 * <<metricbeat-modules,modules>> that define the host as an HTTP URL
 endif::[]
+endif::[]
 
 Example output config with SSL enabled: