diff --git a/docs/common-problems.asciidoc b/docs/common-problems.asciidoc index 1ca725c8e8b..7417b6ea800 100644 --- a/docs/common-problems.asciidoc +++ b/docs/common-problems.asciidoc @@ -5,6 +5,7 @@ This section describes common problems you might encounter with APM Server. * <> * <> +* <> * <> * <> * <> @@ -51,7 +52,7 @@ As a result, Elasticsearch must be configured to allow {ref}/docs-index_.html#in === HTTP 400: Data decoding error / Data validation error The most likely cause for this is that you are using incompatible versions of agent and APM Server. -For instance, APM Server 6.2.0 changed the Intake API spec and requires a minimum version of each agent. +For instance, APM Server 6.2 and 6.5 changed the Intake API spec and require a minimum version of each agent. View the {apm-overview-ref-v}/agent-server-compatibility.html[agent/server compatibility matrix] for more information. diff --git a/docs/configuration-process.asciidoc b/docs/configuration-process.asciidoc index 53586ae12c8..ba88861fdb2 100644 --- a/docs/configuration-process.asciidoc +++ b/docs/configuration-process.asciidoc @@ -6,15 +6,31 @@ Example config file: ["source","yaml"] ---- apm-server: - host: "localhost:8200" + hosts: ["localhost:8200"] rum: enabled: true +output: + elasticsearch: + hosts: ElasticsearchAddress:9200 + queue.mem.events: 4096 max_procs: 4 ---- +NOTE: If you are using an X-Pack secured version of Elastic Stack, +you need to specify credentials in the config file before you run the commands that set up and start APM Server. +For example: + +[source,yaml] +---- +output.elasticsearch: + hosts: ["ElasticsearchAddress:9200"] + username: "elastic" + password: "elastic" +---- + [float] [[configuration-apm-server]] === Configuration options: `apm-server.*` diff --git a/docs/configuring.asciidoc b/docs/configuring.asciidoc index 8d498f4bbb5..a3fea93fb9e 100644 --- a/docs/configuring.asciidoc +++ b/docs/configuring.asciidoc @@ -8,7 +8,7 @@ include::./copied-from-beats/shared-configuring.asciidoc[] * <> * <> * <> -* <> +* <> * <> * <> * <> @@ -25,7 +25,28 @@ include::./copied-from-beats/outputconfig.asciidoc[] include::./configuring-ingest.asciidoc[] +[[configuration-ssl-landing]] +== SSL/TLS settings + +SSL/TLS is available for: + +* <> (APM Agents) +* <> that support SSL, like {es}, {ls}, or Kafka. + +Additional information on getting started with SSL/TLS is available in <>. + +// The leveloffset attribute pushes all headings in the included document down by +// the specified number of levels. It is required here because the shared Beats +// documentation was created as a level 1 heading. In the APM book, this level +// would break the DTD. Using leveloffset +1, we can include this file here. +// It's important too reset the level heading after including the file. +:leveloffset: +1 include::./copied-from-beats/shared-ssl-config.asciidoc[] +:leveloffset: -1 + +include::ssl-input-settings.asciidoc[] + +See <> for more information. include::./template-config.asciidoc[] diff --git a/docs/copied-from-beats/command-reference.asciidoc b/docs/copied-from-beats/command-reference.asciidoc index 42f7fa695ff..1688aa49cc2 100644 --- a/docs/copied-from-beats/command-reference.asciidoc +++ b/docs/copied-from-beats/command-reference.asciidoc @@ -11,6 +11,7 @@ // These attributes are used to resolve short descriptions +tag::attributes[] :global-flags: Also see <>. @@ -47,6 +48,7 @@ endif::[] :test-command-short-desc: Tests the configuration :version-command-short-desc: Shows information about the current version +end::attributes[] [[command-line-options]] === {beatname_uc} command reference @@ -599,7 +601,7 @@ Or: If index lifecycle management is enabled it also ensures that the defined ILM policy and write alias are connected to the indices matching the index template. The ILM policy takes care of the lifecycle of an index, when to do a rollover, -when to move an index from the hot phase to the next phase etc. +when to move an index from the hot phase to the next phase, etc. ifndef::no_dashboards[] * The {kib} dashboards make it easier for you to visualize {beatname_uc} data @@ -611,11 +613,12 @@ ifdef::has_ml_jobs[] necessary to analyze data for anomalies. endif::[] -Use this command if you want to set up the environment without actually running +This command sets up the environment without actually running {beatname_uc} and ingesting data. *SYNOPSIS* +tag::setup-command-tag[] ["source","sh",subs="attributes"] ---- {beatname_lc} setup [FLAGS] @@ -656,6 +659,11 @@ endif::[] Sets up components related to Elasticsearch index management including template, ILM policy, and write alias (if supported and configured). +ifdef::apm-server[] +*`--pipelines`*:: +Registers the <> definitions set in `ingest/pipeline/definition.json`. +end::apm-server[] + *`--template`*:: deprecated:[7.2] Sets up the index template only. @@ -694,6 +702,8 @@ ifndef::no_dashboards[] {beatname_lc} setup --index-management ----- endif::no_dashboards[] + +ifndef::apm-server[] ifdef::no_dashboards[] ["source","sh",subs="attributes"] ----- @@ -701,8 +711,18 @@ ifdef::no_dashboards[] {beatname_lc} setup --index-management ----- endif::no_dashboards[] +endif::apm-server[] + +ifdef::apm-server[] +["source","sh",subs="attributes"] +----- +{beatname_lc} setup --index-management +{beatname_lc} setup --pipelines +----- +endif::apm-server[] endif::[] +end::setup-command-tag[] [[test-command]] ==== `test` command diff --git a/docs/copied-from-beats/https.asciidoc b/docs/copied-from-beats/https.asciidoc index f8eab291d49..06915d9776c 100644 --- a/docs/copied-from-beats/https.asciidoc +++ b/docs/copied-from-beats/https.asciidoc @@ -41,7 +41,7 @@ this document. By default {beatname_uc} uses the list of trusted certificate authorities from the operating system where {beatname_uc} is running. You can configure {beatname_uc} to use a specific list of CA certificates instead of the list from the OS. You can also configure it to use client authentication -by specifying the certificate and key to use when the server requires the Beat to authenticate. Here is an example +by specifying the certificate and key to use when the server requires the {beatname_uc} to authenticate. Here is an example configuration: ["source","yaml",subs="attributes,callouts"] diff --git a/docs/copied-from-beats/outputconfig.asciidoc b/docs/copied-from-beats/outputconfig.asciidoc index 2e8b065617f..e94c7360c22 100644 --- a/docs/copied-from-beats/outputconfig.asciidoc +++ b/docs/copied-from-beats/outputconfig.asciidoc @@ -723,6 +723,7 @@ include::./shared-logstash-config.asciidoc[] Every event sent to Logstash contains the following metadata fields that you can use in Logstash for indexing and filtering: +ifndef::apm-server[] ["source","json",subs="attributes"] ------------------------------------------------------------------------------ { @@ -730,7 +731,6 @@ use in Logstash for indexing and filtering: "@metadata": { <1> "beat": "{beat_default_index_prefix}", <2> "version": "{stack-version}" <3> - "type": "doc" <4> } } ------------------------------------------------------------------------------ @@ -739,21 +739,39 @@ use in Logstash for indexing and filtering: for more about the `@metadata` field. <2> The default is {beat_default_index_prefix}. To change this value, set the <> option in the {beatname_uc} config file. -<3> The beats current version. -<4> The value of `type` is currently hardcoded to `doc`. It was used by previous -Logstash configs to set the type of the document in Elasticsearch. - +<3> The current version of {beatname_uc}. +endif::[] -WARNING: The `@metadata.type` field, added by the Logstash output, is -deprecated, hardcoded to `doc`, and will be removed in {beatname_uc} 7.0. +ifdef::apm-server[] +["source","json",subs="attributes"] +------------------------------------------------------------------------------ +{ + ... + "@metadata": { <1> + "beat": "{beat_default_index_prefix}", <2> + "pipeline":"apm", <3> + "version": "{stack-version}" <4> + } +} +------------------------------------------------------------------------------ +<1> {beatname_uc} uses the `@metadata` field to send metadata to Logstash. See the +{logstash-ref}/event-dependent-configuration.html#metadata[Logstash documentation] +for more about the `@metadata` field. +<2> The default is {beat_default_index_prefix}. To change this value, set the +<> option in the {beatname_uc} config file. +<3> The default pipeline configuration: `apm`. Additional pipelines can be enabled +with a {logstash-ref}/use-ingest-pipelines.html[Logstash pipeline config]. +<4> The current version of {beatname_uc}. +endif::[] You can access this metadata from within the Logstash config file to set values dynamically based on the contents of the metadata. -For example, the following Logstash configuration file for versions 2.x and -5.x sets Logstash to use the index and document type reported by Beats for -indexing events into Elasticsearch: +For example, the following Logstash configuration file for version 7.x sets +Logstash to use the index reported by {beatname_uc} for indexing events +into Elasticsearch: +ifndef::apm-server[] [source,logstash] ------------------------------------------------------------------------------ @@ -774,11 +792,76 @@ output { of the `beat` metadata field and `%{[@metadata][version]}` sets the second part to the Beat's version. For example: +{beat_default_index_prefix}-{version}+. +endif::[] + +ifdef::apm-server[] +[source,logstash] +------ +input { + beats { + port => 5044 + } +} + +filter { + if [@metadata][beat] == "apm" { + if [processor][event] == "sourcemap" { + mutate { + add_field => { "[@metadata][index]" => "%{[@metadata][beat]}-%{[@metadata][version]}-%{[processor][event]}" } <1> + } + } else { + mutate { + add_field => { "[@metadata][index]" => "%{[@metadata][beat]}-%{[@metadata][version]}-%{[processor][event]}-%{+yyyy.MM.dd}" } <2> + } + } + } +} + +output { + elasticsearch { + hosts => ["http://localhost:9200"] + index => "%{[@metadata][index]}" + } +} +------ +<1> Creates a new field named `@metadata.index`. +`%{[@metadata][beat]}` sets the first part of the index name to the value of the `beat` metadata field. +`%{[@metadata][version]}` sets the second part to {beatname_uc}'s version. +`%{[processor][event]}` sets the final part based on the APM event type. +For example: +{beat_default_index_prefix}-{version}-sourcemap+. +<2> In addition to the above rules, this pattern appends a date to the `index` name so Logstash creates a new index each day. +For example: +{beat_default_index_prefix}-{version}-transaction-{sample_date_0}+. +endif::[] Events indexed into Elasticsearch with the Logstash configuration shown here -will be similar to events directly indexed by Beats into Elasticsearch. +will be similar to events directly indexed by {beatname_uc} into Elasticsearch. +ifndef::apm-server[] NOTE: If ILM is not being used, set `index` to `%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}` instead so Logstash creates an index per day, based on the `@timestamp` value of the events coming from Beats. +endif::[] + +ifdef::apm-server[] +==== Logstash and ILM + +When used with {apm-server-ref}/manual-ilm-setup.html[Index lifecycle management], Logstash does not need to create a new index each day. +Here's a sample Logstash configuration file that would accomplish this: + +[source,logstash] +------ +input { + beats { + port => 5044 + } +} + +output { + elasticsearch { + hosts => ["http://localhost:9200"] + index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{[processor][event]}" + } +} +------ +endif::[] ==== Compatibility @@ -796,7 +879,7 @@ You can specify the following options in the `logstash` section of the The enabled config is a boolean setting to enable or disable the output. If set to false, the output is disabled. -The default value is true. +The default value is `true`. [[hosts]] ===== `hosts` diff --git a/docs/copied-from-beats/shared-securing-beat.asciidoc b/docs/copied-from-beats/shared-securing-beat.asciidoc index 7677da1a876..d8f2b76383c 100644 --- a/docs/copied-from-beats/shared-securing-beat.asciidoc +++ b/docs/copied-from-beats/shared-securing-beat.asciidoc @@ -8,6 +8,9 @@ The following topics provide information about securing the {beatname_uc} process and securing communication between {beatname_uc} and other products in the Elastic stack: +ifdef::apm-server[] +* <> +endif::[] * <> ifndef::only-elasticsearch[] * <> @@ -21,10 +24,14 @@ ifdef::beat-specific-security[] include::{beat-specific-security}[] endif::[] -//sets block macro for https.asciidoc included in next section - -- +ifdef::apm-server[] +include::../secure-communication-agents.asciidoc[] +endif::[] + +//sets block macro for https.asciidoc included in next section + [[securing-communication-elasticsearch]] == Secure communication with Elasticsearch @@ -34,7 +41,7 @@ include::./https.asciidoc[] ifndef::only-elasticsearch[] [[configuring-ssl-logstash]] -== Secure communication with Logstash by using SSL +== Secure communication with Logstash include::./shared-ssl-logstash-config.asciidoc[] endif::only-elasticsearch[] diff --git a/docs/copied-from-beats/shared-ssl-config.asciidoc b/docs/copied-from-beats/shared-ssl-config.asciidoc index 0f4d649fb9d..c835e936dc2 100644 --- a/docs/copied-from-beats/shared-ssl-config.asciidoc +++ b/docs/copied-from-beats/shared-ssl-config.asciidoc @@ -1,18 +1,25 @@ [[configuration-ssl]] +ifndef::apm-server[] == Specify SSL settings +endif::apm-server[] +ifdef::apm-server[] +== SSL output settings +You can specify SSL options with any output that supports SSL, like {es}, {ls}, or Kafka. +endif::[] + +ifndef::apm-server[] You can specify SSL options when you configure: * <> that support SSL -ifeval::["{beatname_lc}"!="apm-server"] * the <> -endif::[] ifeval::["{beatname_lc}"=="heartbeat"] * <> that support SSL endif::[] ifeval::["{beatname_lc}"=="metricbeat"] * <> that define the host as an HTTP URL endif::[] +endif::[] Example output config with SSL enabled: diff --git a/docs/copied-from-beats/shared-ssl-logstash-config.asciidoc b/docs/copied-from-beats/shared-ssl-logstash-config.asciidoc index a7b28c7fe99..090b34cfc33 100644 --- a/docs/copied-from-beats/shared-ssl-logstash-config.asciidoc +++ b/docs/copied-from-beats/shared-ssl-logstash-config.asciidoc @@ -126,7 +126,7 @@ curl: (51) SSL: certificate verification failed (result: 5) See the <> for info about resolving this issue. [float] -=== Test the Beats to Logstash connection +=== Test the {beatname_uc} to Logstash connection If you have {beatname_uc} running as a service, first stop the service. Then test your setup by running {beatname_uc} in the foreground so you can quickly see any errors that occur: diff --git a/docs/copied-from-beats/template-config.asciidoc b/docs/copied-from-beats/template-config.asciidoc index ee58f8bf3f9..2a99f1fe618 100644 --- a/docs/copied-from-beats/template-config.asciidoc +++ b/docs/copied-from-beats/template-config.asciidoc @@ -71,7 +71,7 @@ setup.template.settings: NOTE: If you want to use {stack-ov}/xpack-ccr.html[{ccr}] to replicate {beatname_uc} indices to another cluster, you will need to add additional template settings to -{stack-ov}/ccr-requirements.html#ccr-overview-beats[enable soft deletes] on the +{ref}/ccr-requirements.html#ccr-overview-beats[enable soft deletes] on the underlying indices. *`setup.template.settings._source`*:: A dictionary of settings for the `_source` field. For the available settings, diff --git a/docs/data-ingestion.asciidoc b/docs/data-ingestion.asciidoc index 8f669fc35ad..eb941ff04e3 100644 --- a/docs/data-ingestion.asciidoc +++ b/docs/data-ingestion.asciidoc @@ -13,8 +13,6 @@ This section explains how to adapt data ingestion according to your needs. [[tune-apm-server]] == Tune APM Server -Tuning topics: - * <> * <> * <> @@ -87,15 +85,18 @@ Increasing the <> default value will help a [[tune-es]] == Tune Elasticsearch -Get insights about tuning the Elasticsearch ingestion rate, -especially with regards to +The Elasticsearch reference provides insight on tuning Elasticsearch. + +{ref}/tune-for-indexing-speed.html[Tune for indexing speed] provides information on: -* refresh interval -* disable swapping -* optimizing filesystem cache -* considerations regarding faster hardware -* setting the indexing buffer size +* Refresh interval +* Disabling swapping +* Optimizing filesystem cache +* Considerations regarding faster hardware +* Setting the indexing buffer size -directly at the {ref}/tune-for-indexing-speed.html[Elasticsearch reference]. +{ref}/tune-for-disk-usage.html[Tune for disk usage] provides information on: -You might also want to read up on how to {ref}/tune-for-disk-usage.html[tune for disk usage]. +* Disabling unneeded features +* Shard size +* Shrink index \ No newline at end of file diff --git a/docs/getting-started-apm-server.asciidoc b/docs/getting-started-apm-server.asciidoc new file mode 100644 index 00000000000..ccfad4dc659 --- /dev/null +++ b/docs/getting-started-apm-server.asciidoc @@ -0,0 +1,309 @@ +[[getting-started-apm-server]] +== Getting Started With APM Server + +// This section was copied from the APM Overview +// Switch to tagged regions when available +The easiest way to get started with APM Server is by using our +https://www.elastic.co/cloud/elasticsearch-service[hosted {es} Service] on +Elastic Cloud. The {es} Service is available on AWS, GCP, and Azure, +and automatically configures APM Server to work with {es} and {kib}. + +[float] +=== Hosted Elasticsearch Service + +Skip managing your own {es}, {kib}, and APM Server by using our +https://www.elastic.co/cloud/elasticsearch-service[hosted {es} Service] on +Elastic Cloud. + +image::images/apm-architecture-cloud.png[Install Elastic APM on cloud] + +https://www.elastic.co/cloud/elasticsearch-service/signup[Try out the {es} Service for free], +and then begin configuring your own user settings right in the Elasticsearch Service Console. +Any changes are automatically appended to the `apm-server.yml` configuration file for your instance. +A full list of configuration options is available in Elastic Cloud's +{cloud}/ec-manage-apm-settings.html[Add APM user settings]. + +[float] +=== Install and manage the stack yourself + +If you'd rather install the stack yourself, first see the https://www.elastic.co/support/matrix[Elastic Support Matrix] for information about supported operating systems and product compatibility. + +You'll need: + +* *{es}* for storing and indexing data. +* *{kib}* for visualizing with the APM UI. + +We recommend you use the same version of Elasticsearch, Kibana, and APM Server. + +image::images/apm-architecture-diy.png[Install Elastic APM yourself] + +See {stack-gs}/get-started-elastic-stack.html[Getting started with the {stack}] +for more information about installing these products. +After installing the {stack}, read the following topics to learn how to install, +configure, and start APM Server: + +* <> +* <> +* <> +* <> + +// ******************************************************* +// STEP 1 +// ******************************************************* + +[[installing]] +=== Step 1: Install + +NOTE: *Before you begin*: If you haven't installed the {stack}, do that now. +See {stack-gs}/get-started-elastic-stack.html[Getting started with the {stack}]. + +To download and install {beatname_uc}, use the commands below that work with your system. +If you use `apt` or `yum`, you can <> +to update to the newest version more easily. + +ifeval::["{release-state}"!="unreleased"] +See our https://www.elastic.co/downloads/apm[download page] +for other installation options, such as 32-bit images. +endif::[] + +[[deb]] +*deb:* + +ifeval::["{release-state}"=="unreleased"] + +Version {version} of APM Server has not yet been released. + +endif::[] + +ifeval::["{release-state}"!="unreleased"] + +["source","sh",subs="attributes"] +---------------------------------------------------------------------- +curl -L -O {downloads}/apm-server-{version}-amd64.deb +sudo dpkg -i apm-server-{version}-amd64.deb +---------------------------------------------------------------------- + +endif::[] + +[[rpm]] +*RPM:* + +ifeval::["{release-state}"=="unreleased"] + +Version {version} of APM Server has not yet been released. + +endif::[] + +ifeval::["{release-state}"!="unreleased"] + +["source","sh",subs="attributes"] +---------------------------------------------------------------------- +curl -L -O {downloads}/apm-server-{version}-x86_64.rpm +sudo rpm -vi apm-server-{version}-x86_64.rpm +---------------------------------------------------------------------- + +endif::[] + +[[linux]] +*Other Linux:* + +ifeval::["{release-state}"=="unreleased"] + +Version {version} of APM Server has not yet been released. + +endif::[] + +ifeval::["{release-state}"!="unreleased"] + +["source","sh",subs="attributes"] +------------------------------------------------ +curl -L -O {downloads}/apm-server-{version}-linux-x86_64.tar.gz +tar xzvf apm-server-{version}-linux-x86_64.tar.gz +------------------------------------------------ +endif::[] + +[[mac]] +*Mac:* + +ifeval::["{release-state}"=="unreleased"] + +Version {version} of APM Server has not yet been released. + +endif::[] + +ifeval::["{release-state}"!="unreleased"] + +["source","sh",subs="attributes"] +------------------------------------------------ +curl -L -O {downloads}/apm-server-{version}-darwin-x86_64.tar.gz +tar xzvf apm-server-{version}-darwin-x86_64.tar.gz +------------------------------------------------ + +endif::[] + +[[installing-on-windows]] +*Windows:* + +ifeval::["{release-state}"=="unreleased"] + +Version {version} of APM Server has not yet been released. + +endif::[] + +ifeval::["{release-state}"!="unreleased"] + +. Download the APM Server Windows zip file from the +https://www.elastic.co/downloads/apm/apm-server[downloads page]. + +. Extract the contents of the zip file into `C:\Program Files`. + +. Rename the `apm-server--windows` directory to `APM-Server`. + +. Open a PowerShell prompt as an Administrator (right-click the PowerShell icon and select *Run As Administrator*). +If you are running Windows XP, you may need to download and install PowerShell. + +. From the PowerShell prompt, run the following commands to install APM Server as a Windows service: ++ +[source,shell] +---------------------------------------------------------------------- +PS > cd 'C:\Program Files\APM-Server' +PS C:\Program Files\APM-Server> .\install-service-apm-server.ps1 +---------------------------------------------------------------------- + +NOTE: If script execution is disabled on your system, +you need to set the execution policy for the current session to allow the script to run. +For example: `PowerShell.exe -ExecutionPolicy UnRestricted -File .\install-service-apm-server.ps1`. + +endif::[] + +[[docker]] +*Docker:* + +See <> for deploying Docker containers. + +// ******************************************************* +// STEP 2 +// ******************************************************* + +[[apm-server-configuration]] +=== Step 2: Set up and configure + +[float] +==== Elastic Cloud + +If you're running APM Server in Elastic cloud, you can configure your own user settings right in the Elasticsearch Service Console. +Any changes are automatically appended to the `apm-server.yml` configuration file for your instance. + +Full details are available in the {cloud}/ec-manage-apm-settings.html[APM user settings] documentation. + +[float] +[[index-template-config]] +==== Self installation + +It is recommend that you run the `setup` command before starting {beatname_uc}. + +The `setup` command sets up the initial environment, including the Elasticsearch index template, +and ILM write alias. In Elasticsearch, {ref}/indices-templates.html[index templates] +are used to define field settings and mappings. + +IMPORTANT: The {kibana-ref}/xpack-apm.html[Kibana APM UI] relies on specific mappings. +Changing the default mapping or failing to setup the initial mapping may break the APM UI. + +The recommended index template file for {beatname_uc} is installed by the {beatname_uc} packages. +To see the template that comes bundled with {beatname_uc}, run `apm-server export template`. +If you accept the default configuration, +{beatname_uc} will load the template automatically after successfully connecting to Elasticsearch. +If a template already exists, it will not be overwritten unless you've explicitly configured {beatname_uc} to do so. +Running `apm-server -e setup template` will force a template overwrite. + +NOTE: Index templates cannot be applied retroactively. +They are only applied at index creation time. +Changing a template will have no impact on existing indices. +This means you need to ensure the correct index template is loaded from day one. + +Full details on the `setup` command are below: + +// This include attribute uses tagged regions to pull the **setup-command** instructions +// in from command-reference.asciidoc. There is also a tagged region for **attributes** +// as these are defined locally in the command reference file, but must also be accessed here. +include::./copied-from-beats/command-reference.asciidoc[tag=attributes] +include::./copied-from-beats/command-reference.asciidoc[tag=setup-command-tag] + +Once APM Server has been set up, you can edit the `apm-server.yml` configuration file to customize it to your needs. +The location of this file varies by platform, but the <> will help you locate it. +All available configuration options are outlined in +{apm-server-ref-v}/configuring-howto-apm-server.html[configuring APM Server]. + +// ******************************************************* +// STEP 3 +// ******************************************************* + +[[apm-server-starting]] +=== Step 3: Start + +In a production environment, you would put APM Server on its own machines, +similar to how you run Elasticsearch. +You _can_ run it on the same machines as Elasticsearch, but this is not recommended, +as the processes will be competing for resources. + +To start APM Server, run: + +[source,bash] +---------------------------------- +./apm-server -e +---------------------------------- + +NOTE: The `-e` <> enables logging to stderr and disables syslog/file output. +Remove this flag if you've enabled logging in the configuration file. +For linux systems, see <>. + +You should see APM Server start up. +It will try to connect to Elasticsearch on localhost port `9200` and expose an API to agents on port `8200`. +You can change the defaults in `apm-server.yml` or by supplying a different address on the command line: + +[source,bash] +---------------------------------- +./apm-server -e -E output.elasticsearch.hosts=ElasticsearchAddress:9200 -E apm-server.host=localhost:8200 +---------------------------------- + +[float] +[[running-deb-rpm]] +==== Debian Package / RPM + +The Debian package and RPM installations of APM Server create an `apm-server` user. +To start the APM Server in this case, run: + +[source,bash] +---------------------------------- +sudo -u apm-server apm-server [] +---------------------------------- + +By default, APM Server loads its configuration file from `/etc/apm-server/apm-server.yml`. +See the <<_deb_and_rpm,deb & rpm default paths>> for a full directory layout. + +// ******************************************************* +// STEP 4 +// ******************************************************* + +[[next-steps]] +=== Step 4: Next steps + +// Use a tagged region to pull APM Agent information from the APM Overview +If you haven't already, you can now install APM Agents in your services! + +* {apm-go-ref-v}/introduction.html[Go agent] +* {apm-java-ref-v}/intro.html[Java agent] +* {apm-dotnet-ref-v}/intro.html[.NET agent] +* {apm-node-ref-v}/intro.html[Node.js agent] +* {apm-py-ref-v}/getting-started.html[Python agent] +* {apm-ruby-ref-v}/introduction.html[Ruby agent] +* {apm-rum-ref-v}/intro.html[JavaScript Real User Monitoring (RUM) agent] + +Once you have at least one Agent sending data to APM Server, +you can start visualizing your data in the {kibana-ref}/xpack-apm.html[Kibana APM UI]. + +// Shared APM & YUM +include::./copied-from-beats/repositories.asciidoc[] + +// Shared docker +include::./copied-from-beats/shared-docker.asciidoc[] \ No newline at end of file diff --git a/docs/guide/apm-doc-directory.asciidoc b/docs/guide/apm-doc-directory.asciidoc index e602dfff393..8f93e3f9bda 100644 --- a/docs/guide/apm-doc-directory.asciidoc +++ b/docs/guide/apm-doc-directory.asciidoc @@ -28,7 +28,7 @@ Each agent has its own documentation: === APM Server APM Server is an open source application that receives performance data from your APM agents. -It's a {apm-server-ref-v}/why-separate-component.html[separate component by design], +It's a {apm-server-ref-v}/overview.html#why-separate-component[separate component by design], which helps keep the agents light, prevents certain security risks, and improves compatibility across the Elastic Stack. After the APM Server has validated and processed events from the APM agents, diff --git a/docs/guide/install-and-run.asciidoc b/docs/guide/install-and-run.asciidoc index 05df4a5c6c8..eba2be17c15 100644 --- a/docs/guide/install-and-run.asciidoc +++ b/docs/guide/install-and-run.asciidoc @@ -3,7 +3,7 @@ The easiest way to get started with Elastic APM is by using our https://www.elastic.co/cloud/elasticsearch-service[hosted {es} Service] on -Elastic Cloud. The {es} Service is available on both AWS and GCP, +Elastic Cloud. The {es} Service is available on AWS, GCP, and Azure, and automatically configures APM Server to work with {es} and {kib}. [float] @@ -20,7 +20,7 @@ then jump straight to <>. [float] [[before-installation]] -=== Install the stack yourself +=== Install and manage the stack yourself image::images/apm-architecture-diy.png[Install Elastic APM yourself] diff --git a/docs/ilm-setup.asciidoc b/docs/ilm-setup.asciidoc index 6c80e267f4b..a652a6bbdda 100644 --- a/docs/ilm-setup.asciidoc +++ b/docs/ilm-setup.asciidoc @@ -164,7 +164,7 @@ Repeat the previous steps for each index that will be using ILM. . *Modify APM Server's configuration.* + -Finally, modify the default index configuration in <>. +Finally, modify the default index configuration in <>. Trim off the date template from each index you are setting up ILM for, so that APM Server is always writing events to the same place. The name of your modified index configuration must match the `is_write_index` alias created previously diff --git a/docs/images/apm-architecture-cloud.png b/docs/images/apm-architecture-cloud.png new file mode 100644 index 00000000000..6bc7001fb9f Binary files /dev/null and b/docs/images/apm-architecture-cloud.png differ diff --git a/docs/images/apm-architecture-diy.png b/docs/images/apm-architecture-diy.png new file mode 100644 index 00000000000..d4e96466081 Binary files /dev/null and b/docs/images/apm-architecture-diy.png differ diff --git a/docs/index.asciidoc b/docs/index.asciidoc index d7d903f1931..f2d4b64cfeb 100644 --- a/docs/index.asciidoc +++ b/docs/index.asciidoc @@ -33,6 +33,8 @@ include::{asciidoc-dir}/../../shared/attributes.asciidoc[] :win_os: :linux_os: +:downloads: https://artifacts.elastic.co/downloads/apm-server + ifdef::env-github[] NOTE: For the best reading experience, please view this documentation at https://www.elastic.co/guide/en/apm/server[elastic.co] @@ -43,12 +45,12 @@ endif::[] include::./overview.asciidoc[] -include::./installing.asciidoc[] - -include::./upgrading.asciidoc[] +include::./getting-started-apm-server.asciidoc[] include::./setting-up-and-running.asciidoc[] +include::./upgrading.asciidoc[] + :beat-specific-output-config: ./../configuring-output-after.asciidoc include::./configuring.asciidoc[] diff --git a/docs/installing-on-windows.asciidoc b/docs/installing-on-windows.asciidoc deleted file mode 100644 index 420d3e88ae8..00000000000 --- a/docs/installing-on-windows.asciidoc +++ /dev/null @@ -1,32 +0,0 @@ -[[installing-on-windows]] -=== Installing {beatname_uc} on Windows - -ifeval::["{release-state}"=="unreleased"] - -Version {version} of {beatname_uc} has not yet been released. - -endif::[] - -ifeval::["{release-state}"!="unreleased"] - -. Download the APM Server Windows zip file from the -https://www.elastic.co/downloads/apm/apm-server[downloads page]. - -. Extract the contents of the zip file into `C:\Program Files`. - -. Rename the `apm-server--windows` directory to `APM-Server`. - -. Open a PowerShell prompt as an Administrator (right-click the PowerShell icon and select *Run As Administrator*). If you are running Windows XP, you may need to download and install PowerShell. - -. From the PowerShell prompt, run the following commands to install APM Server as a -Windows service: -+ -[source,shell] ----------------------------------------------------------------------- -PS > cd 'C:\Program Files\APM-Server' -PS C:\Program Files\APM-Server> .\install-service-apm-server.ps1 ----------------------------------------------------------------------- - -NOTE: If script execution is disabled on your system, you need to set the execution policy for the current session to allow the script to run. For example: `PowerShell.exe -ExecutionPolicy UnRestricted -File .\install-service-apm-server.ps1`. - -endif::[] \ No newline at end of file diff --git a/docs/overview.asciidoc b/docs/overview.asciidoc index 91fea1f104d..595c7cb8ab9 100644 --- a/docs/overview.asciidoc +++ b/docs/overview.asciidoc @@ -10,7 +10,7 @@ The APM Server works in conjunction with {apm-agents-ref}/index.html[APM agents] NOTE: APM Server is built with the {beats-ref}[Beats] framework and leverages its functionality. - +[float] [[why-separate-component]] === Why is APM Server a separate component? diff --git a/docs/rum.asciidoc b/docs/rum.asciidoc index c6cd5689cf9..3bc43d01fce 100644 --- a/docs/rum.asciidoc +++ b/docs/rum.asciidoc @@ -28,9 +28,9 @@ Source maps are cached in memory for as long as the <> looks like in Elasticsearch. +See what an <> looks like in Elasticsearch. diff --git a/docs/secure-communication-agents.asciidoc b/docs/secure-communication-agents.asciidoc new file mode 100644 index 00000000000..e5f7674b57d --- /dev/null +++ b/docs/secure-communication-agents.asciidoc @@ -0,0 +1,89 @@ +[[secure-communication-agents]] +== Secure communication with APM Agents + +To secure the communication between APM Agents and the APM Server: + +. <> +. <> +. <> + +[[secret-token]] +[float] +=== Secret token + +You can configure a secret token to authorize requests to the APM Server. +This ensures that only your agents are able to send data to your APM servers. +Both the agents and the APM servers have to be configured with the same secret token, +and secret tokens only provide security when used in combination with SSL/TLS. + +**APM Server configuration** + +Here's how you set the secret token in APM Server: + +[source,yaml] +---- +apm-server.secret_token: +---- + +We recommend saving the token in the APM Server <>. + +IMPORTANT: Secret tokens are not applicable for the RUM Agent, +as there is no way to prevent them from being publicly exposed. + +**Agent specific configuration** + +Each Agent has a configuration for setting the value of the secret token: + +* *Go Agent*: {apm-go-ref}/configuration.html#config-secret-token[`ELASTIC_APM_SECRET_TOKEN`] +* *Java Agent*: {apm-java-ref}/config-reporter.html#config-secret-token[`secret_token`] +* *.NET Agent*: {apm-dotnet-ref}/config-reporter.html#config-secret-token[`ELASTIC_APM_SECRET_TOKEN`] +* *Node.js Agent*: {apm-node-ref}/configuration.html#secret-token[`Secret Token`] +* *Python Agent*: {apm-py-ref}/configuration.html#config-secret-token[`secret_token`] +* *Ruby Agent*: {apm-ruby-ref}/configuration.html#config-secret-token[`secret_token`] + +[[ssl-setup]] +[float] +=== SSL/TLS communication in APM Server + +To enable SSL/TLS, you need to enable SSL and provide both a private key and a certificate +issued by a certificate authority (CA). +You can then specify the path to those files in your configuration properties. +This will make APM Server serve HTTPS requests instead of HTTP. + +Here's a basic APM Server SSL config with secure communication enabled: + +[source,yaml] +---- +apm-server.ssl.enabled: true +apm-server.ssl.key: "/etc/pki/key.pem" +apm-server.ssl.certificate: "/etc/pki/client.pem" +---- + +A full list of configuration options is available in <>. + +[[https-in-agents]] +[float] +=== HTTPS communication in APM Agents + +To enable secure communication in your Agents, you need to update the configured server URL to use `HTTPS` instead of `HTTP`. + +* *Go Agent*: {apm-go-ref}/configuration.html#config-server-url[`ELASTIC_APM_SERVER_URL`] +* *Java Agent*: {apm-java-ref}/config-reporter.html#config-server-urls[`server_urls`] +* *.NET Agent*: {apm-dotnet-ref}/config-reporter.html#config-server-urls[`ServerUrls`] +* *Node.js Agent*: {apm-node-ref}/configuration.html#server-url[`serverUrl`] +* *Python Agent*: {apm-py-ref}/[`server_url`] +* *Ruby Agent*: {apm-ruby-ref}/configuration.html#config-server-url[`server_url`] + +Some Agents also allow you to specify a custom certificate authority for connecting to APM Server. + +* *Go Agent*: {apm-go-ref}/configuration.html#config-server-cert[`ELASTIC_APM_SERVER_CERT`] +// * *.NET Agent*: {apm-dotnet-ref}/ +* *Python Agent*: {apm-py-ref}/configuration.html#config-server-cert[`ELASTIC_APM_SERVER_CERT`] +* *Ruby Agent*: {apm-ruby-ref}/configuration.html#config-ssl-ca-cert[`server_ca_certedit`] + +Most Agents that don't allow you specify a custom certificate will allow you to +disable verification of the SSL certificate. +This ensures encryption, but does not verify that you are sending data to the correct APM Server. + +* *Java Agent*: {apm-java-ref}/config-reporter.html#config-verify-server-cert[`verify_server_cert`] +* *Node.js Agent*: {apm-node-ref}/configuration.html#validate-server-cert[`verifyServerCert`] diff --git a/docs/security.asciidoc b/docs/security.asciidoc index 15bf075d3e3..57b6a8767a3 100644 --- a/docs/security.asciidoc +++ b/docs/security.asciidoc @@ -1,30 +1,9 @@ -[[security-overview]] +A reference of all available <> is also available. + [float] -=== Security Overview +[[security-overview]] +== Security Overview -APM Server exposes an HTTP endpoint and as with anything that opens ports on your servers, +APM Server exposes an HTTP endpoint, and as with anything that opens ports on your servers, you should be careful about who can connect to it. Firewall rules are recommended to ensure only authorized systems can connect. - -[[secret-token]] -[float] -==== Secret token - -You can configure a secret token to authorize requests to the APM Server, -and ensure that only your agents can send data to your APM servers. -Both the agents and the APM servers have to be configured with the same secret token. - -NOTE: Secret tokens provide security only when used in combination with SSL/TLS. -Secret tokens are not applicable for RUM, as they would be publicly exposed. - -[[ssl-setup]] -[float] -==== SSL/TLS setup - -To enable SSL/TLS you need a private key and a certificate issued by a certification authority (CA). -You can then specify the path to those files in your configuration properties: -`apm-server.ssl.key` and `apm-server.ssl.certificate` respectively. -This will make the APM Server serve HTTPS requests instead of HTTP. -Don't forget, you also need to enable SSL in the agent. -For agent specific details on enabling SSL/TLS, -please see the {apm-agents-ref}/index.html[agent documentation]. \ No newline at end of file diff --git a/docs/setting-up-and-running.asciidoc b/docs/setting-up-and-running.asciidoc index ed7732e526f..6634d087da4 100644 --- a/docs/setting-up-and-running.asciidoc +++ b/docs/setting-up-and-running.asciidoc @@ -1,115 +1,24 @@ [[setting-up-and-running]] -== Setting up and running APM Server +== Setting up APM Server -In a production environment, -you would put APM Server on its own machines, -similar to how you run Elasticsearch. -You _can_ run it on the same machines as Elasticsearch, -but this is not recommended, -as the processes will be competing for resources. +Before reading this section, see the <> +for basic installation and running instructions. -// This needs to be added to allow a link from monitoring-metricbeat.asciidoc to work -// Update this when redoing the installation section -[id="{beatname_lc}-starting"] -To start APM Server, run: +This section includes additional information on how to set up and run APM Server, including: -[source,bash] ----------------------------------- -./apm-server -e ----------------------------------- +* <> +* <> +* <> +* <> +* <> -NOTE: The `-e` <> enables logging to stderr and disables syslog/file output. -Remove this flag if you've enabled logging in the configuration file. -For linux systems, see <>. - -You should see APM Server start up. -It will try to connect to Elasticsearch on localhost port 9200 and expose an API to agents on port 8200. -You can change the defaults by supplying a different address on the command line: - -[source,bash] ----------------------------------- -./apm-server -e -E output.elasticsearch.hosts=ElasticsearchAddress:9200 -E apm-server.host=localhost:8200 ----------------------------------- - -[float] -[[running-deb-rpm]] -=== Debian Package / RPM - -The Debian package and RPM installations of APM Server create an `apm-server` user. -To start the APM Server in this case, run: - -[source,bash] ----------------------------------- -sudo -u apm-server apm-server [] ----------------------------------- - -By default, APM Server loads its configuration file from `/etc/apm-server/apm-server.yml`. -See the <<_deb_and_rpm,deb & rpm default paths>> for a full directory layout. - -[float] -[[running-brew]] -=== Brew - -To start APM Server, run: - -["source","sh"] ------ -apm-server -e ------ - -To have launchd start APM Server and restart it at login, run: - -["source","sh"] ------ -brew services start elastic/tap/apm-server-full ------ - -[[apm-server-configuration]] -=== Configuration file -To configure APM Server, you can also update the `apm-server.yml` configuration file. - -For rpm and deb, -you’ll find the configuration file at +/etc/{beatname_lc}/{beatname_lc}.yml+. -For mac and win, look in the archive that you extracted. - -See {beats-ref}/config-file-format.html[Config File Format] in _Beats Platform Reference_ for more about the structure of the config file. - -[source,yaml] ----------------------------------- -apm-server: - host: localhost:8200 - -output: - elasticsearch: - hosts: ElasticsearchAddress:9200 ----------------------------------- - -The output section, like in other Beats, contains settings describing how and where to store the data. - -NOTE: If you are using an X-Pack secured version of Elastic Stack, -you need to specify credentials in the config file before you run the commands that set up and start APM Server. -For example: - -[source,yaml] ----- -output.elasticsearch: - hosts: ["ElasticsearchAddress:9200"] - username: "elastic" - password: "elastic" ----- - -See <> for more configuration options. - - -include::./high-availability.asciidoc[] +include::./copied-from-beats/shared-directory-layout.asciidoc[] include::./copied-from-beats/keystore.asciidoc[] include::./copied-from-beats/command-reference.asciidoc[] -include::./copied-from-beats/shared-directory-layout.asciidoc[] - -include::./copied-from-beats/shared-docker.asciidoc[] +include::./high-availability.asciidoc[] include::./copied-from-beats/shared-systemd.asciidoc[] diff --git a/docs/ssl-input-settings.asciidoc b/docs/ssl-input-settings.asciidoc new file mode 100644 index 00000000000..272ccc45318 --- /dev/null +++ b/docs/ssl-input-settings.asciidoc @@ -0,0 +1,75 @@ +[[agent-server-ssl]] +=== SSL input settings + +You can specify the following options in the `apm-server.ssl` section of the +{beatname_lc}.yml+ config file. +They apply to SSL/TLS communication between the APM Server and APM Agents. + +[float] +==== `enabled` + +The `enabled` setting can be used to enable the ssl configuration by setting +it to `true`. The default value is `false`. + +[float] +==== `certificate` + +The path to the file containing the certificate for Server authentication. +Required if `apm-server.enabled` is `true`. + +[float] +==== `key` + +The path to the file containing the Server certificate key. +Required if `apm-server.enabled` is `true`. + +[float] +==== `certificate_authorities` + +The list of root certificates for verifying client certificates. +If `certificate_authorities` is empty or not set, the trusted certificate authorities of the host system are used. + +[float] +==== `key_passphrase` + +The passphrase used to decrypt an encrypted key stored in the configured `key` file. +We recommend saving the `key_passphrase` in the APM Server <>. + +[float] +==== `supported_protocols` + +This setting is a list of allowed protocol versions: +`SSLv3`, `TLSv1.0`, `TLSv1.1` and `TLSv1.2`. We do not recommend using `SSLv3` or `TLSv1.0`. +The default value is `[TLSv1.1, TLSv1.2]`. + +[float] +==== `cipher_suites` + +The list of cipher suites to use. The first entry has the highest priority. +If this option is omitted, the Go crypto library's default +suites are used (recommended). + +[float] +==== `curve_types` + +The list of curve types for ECDHE (Elliptic Curve Diffie-Hellman ephemeral key exchange). + +[float] +==== `client_authentication` + +This configures what types of client authentication are supported. The valid options +are `none`, `optional`, and `required`. The default is `optional`. +If `certificate_authorities` has been specified, this setting will automatically change to `required`. + +* `none` - Disables client authentication. +* `optional` - When a client certificate is given, the server will verify it. +* `required` - Requires clients to provide a valid certificate. + +[float] +==== `verification_mode` + +This option controls whether the client verifies server certificates and host +names. Valid values are `none` and `full`. If `none` is used, +SSL-based connections are susceptible to man-in-the-middle attacks. Use this +option for testing only. + +The default is `full`. diff --git a/docs/storage-management.asciidoc b/docs/storage-management.asciidoc index cc0a30d2203..abb888e2cdc 100644 --- a/docs/storage-management.asciidoc +++ b/docs/storage-management.asciidoc @@ -151,9 +151,17 @@ deleting data collected for specific services or customers, or deleting specific indices. Depending on your use case, -you can delete data periodically with {curator-ref-current}[Curator], -by using the {ref}/docs-delete-by-query.html[Delete By Query API], -or by using the {kibana-ref}/managing-indices.html[Kibana Index Management UI]. +you can delete data periodically with <>, +{curator-ref-current}[Curator], the {ref}/docs-delete-by-query.html[Delete By Query API], +or in the {kibana-ref}/managing-indices.html[Kibana Index Management UI]. + +[float] +[[delete-data-ilm]] +==== Delete data with ILM + +Index Lifecycle management (ILM) enables you to automate how you want to manage your indices over time. +You can base actions on factors such as shard size and performance requirements. +See <> to learn more. [float] [[delete-data-periodically]] diff --git a/docs/troubleshooting.asciidoc b/docs/troubleshooting.asciidoc index 1a51f9babb6..325d557c0f8 100644 --- a/docs/troubleshooting.asciidoc +++ b/docs/troubleshooting.asciidoc @@ -11,24 +11,27 @@ If you have issues installing or running APM Server, read the following tips: * <> -* <> * <> +* <> -//sets block macro for getting-help.asciidoc included in next section +Other sections in the documentation may also be helpful: + +* <> +* <> +* <> +* <> +* {apm-overview-ref-v}/agent-server-compatibility.html[Agent/Server compatibility matrix] -- include::common-problems.asciidoc[] - -[[getting-help]] -== Get help - -include::copied-from-beats/getting-help.asciidoc[] - -//sets block macro for debugging.asciidoc included in next section - [[enable-apm-server-debugging]] == Debug include::copied-from-beats/debugging.asciidoc[] + +[[getting-help]] +== Get help + +include::copied-from-beats/getting-help.asciidoc[] \ No newline at end of file diff --git a/docs/version.asciidoc b/docs/version.asciidoc index 8049b303f3c..ca281b06ae7 100644 --- a/docs/version.asciidoc +++ b/docs/version.asciidoc @@ -21,7 +21,7 @@ :node-branch: 2.x :py-branch: 5.x :ruby-branch: 2.x -:dotnet-branch: current +:dotnet-branch: 1.x // Agent links :apm-py-ref-v: https://www.elastic.co/guide/en/apm/agent/python/{py-branch}