diff --git a/docs/static/docker.asciidoc b/docs/static/docker.asciidoc index a05f947a605..36690263245 100644 --- a/docs/static/docker.asciidoc +++ b/docs/static/docker.asciidoc @@ -153,7 +153,7 @@ Some example translations are shown here: **Environment Variable**:: **Logstash Setting** `PIPELINE_WORKERS`:: `pipeline.workers` `LOG_LEVEL`:: `log.level` -`XPACK_MONITORING_ENABLED`:: `xpack.monitoring.enabled` +`MONITORING_ENABLED`:: `monitoring.enabled` In general, any setting listed in the <> can be configured with this technique. @@ -170,9 +170,9 @@ images: [horizontal] `http.host`:: `0.0.0.0` -`xpack.monitoring.elasticsearch.hosts`:: `http://elasticsearch:9200` +`monitoring.elasticsearch.hosts`:: `http://elasticsearch:9200` -NOTE: The setting `xpack.monitoring.elasticsearch.hosts` is not +NOTE: The setting `monitoring.elasticsearch.hosts` is not defined in the `-oss` image. These settings are defined in the default `logstash.yml`. They can be overridden diff --git a/docs/static/monitoring/collectors.asciidoc b/docs/static/monitoring/collectors.asciidoc index 30a373f24bb..2dc8876a685 100644 --- a/docs/static/monitoring/collectors.asciidoc +++ b/docs/static/monitoring/collectors.asciidoc @@ -40,8 +40,8 @@ data collection. For more information about the configuration options for the collectors, see <>. WARNING: Unlike {monitoring} for {es} and {kib}, there is no -`xpack.monitoring.collection.enabled` setting on Logstash. You must use the -`xpack.monitoring.enabled` setting to enable and disable data collection. +`pack.monitoring.collection.enabled` setting on Logstash. You must use the +`monitoring.enabled` setting to enable and disable data collection. If gaps exist in the monitoring charts in {kib}, it is typically because either a collector failed or the monitoring cluster did not receive the data (for diff --git a/docs/static/monitoring/monitoring-internal-legacy.asciidoc b/docs/static/monitoring/monitoring-internal-legacy.asciidoc new file mode 100644 index 00000000000..3dcf1bc2d59 --- /dev/null +++ b/docs/static/monitoring/monitoring-internal-legacy.asciidoc @@ -0,0 +1,17 @@ +[role="xpack"] +[[monitoring-internal-collection-legacy]] +=== Collect {ls} monitoring data using internal legacy collectors +++++ +Legacy internal collection (deprecated) +++++ + +deprecated[7.7.0] + +Starting in {ls} version 7.7, the legacy settings for internal collection are deprecated and +will be removed in version 8.0.0. Instead of sending monitoring data to {es} exporter, +it’s recommended that you use the <> +to route monitoring data directly to your monitoring indices. + +Or as an alternative to internal collection, use <>. + + diff --git a/docs/static/monitoring/monitoring-internal.asciidoc b/docs/static/monitoring/monitoring-internal.asciidoc index c58fffadbc1..0ae32ca2c9e 100644 --- a/docs/static/monitoring/monitoring-internal.asciidoc +++ b/docs/static/monitoring/monitoring-internal.asciidoc @@ -53,12 +53,12 @@ monitoring cluster will show the Logstash metrics under the _monitoring_ cluster -- -. Verify that the `xpack.monitoring.collection.enabled` setting is `true` on the +. Verify that the `monitoring.collection.enabled` setting is `true` on the production cluster. If that setting is `false`, the collection of monitoring data is disabled in {es} and data is ignored from all other sources. . Configure your Logstash nodes to send metrics by setting the -`xpack.monitoring.elasticsearch.hosts` in `logstash.yml`. If {security} is +`monitoring.elasticsearch.hosts` in `logstash.yml`. If {security} is enabled, you also need to specify the credentials for the {ref}/built-in-users.html[built-in `logstash_system` user]. For more information about these settings, see <>. @@ -66,9 +66,9 @@ information about these settings, see <>. -- [source,yaml] -------------------------------------------------- -xpack.monitoring.elasticsearch.hosts: ["http://es-prod-node-1:9200", "http://es-prod-node-2:9200"] <1> -xpack.monitoring.elasticsearch.username: "logstash_system" <2> -xpack.monitoring.elasticsearch.password: "changeme" +monitoring.elasticsearch.hosts: ["http://es-prod-node-1:9200", "http://es-prod-node-2:9200"] <1> +monitoring.elasticsearch.username: "logstash_system" <2> +monitoring.elasticsearch.password: "changeme" -------------------------------------------------- <1> If SSL/TLS is enabled on the production cluster, you must @@ -91,7 +91,7 @@ can specify the location of the PEM encoded certificate with the [source,yaml] -------------------------------------------------- -xpack.monitoring.elasticsearch.ssl.certificate_authority: /path/to/ca.crt +monitoring.elasticsearch.ssl.certificate_authority: /path/to/ca.crt -------------------------------------------------- Alternatively, you can configure trusted certificates using a truststore @@ -99,8 +99,8 @@ Alternatively, you can configure trusted certificates using a truststore [source,yaml] -------------------------------------------------- -xpack.monitoring.elasticsearch.ssl.truststore.path: /path/to/file -xpack.monitoring.elasticsearch.ssl.truststore.password: password +monitoring.elasticsearch.ssl.truststore.path: /path/to/file +monitoring.elasticsearch.ssl.truststore.password: password -------------------------------------------------- Also, optionally, you can set up client certificate using a keystore @@ -108,8 +108,8 @@ Also, optionally, you can set up client certificate using a keystore [source,yaml] -------------------------------------------------- -xpack.monitoring.elasticsearch.ssl.keystore.path: /path/to/file -xpack.monitoring.elasticsearch.ssl.keystore.password: password +monitoring.elasticsearch.ssl.keystore.path: /path/to/file +monitoring.elasticsearch.ssl.keystore.password: password -------------------------------------------------- Set sniffing to `true` to enable discovery of other nodes of the {es} cluster. @@ -117,7 +117,7 @@ It defaults to `false`. [source,yaml] -------------------------------------------------- -xpack.monitoring.elasticsearch.sniffing: false +monitoring.elasticsearch.sniffing: false -------------------------------------------------- -- diff --git a/docs/static/monitoring/monitoring-mb.asciidoc b/docs/static/monitoring/monitoring-mb.asciidoc index d267211b115..444255aad84 100644 --- a/docs/static/monitoring/monitoring-mb.asciidoc +++ b/docs/static/monitoring/monitoring-mb.asciidoc @@ -29,7 +29,7 @@ commented out: [source,yaml] ---------------------------------- -xpack.monitoring.enabled: false +monitoring.enabled: false ---------------------------------- Remove the `#` at the beginning of the line to enable the setting. diff --git a/docs/static/monitoring/monitoring-output.asciidoc b/docs/static/monitoring/monitoring-output.asciidoc index 08ce60ecf26..6a8cb04d5f2 100644 --- a/docs/static/monitoring/monitoring-output.asciidoc +++ b/docs/static/monitoring/monitoring-output.asciidoc @@ -7,7 +7,7 @@ Like all Logstash pipelines, the purpose of the dedicated monitoring pipeline is to send events to outputs. In the case of {monitoring} for Logstash, the output is always an `elasticsearch` output. However, unlike ordinary Logstash pipelines, the output is configured within the `logstash.yml` settings file via the -`xpack.monitoring.elasticsearch.*` settings. +`monitoring.elasticsearch.*` settings. Other than its unique manner of configuration, this `elasticsearch` output behaves like all `elasticsearch` outputs, including its ability to pause data @@ -25,7 +25,7 @@ the following default configuration is used: [source,yaml] --------------------------------------------------- -xpack.monitoring.elasticsearch.hosts: [ "http://localhost:9200" ] +monitoring.elasticsearch.hosts: [ "http://localhost:9200" ] --------------------------------------------------- All data produced by {monitoring} for Logstash is indexed in the monitoring @@ -38,7 +38,7 @@ necessary to properly configure Logstash. For more information, see IMPORTANT: When discussing security relative to the `elasticsearch` output, it is critical to remember that all users are managed on the production cluster, -which is identified in the `xpack.monitoring.elasticsearch.hosts` setting. +which is identified in the `monitoring.elasticsearch.hosts` setting. This is particularly important to remember when you move from development environments to production environments, where you often have dedicated monitoring clusters. diff --git a/docs/static/monitoring/monitoring-overview.asciidoc b/docs/static/monitoring/monitoring-overview.asciidoc index 1764aef29c6..2d997603cc4 100644 --- a/docs/static/monitoring/monitoring-overview.asciidoc +++ b/docs/static/monitoring/monitoring-overview.asciidoc @@ -17,11 +17,13 @@ Make sure monitoring is enabled on your {es} cluster. Then configure *one* of these methods to collect {ls} metrics: * <> -* <> +* <> +* <> include::monitoring-mb.asciidoc[] include::monitoring-internal.asciidoc[] +include::monitoring-internal-legacy.asciidoc[] include::monitoring-ui.asciidoc[] include::pipeline-viewer.asciidoc[] include::troubleshooting.asciidoc[] diff --git a/docs/static/security/logstash.asciidoc b/docs/static/security/logstash.asciidoc index a7729e24d87..e6d56097891 100644 --- a/docs/static/security/logstash.asciidoc +++ b/docs/static/security/logstash.asciidoc @@ -226,8 +226,8 @@ Then configure the user and password in the `logstash.yml` configuration file: [source,yaml] ---------------------------------------------------------- -xpack.monitoring.elasticsearch.username: logstash_system -xpack.monitoring.elasticsearch.password: t0p.s3cr3t +monitoring.elasticsearch.username: logstash_system +monitoring.elasticsearch.password: t0p.s3cr3t ---------------------------------------------------------- If you initially installed an older version of {xpack}, and then upgraded, the diff --git a/docs/static/settings/monitoring-settings.asciidoc b/docs/static/settings/monitoring-settings.asciidoc index 4309ad89df4..db83d07c596 100644 --- a/docs/static/settings/monitoring-settings.asciidoc +++ b/docs/static/settings/monitoring-settings.asciidoc @@ -5,7 +5,7 @@ Monitoring Settings ++++ -You can set the following `xpack.monitoring` settings in `logstash.yml` to +You can set the following `monitoring` settings in `logstash.yml` to control how monitoring data is collected from your Logstash nodes. However, the defaults work best in most circumstances. For more information about configuring Logstash, see <>. @@ -14,11 +14,14 @@ Logstash, see <>. [[monitoring-general-settings]] ===== General monitoring settings -`xpack.monitoring.enabled`:: +NOTE: In {ls} 7.7 the {xpack} is still able to use the deprecated `xpack.monitoring.*` settings, but if it +find that the settings is mixed with the new ones, it exits with an error. + +`monitoring.enabled`:: Monitoring is disabled by default. Set to `true` to enable {xpack} monitoring. -`xpack.monitoring.elasticsearch.hosts`:: +`monitoring.elasticsearch.hosts`:: The {es} instances that you want to ship your Logstash metrics to. This might be the same {es} instance specified in the `outputs` section in your Logstash @@ -28,13 +31,13 @@ Logstash metrics must be routed through your production cluster. You can specify a single host as a string, or specify multiple hosts as an array. Defaults to `http://localhost:9200`. -NOTE: If your Elasticsearch cluster is configured with dedicated master-eliglble +NOTE: If your Elasticsearch cluster is configured with dedicated master-eligible nodes, Logstash metrics should _not_ be routed to these nodes, as doing so can create resource contention and impact the stability of the Elasticsearch cluster. Therefore, do not include such nodes in -`xpack.monitoring.elasticsearch.hosts`. +`monitoring.elasticsearch.hosts`. -`xpack.monitoring.elasticsearch.username` and `xpack.monitoring.elasticsearch.password`:: +`monitoring.elasticsearch.username` and `monitoring.elasticsearch.password`:: If your {es} is protected with basic authentication, these settings provide the username and password that the Logstash instance uses to authenticate for @@ -44,12 +47,17 @@ shipping monitoring data. [[monitoring-collection-settings]] ===== Monitoring collection settings -`xpack.monitoring.collection.interval`:: +`monitoring.collection.interval`:: Controls how often data samples are collected and shipped on the Logstash side. Defaults to `10s`. If you modify the collection interval, set the -`xpack.monitoring.min_interval_seconds` option in `kibana.yml` to the same value. +`monitoring.min_interval_seconds` option in `kibana.yml` to the same value. + +`monitoring.cluster_uuid`:: +By default {ls} identify the `cluster uuid` bringing it from the first `ouput-elasticsearch` plugin defined +in the pipelines. +If it's not found, then uses this `cluster_uuid` setting to tag the data shipped to the monitoring cluster. [[monitoring-ssl-settings]] ===== {monitoring} TLS/SSL settings @@ -58,44 +66,44 @@ You can configure the following Transport Layer Security (TLS) or Secure Sockets Layer (SSL) settings. For more information, see <>. -`xpack.monitoring.elasticsearch.ssl.certificate_authority`:: +`monitoring.elasticsearch.ssl.certificate_authority`:: Optional setting that enables you to specify a path to the `.pem` file for the certificate authority for your {es} instance. -`xpack.monitoring.elasticsearch.ssl.truststore.path`:: +`monitoring.elasticsearch.ssl.truststore.path`:: Optional settings that provide the paths to the Java keystore (JKS) to validate the server’s certificate. -`xpack.monitoring.elasticsearch.ssl.truststore.password`:: +`monitoring.elasticsearch.ssl.truststore.password`:: Optional settings that provide the password to the truststore. -`xpack.monitoring.elasticsearch.ssl.keystore.path`:: +`monitoring.elasticsearch.ssl.keystore.path`:: Optional settings that provide the paths to the Java keystore (JKS) to validate the client’s certificate. -`xpack.monitoring.elasticsearch.ssl.keystore.password`:: +`monitoring.elasticsearch.ssl.keystore.password`:: Optional settings that provide the password to the keystore. [[monitoring-additional-settings]] ===== Additional settings -`xpack.monitoring.elasticsearch.cloud_id`:: +`monitoring.elasticsearch.cloud_id`:: If you're using {es} in {ecloud}, you should specify the identifier here. -This setting is an alternative to `xpack.monitoring.elasticsearch.hosts`. -If `cloud_id` is configured, `xpack.monitoring.elasticsearch.hosts` should not be used. +This setting is an alternative to `monitoring.elasticsearch.hosts`. +If `cloud_id` is configured, `monitoring.elasticsearch.hosts` should not be used. The {es} instances that you want to ship your Logstash metrics to. This might be the same {es} instance specified in the `outputs` section in your Logstash configuration, or a different one. -`xpack.monitoring.elasticsearch.cloud_auth`:: +`monitoring.elasticsearch.cloud_auth`:: If you're using {es} in {ecloud}, you can set your auth credentials here. -This setting is an alternative to both `xpack.monitoring.elasticsearch.username` -and `xpack.monitoring.elasticsearch.password`. If `cloud_auth` is configured, +This setting is an alternative to both `monitoring.elasticsearch.username` +and `monitoring.elasticsearch.password`. If `cloud_auth` is configured, those settings should not be used.