Skip to content
172 changes: 77 additions & 95 deletions docs/en/stack/monitoring/production.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -2,16 +2,21 @@
[[monitoring-production]]
== Monitoring in a production environment

By default, {xpack} monitoring agents on {es} index data
By default, the {monitoring} agents on {es} index data
into the cluster where they're running. In production, you should
send data to a separate monitoring cluster so that historical monitoring
data is available even if the nodes you are monitoring are not. Using
a dedicated monitoring cluster also enables you to monitor multiple
clusters from a central location.
send data to a separate _monitoring cluster_ so that historical monitoring
data is available even if the nodes you are monitoring are not.

beta[] In 6.4 and later, you can use {metricbeat} to ship monitoring data about
{kib} to a separate monitoring cluster. In 6.5 and later, you can do the same
for {es}.

If you have at least a gold license, using a dedicated monitoring cluster also
enables you to monitor multiple clusters from a central location.

To store monitoring data in a separate cluster:

. {ref}/configuring-monitoring.html[Set up the {es} cluster you want to use for monitoring].
. Set up the {es} cluster you want to use as the monitoring cluster.
For example, you might set up a two host cluster with the nodes `es-mon-1` and
`es-mon-2`.
+
Expand All @@ -21,22 +26,51 @@ NOTE: To monitor an {es} 7.x cluster, you must run {es}

--

. {kibana-ref}/monitoring-xpack-kibana.html[Configure {monitoring}] in {kib}.
.. (Optional) Verify that the collection of monitoring data is disabled on the
monitoring cluster. By default, the `xpack.monitoring.collection.enabled` setting
is `false`.
+
--
For example, you can use the following APIs to review and change this setting:

[source,js]
----------------------------------
GET _cluster/settings

PUT _cluster/settings
{
"persistent": {
"xpack.monitoring.collection.enabled": false
}
}
----------------------------------
// CONSOLE
--

.. If the {es} {security-features} are enabled on the monitoring cluster, create
users that can send and retrieve monitoring data.
+
--
NOTE: {kib} makes requests to the monitoring cluster as the logged in user.
The username and password credentials must therefore be valid on both the {kib}
server and the monitoring cluster.
NOTE: If you plan to use {kib} to view monitoring data, username and password
credentials must be valid on both the {kib} server and the monitoring cluster.

--

. Create a user on the monitoring cluster that has the
`remote_monitoring_agent` role. These credentials will be used when
data is shipped from the {es} cluster you are monitoring to
your dedicated monitoring cluster. For example, the following request
creates a `remote_monitor` user that has the `remote_monitoring_agent` role:
*** beta[] If you plan to use {metricbeat} to collect data about {es} or {kib},
create a user that has the `remote_monitoring_collector` built-in role and a
user that has the `remote_monitoring_agent`
<<built-in-roles-remote-monitoring-agent,built-in role>>. Alternatively, use the
`remote_monitoring_user` <<built-in-users,built-in user>>.

*** If you plan to use HTTP exporters to route data through your production
cluster, create a user that has the `remote_monitoring_agent`
<<built-in-roles-remote-monitoring-agent,built-in role>>.
+
--
For example, the
following request creates a `remote_monitor` user that has the
`remote_monitoring_agent` role:

[source, sh]
---------------------------------------------------------------
POST /_xpack/security/user/remote_monitor
Expand All @@ -47,94 +81,42 @@ POST /_xpack/security/user/remote_monitor
}
---------------------------------------------------------------
// CONSOLE
--
// TEST[skip:needs-gold+-license]

. Configure each {es} node in the cluster you are
monitoring to send metrics to your monitoring cluster by
configuring an HTTP exporter in the
`xpack.monitoring.exporters` settings in `elasticsearch.yml`.
+
--
[source,yaml]
--------------------------------------------------
xpack.monitoring.exporters:
id1:
type: http
host: ["http://es-mon-1:9200", "http://es-mon2:9200"] <1>
auth.username: remote_monitor <2>
auth.password: changeme
--------------------------------------------------
<1> If SSL/TLS is enabled on the monitoring cluster, you must
connect through HTTPS.
<2> If {security} is disabled on the monitoring cluster, you can
omit `auth.username` and `auth.password`.
Alternatively, use the `remote_monitoring_user` <<built-in-users,built-in user>>.
--

. If SSL/TLS is enabled on the monitoring cluster, specify the trusted
CA certificates that will be used to verify the identity of the nodes
in the monitoring cluster.
+
--
To add a CA certificate to an {es} node's trusted certificates, you
can specify the location of the PEM encoded certificate with the
`certificate_authorities` setting:

[source,yaml]
--------------------------------------------------
xpack.monitoring.exporters:
id1:
type: http
host: ["https://es-mon1:9200", "https://es-mon2:9200"]
auth:
username: agent-user
password: password
ssl:
certificate_authorities: [ "/path/to/ca.crt" ]
--------------------------------------------------

Alternatively, you can configure trusted certificates using a truststore
(a Java Keystore file that contains the certificates):

[source,yaml]
--------------------------------------------------
xpack.monitoring.exporters:
id1:
type: http
host: ["https://es-mon1:9200", "https://es-mon2:9200"]
auth:
username: remote_monitor
password: changeme
ssl:
truststore.path: /path/to/file
truststore.password: password
--------------------------------------------------
. Configure your production cluster to collect data and send it to the
monitoring cluster.

--
** beta[] {ref}/configuring-metricbeat.html[Use {metricbeat}]. This option
is available in 6.5 and later versions.

** {ref}/configuring-monitoring.html[Use HTTP exporters].

. Restart {es} on the nodes in your production cluster.
. (Optional)
{logstash-ref}/configuring-logstash.html[Configure {ls} to collect data and send it to the monitoring cluster].
+
--
TIP: You may want to temporarily {ref}/modules-cluster.html[disable shard
allocation] before you restart your nodes to avoid unnecessary shard
reallocation during the install process.
NOTE: You must configure HTTP exporters in the production cluster to route this
data to the monitoring cluster. It cannot be accomplished by using {metricbeat}.

--

. To verify your {monitoring} installation, point your web browser at your {kib}
host, and select **Monitoring** from the side navigation. When security is enabled,
to view the monitoring dashboards you must log in to {kib} as a user who has
both the `kibana_user` and `monitoring_user` roles. For example:
+
--
[source,js]
--------------------------------------------------
POST /_xpack/security/user/stack-monitor
{
"password" : "changeme",
"roles" : [ "kibana_user", "monitoring_user" ]
}
--------------------------------------------------
// CONSOLE
. (Optional) Configure {kib} to collect data and send it to the monitoring cluster:

//image:images/monitoring.jpg["Monitoring",link="images/monitoring.jpg"]
--
** beta[] {kibana-ref}/monitoring-metricbeat.html[Use {metricbeat}]. This
option is available in 6.4 and later versions.

** {kibana-ref}/monitoring-kibana.html[Use HTTP exporters].

. (Optional) Create a dedicated {kib} instance for monitoring, rather than using
a single {kib} instance to access both your production cluster and monitoring
cluster.

.. (Optional) Disable the collection of monitoring data in this {kib} instance.
Set the `xpack.monitoring.kibana.collection.enabled` setting to `false` in the
`kibana.yml` file. For more information about this setting, see
{kibana-ref}/monitoring-settings-kb.html[Monitoring settings in {kib}].

. {kibana-ref}/monitoring-data.html[Configure {kib} to retrieve and display the monitoring data].