[6.4][DOCS] Adds troubleshooting and production monitoring pages

docs/reference/monitoring/collectors.asciidoc

@@ -140,4 +140,4 @@ related to monitoring data, which can be very useful when there are a large
 number of Logstash nodes or Beats.
 
 For more information about typical monitoring architectures, see
-{xpack-ref}/how-monitoring-works.html[How Monitoring Works].
+<<monitoring-overview>>.

docs/reference/monitoring/configuring-monitoring.asciidoc

@@ -51,7 +51,8 @@ as {kib}, Beats, and Logstash to a monitoring cluster:
 .. Verify that `xpack.monitoring.collection.enabled` settings are `true` on each 
 node in the cluster. 
 
-..  {stack-ov}/xpack-monitoring.html[Configure monitoring across the {stack}].
+.. Configure monitoring across the {stack}. For example, see
+<<monitoring-production>>.
 
 . Identify where to store monitoring data. 
 +
@@ -68,7 +69,7 @@ cluster that stores the monitoring data must have at least one
 <<ingest,ingest node>>.
 
 For more information about typical monitoring architectures, 
-see {stack-ov}/how-monitoring-works.html[How Monitoring Works].
+see <<monitoring-overview>>.
 --
 
 . If {es} {security-features} are enabled and you are using an `http` exporter

docs/reference/monitoring/index.asciidoc

@@ -11,6 +11,7 @@ performance of your {es} cluster.
 
 include::overview.asciidoc[]
 include::how-monitoring-works.asciidoc[]
+include::production.asciidoc[]
 include::configuring-monitoring.asciidoc[]
 include::pause-export.asciidoc[]
 include::indices.asciidoc[]
@@ -19,3 +20,4 @@ include::collectors.asciidoc[]
 include::exporters.asciidoc[]
 include::local-export.asciidoc[]
 include::http-export.asciidoc[]
+include::troubleshooting.asciidoc[]

docs/reference/monitoring/production.asciidoc

@@ -0,0 +1,147 @@
+[role="xpack"]
+[testenv="gold"]
+[[monitoring-production]]
+== Monitoring in a production environment
+
+In production, you should send monitoring data to a separate _monitoring cluster_
+so that historical data is available even when the nodes you are monitoring are
+not. For example, you can use {metricbeat} to ship monitoring data about {kib}
+to a separate monitoring cluster.
+
+If you have an appropriate license, using a dedicated monitoring cluster also 
+enables you to monitor multiple clusters from a central location.
+
+To store monitoring data in a separate cluster:
+
+. <<configuring-monitoring,Set up the {es} cluster you want to use for monitoring>>. 
+For example, you might set up a two host cluster with the nodes `es-mon-1` and 
+`es-mon-2`.
++
+--
+[IMPORTANT]
+===============================
+* To monitor an {es} 6.x cluster, you must run {es}
+6.x on the monitoring cluster.
+* There must be at least one <<ingest,ingest node>> in the monitoring
+cluster; it does not need to be a dedicated ingest node.
+===============================
+
+--
+
+. {kibana-ref}/monitoring-xpack-kibana.html[Configure {monitoring}] in {kib}.
++
+--
+NOTE: {kib} makes requests to the monitoring cluster as the logged in user.
+The username and password credentials must therefore be valid on both the {kib}
+server and the monitoring cluster.
+
+--
+
+. Create a user on the monitoring cluster that has the
+`remote_monitoring_agent` role. These credentials will be used when
+data is shipped from the {es} cluster you are monitoring to
+your dedicated monitoring cluster. For example, the following request
+creates a `remote_monitor` user that has the `remote_monitoring_agent` role:
++
+--
+[source, sh]
+---------------------------------------------------------------
+POST /_xpack/security/user/remote_monitor
+{
+  "password" : "changeme",
+  "roles" : [ "remote_monitoring_agent"],
+  "full_name" : "Internal Agent For Remote Monitoring"
+}
+---------------------------------------------------------------
+// CONSOLE
+--
+
+. Configure each {es} node in the cluster you are
+monitoring to send metrics to your monitoring cluster by
+configuring an HTTP exporter in the
+`xpack.monitoring.exporters` settings in `elasticsearch.yml`.
++
+--
+[source,yaml]
+--------------------------------------------------
+xpack.monitoring.exporters:
+  id1:
+    type: http
+    host: ["http://es-mon-1:9200", "http://es-mon2:9200"] <1>
+    auth.username: remote_monitor <2>
+    auth.password: changeme
+--------------------------------------------------
+<1> If SSL/TLS is enabled on the monitoring cluster, you must
+connect through HTTPS.
+<2> If {security} is disabled on the monitoring cluster, you can
+omit `auth.username` and `auth.password`.
+--
+
+. If SSL/TLS is enabled on the monitoring cluster, specify the trusted
+CA certificate(s) that will be used to verify the identity of the nodes
+in the monitoring cluster.
++
+--
+To add a CA certificate to an {es} node's trusted certificates, you
+can specify the location of the PEM encoded certificate with the
+`certificate_authorities` setting:
+
+[source,yaml]
+--------------------------------------------------
+xpack.monitoring.exporters:
+  id1:
+    type: http
+    host: ["https://es-mon1:9200", "https://es-mon2:9200"]
+    auth:
+      username: agent-user
+      password: password
+    ssl:
+      certificate_authorities: [ "/path/to/ca.crt" ]
+--------------------------------------------------
+
+Alternatively, you can configure trusted certificates using a truststore
+(a Java Keystore file that contains the certificates):
+
+[source,yaml]
+--------------------------------------------------
+xpack.monitoring.exporters:
+  id1:
+    type: http
+    host: ["https://es-mon1:9200", "https://es-mon2:9200"]
+    auth:
+      username: remote_monitor
+      password: changeme
+    ssl:
+      truststore.path: /path/to/file
+      truststore.password: password
+--------------------------------------------------
+
+--
+
+. Restart {es} on the nodes in your production cluster.
++
+--
+TIP: You may want to temporarily
+<<modules-cluster,disable shard allocation>> before you restart your nodes to
+avoid unnecessary shard reallocation during the install process.
+
+--
+
+. To verify your {monitoring} installation, point your web browser at your {kib}
+host, and select **Monitoring** from the side navigation. When security is enabled,
+to view the monitoring dashboards you must log in to {kib} as a user who has
+both the `kibana_user` and `monitoring_user` roles. For example:
++
+--
+[source,js]
+--------------------------------------------------
+POST /_xpack/security/user/stack-monitor
+{
+  "password" : "changeme",
+  "roles" : [ "kibana_user", "monitoring_user" ]
+}
+--------------------------------------------------
+// CONSOLE
+
+//image:images/monitoring.jpg["Monitoring",link="images/monitoring.jpg"]
+--

docs/reference/monitoring/troubleshooting.asciidoc

@@ -0,0 +1,15 @@
+[[monitoring-troubleshooting]]
+== Troubleshooting monitoring
+++++
+<titleabbrev>Troubleshooting</titleabbrev>
+++++
+
+Use the information in this section to troubleshoot common problems and find
+answers for frequently asked questions. See also 
+{logstash-ref}/monitoring-troubleshooting.html[Troubleshooting {monitoring} in {ls}].
+
+For issues that you cannot fix yourself … we’re here to help.
+If you are an existing Elastic customer with a support contract, please create
+a ticket in the
+https://support.elastic.co/customers/s/login/[Elastic Support portal].
+Or post in the https://discuss.elastic.co/[Elastic forum].

docs/reference/settings/monitoring-settings.asciidoc

@@ -22,9 +22,6 @@ Logstash, configure
 {logstash-ref}/configuring-logstash.html#monitoring-settings[`xpack.monitoring` settings]
 in `logstash.yml`.
 
-For more information, see
-{xpack-ref}/xpack-monitoring.html[Monitoring the Elastic Stack].
-
 [float]
 [[general-monitoring-settings]]
 ==== General Monitoring Settings
@@ -115,9 +112,9 @@ affect the {watcher} cleaner service too. For more information, see the
 Configures where the agent stores monitoring data. By default, the agent uses a
 local exporter that indexes monitoring data on the cluster where it is installed.
 Use an HTTP exporter to send data to a separate monitoring cluster. For more
-information, see <<local-exporter-settings,Local Exporter Settings>>,
-<<http-exporter-settings,HTTP Exporter Settings>>, and
-{xpack-ref}/how-monitoring-works.html[How Monitoring Works].
+information, see <<local-exporter-settings,local exporter settings>>,
+<<http-exporter-settings,HTTP exporter settings>>, and
+<<how-monitoring-works>>.
 
 [float]
 [[local-exporter-settings]]