Docs deprecate setup dashboards (#1142)

* Docs: deprecate loading dashboards and index pattern.

Add deprecation notes to apm-server.yml and docs
for loading dashboards and index patterns via Kibana.
Add deprecation note for setting up Kibana endpoint as it won't be
needed any more.

implements #1135

Author: simitt

CHANGELOG.asciidoc

@@ -24,6 +24,7 @@ https://github.com/elastic/apm-server/compare/6.3\...master[View commits]
 - Add source_mapping.elasticsearch configuration option {pull}1114[1114].
 - Add /v1/metrics endpoint {pull}1000[1000] {pull}1121[1121].
 - Push onboarding doc to separate ES index {pull}1159[1159].
+- Deprecate usage of `apm-server setup` {pull}1142[1142].
 
 
 [[release-notes-6.3]]

NOTICE.txt

@@ -1694,6 +1694,7 @@ Apache License 2.0
 
 --------------------------------------------------------------------
 Dependency: github.com/spf13/pflag
+Version: v1.0.1
 Revision: e57e3eeb33f795204c1ca35f56c44f83227c6e66
 License type (autodetected): BSD-3-Clause
 ./vendor/github.com/spf13/pflag/LICENSE:

_meta/beat.yml

@@ -155,7 +155,51 @@ setup.template.settings:
     #number_of_routing_shards: 30
 
 
-#============================== Dashboards =====================================
+#============================== Template =====================================
+
+# A template is used to set the mapping in Elasticsearch
+# By default template loading is enabled and the template is loaded.
+# These settings can be adjusted to load your own template or overwrite existing ones.
+
+# Set to false to disable template loading.
+#setup.template.enabled: true
+
+# Template name. By default the template name is "apm-%{[beat.version]}"
+# The template name and pattern has to be set in case the elasticsearch index pattern is modified.
+#setup.template.name: "apm-%{[beat.version]}"
+
+# Template pattern. By default the template pattern is "apm-%{[beat.version]}-*" to apply to the default index settings.
+# The first part is the version of the beat and then -* is used to match all daily indices.
+# The template name and pattern has to be set in case the elasticsearch index pattern is modified.
+#setup.template.pattern: "apm-%{[beat.version]}-*"
+
+# Path to fields.yml file to generate the template
+#setup.template.fields: "${path.config}/fields.yml"
+
+# Overwrite existing template
+#setup.template.overwrite: false
+
+# Elasticsearch template settings
+#setup.template.settings:
+
+  # A dictionary of settings to place into the settings.index dictionary
+  # of the Elasticsearch template. For more details, please check
+  # https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html
+  #index:
+    #number_of_shards: 1
+    #codec: best_compression
+    #number_of_routing_shards: 30
+
+  # A dictionary of settings for the _source field. For more details, please check
+  # https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-source-field.html
+  #_source:
+    #enabled: false
+
+
+#============================== Deprecated: Dashboards =====================================
+#
+# Deprecated: Loading dashboards from the APM Server into Kibana is deprecated from 6.4 on.
+#             We suggest to use the Kibana UI to load APM Server dashboards and index pattern instead.
 #
 # These settings control loading the sample dashboards to the Kibana index. Loading
 # the dashboards are disabled by default and can be enabled either by setting the
@@ -197,51 +241,14 @@ setup.template.settings:
 #setup.dashboards.retry.maximum: 0
 
 
-#============================== Template =====================================
-
-# A template is used to set the mapping in Elasticsearch
-# By default template loading is enabled and the template is loaded.
-# These settings can be adjusted to load your own template or overwrite existing ones.
+#============================== Deprecated: Kibana =====================================
 
-# Set to false to disable template loading.
-#setup.template.enabled: true
-
-# Template name. By default the template name is "apm-%{[beat.version]}"
-# The template name and pattern has to be set in case the elasticsearch index pattern is modified.
-#setup.template.name: "apm-%{[beat.version]}"
-
-# Template pattern. By default the template pattern is "apm-%{[beat.version]}-*" to apply to the default index settings.
-# The first part is the version of the beat and then -* is used to match all daily indices.
-# The template name and pattern has to be set in case the elasticsearch index pattern is modified.
-#setup.template.pattern: "apm-%{[beat.version]}-*"
-
-# Path to fields.yml file to generate the template
-#setup.template.fields: "${path.config}/fields.yml"
-
-# Overwrite existing template
-#setup.template.overwrite: false
-
-# Elasticsearch template settings
-#setup.template.settings:
-
-  # A dictionary of settings to place into the settings.index dictionary
-  # of the Elasticsearch template. For more details, please check
-  # https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html
-  #index:
-    #number_of_shards: 1
-    #codec: best_compression
-    #number_of_routing_shards: 30
-
-  # A dictionary of settings for the _source field. For more details, please check
-  # https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-source-field.html
-  #_source:
-    #enabled: false
-
-
-#============================== Kibana =====================================
+# Deprecated: Starting with APM Server version 6.4, loading dashboards and index pattern 
+#             from the APM Server into Kibana is deprecated.
+#             We suggest to use the Kibana UI to load APM Server dashboards and index pattern instead.
+#
+#             Setting up a Kibana endpoint is not necessary when loading the index pattern and dashboards via the UI.
 
-# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
-# This requires a Kibana endpoint configuration.
 #setup.kibana:
 
   # Kibana Host

apm-server.yml

@@ -155,7 +155,51 @@ setup.template.settings:
     #number_of_routing_shards: 30
 
 
-#============================== Dashboards =====================================
+#============================== Template =====================================
+
+# A template is used to set the mapping in Elasticsearch
+# By default template loading is enabled and the template is loaded.
+# These settings can be adjusted to load your own template or overwrite existing ones.
+
+# Set to false to disable template loading.
+#setup.template.enabled: true
+
+# Template name. By default the template name is "apm-%{[beat.version]}"
+# The template name and pattern has to be set in case the elasticsearch index pattern is modified.
+#setup.template.name: "apm-%{[beat.version]}"
+
+# Template pattern. By default the template pattern is "apm-%{[beat.version]}-*" to apply to the default index settings.
+# The first part is the version of the beat and then -* is used to match all daily indices.
+# The template name and pattern has to be set in case the elasticsearch index pattern is modified.
+#setup.template.pattern: "apm-%{[beat.version]}-*"
+
+# Path to fields.yml file to generate the template
+#setup.template.fields: "${path.config}/fields.yml"
+
+# Overwrite existing template
+#setup.template.overwrite: false
+
+# Elasticsearch template settings
+#setup.template.settings:
+
+  # A dictionary of settings to place into the settings.index dictionary
+  # of the Elasticsearch template. For more details, please check
+  # https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html
+  #index:
+    #number_of_shards: 1
+    #codec: best_compression
+    #number_of_routing_shards: 30
+
+  # A dictionary of settings for the _source field. For more details, please check
+  # https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-source-field.html
+  #_source:
+    #enabled: false
+
+
+#============================== Deprecated: Dashboards =====================================
+#
+# Deprecated: Loading dashboards from the APM Server into Kibana is deprecated from 6.4 on.
+#             We suggest to use the Kibana UI to load APM Server dashboards and index pattern instead.
 #
 # These settings control loading the sample dashboards to the Kibana index. Loading
 # the dashboards are disabled by default and can be enabled either by setting the
@@ -197,51 +241,14 @@ setup.template.settings:
 #setup.dashboards.retry.maximum: 0
 
 
-#============================== Template =====================================
-
-# A template is used to set the mapping in Elasticsearch
-# By default template loading is enabled and the template is loaded.
-# These settings can be adjusted to load your own template or overwrite existing ones.
+#============================== Deprecated: Kibana =====================================
 
-# Set to false to disable template loading.
-#setup.template.enabled: true
-
-# Template name. By default the template name is "apm-%{[beat.version]}"
-# The template name and pattern has to be set in case the elasticsearch index pattern is modified.
-#setup.template.name: "apm-%{[beat.version]}"
-
-# Template pattern. By default the template pattern is "apm-%{[beat.version]}-*" to apply to the default index settings.
-# The first part is the version of the beat and then -* is used to match all daily indices.
-# The template name and pattern has to be set in case the elasticsearch index pattern is modified.
-#setup.template.pattern: "apm-%{[beat.version]}-*"
-
-# Path to fields.yml file to generate the template
-#setup.template.fields: "${path.config}/fields.yml"
-
-# Overwrite existing template
-#setup.template.overwrite: false
-
-# Elasticsearch template settings
-#setup.template.settings:
-
-  # A dictionary of settings to place into the settings.index dictionary
-  # of the Elasticsearch template. For more details, please check
-  # https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html
-  #index:
-    #number_of_shards: 1
-    #codec: best_compression
-    #number_of_routing_shards: 30
-
-  # A dictionary of settings for the _source field. For more details, please check
-  # https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-source-field.html
-  #_source:
-    #enabled: false
-
-
-#============================== Kibana =====================================
+# Deprecated: Starting with APM Server version 6.4, loading dashboards and index pattern 
+#             from the APM Server into Kibana is deprecated.
+#             We suggest to use the Kibana UI to load APM Server dashboards and index pattern instead.
+#
+#             Setting up a Kibana endpoint is not necessary when loading the index pattern and dashboards via the UI.
 
-# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
-# This requires a Kibana endpoint configuration.
 #setup.kibana:
 
   # Kibana Host

cmd/root.go

@@ -37,4 +37,6 @@ var RootCmd *cmd.BeatsRootCmd
 func init() {
 	var runFlags = pflag.NewFlagSet(Name, pflag.ExitOnError)
 	RootCmd = cmd.GenRootCmdWithIndexPrefixWithRunFlags(Name, IdxPattern, "", beater.New, runFlags)
+	RootCmd.RunCmd.Flags().MarkDeprecated("setup", "use Kibana UI for initial setup.")
+	RootCmd.SetupCmd.Deprecated = "use Kibana UI for initial setup."
 }

docs/copied-from-beats/command-reference.asciidoc

@@ -20,12 +20,22 @@
 :modules-command-short-desc: Manages configured modules
 :run-command-short-desc: Runs {beatname_uc}. This command is used by default if you start {beatname_uc} without specifying a command
 
+ifndef::deprecate_dashboard_loading[]
+
 ifdef::has_ml_jobs[]
 :setup-command-short-desc: Sets up the initial environment, including the index template, Kibana dashboards (when available), and machine learning jobs (when available)
 endif::[]
 
 ifndef::has_ml_jobs[]
-:setup-command-short-desc: Sets up the initial environment, including the index template, Kibana dashboards (when available)
+:setup-command-short-desc: Sets up the initial environment, including the index template and Kibana dashboards (when available)
+endif::[]
+
+endif::[]
+
+ifdef::deprecate_dashboard_loading[]
+
+:setup-command-short-desc: Sets up the initial environment, including the ES index template and Kibana dashboards (deprecated).
+
 endif::[]
 
 :test-command-short-desc: Tests the configuration
@@ -39,9 +49,17 @@ endif::[]
 <titleabbrev>Command reference</titleabbrev>
 ++++
 
+ifndef::deprecate_dashboard_loading[]
 {beatname_uc} provides a command-line interface for starting {beatname_uc} and
-performing common tasks, like testing configuration files and loading
-dashboards. The command-line also supports <<global-flags,global flags>>
+performing common tasks, like testing configuration files and loading dashboards.
+endif::[]
+
+ifdef::deprecate_dashboard_loading[]
+{beatname_uc} provides a command-line interface for starting {beatname_uc} and
+performing common tasks, like testing configuration files and loading dashboards (deprecated).
+endif::[]
+
+The command-line also supports <<global-flags,global flags>>
 for controlling global behaviors.
 
 ifeval::["{beatname_lc}"!="winlogbeat"]
@@ -391,8 +409,19 @@ the end of the file is reached. By default harvesters are closed after
 endif::[]
 
 *`--setup`*::
-Loads the sample Kibana dashboards. If you want to load the dashboards without
-running {beatname_uc}, use the <<setup-command,`setup`>> command instead.
+ifdef::deprecate_dashboard_loading[]
+deprecated[{deprecate_dashboard_loading}]
+endif::[]
++
+ifdef::has_ml_jobs[]
+Loads the initial setup, including Elasticsearch template, Kibana index pattern,
+Kibana dashboards and Machine learning jobs.
+endif::[]
+ifndef::has_ml_jobs[]
+Loads the initial setup, including Elasticsearch template, Kibana index pattern and Kibana dashboards.
+endif::[]
+If you want to use the command without running {beatname_uc}, use the <<setup-command,`setup`>> command instead.
+
 
 ifeval::["{beatname_lc}"=="metricbeat"]
 
@@ -431,13 +460,17 @@ Or:
 [[setup-command]]
 ==== `setup` command
 
-{setup-command-short-desc}.
+{setup-command-short-desc}
 
 * The index template ensures that fields are mapped correctly in Elasticsearch.
+
 * The Kibana dashboards make it easier for you to visualize {beatname_uc} data
 in Kibana.
+
+ifdef::has_ml_jobs[]
 * The machine learning jobs contain the configuration information and metadata
 necessary to analyze data for anomalies.
+endif::[]
 
 Use this command instead of `run --setup` when you want to set up the
 environment without actually running {beatname_uc} and ingesting data.
@@ -452,21 +485,41 @@ environment without actually running {beatname_uc} and ingesting data.
 
 *FLAGS*
 
+ifndef::deprecate_dashboard_loading[]
+*`--dashboards`*::
+Sets up the Kibana dashboards only. This option loads the dashboards from the
+{beatname_uc} package. For more options, such as loading customized dashboards,
+see {beatsdevguide}/import-dashboards.html[Importing Existing Beat Dashboards]
+in the _Beats Developer Guide_.
+endif::[]
+
+ifdef::deprecate_dashboard_loading[]
 *`--dashboards`*::
+
+deprecated[{deprecate_dashboard_loading}]
++
 Sets up the Kibana dashboards only.
+endif::[]
 
 *`-h, --help`*::
 Shows help for the `setup` command.
 
+ifdef::has_ml_jobs[]
+
 *`--machine-learning`*::
 Sets up machine learning job configurations only.
 
+endif::[]
+
 ifeval::["{beatname_lc}"=="filebeat"]
 
 *`--modules MODULE_LIST`*::
 Specifies a comma-separated list of modules. Use this flag to avoid errors when
 there are no modules defined in the +{beatname_lc}.yml+ file.
 
+*`--pipelines`*::
+Sets up ingest pipelines for configured filesets.
+
 endif::[]
 
 *`--template`*::

docs/copied-from-beats/dashboards.asciidoc

@@ -18,14 +18,17 @@ command (as described here) or
 +{beatname_lc}.yml+ config file.
 
 This requires a Kibana endpoint configuration. If you didn't already configure
-a Kibana endpoint, see <<{beatname_lc}-configuration,configure {beatname_uc}>>.
+a Kibana endpoint, see <<{beatname_lc}-configuration,configure {beatname_uc}>>. 
 
 Make sure Kibana is running before you perform this step. If you are accessing a
 secured Kibana instance, make sure you've configured credentials as described in
 <<{beatname_lc}-configuration>>.
 
 To set up the Kibana dashboards for {beatname_uc}, use the appropriate command
-for your system.
+for your system. The command shown here loads the dashboards from the {beatname_uc}
+package. For more options, such as loading customized dashboards, see
+{beatsdevguide}/import-dashboards.html[Importing Existing Beat Dashboards] in
+the _Beats Developer Guide_.
 ifndef::only-elasticsearch[]
 If you've configured the Logstash output, see
 <<load-dashboards-logstash>>.