Change integrations to have a unique instance key

CHANGELOG.md

@@ -4,6 +4,7 @@
 
 - [BUGFIX] Validate logs config when using logs_instance with automatic logging processor (@mapno)
 
+- [CHANGE] Self-scraped integrations will now use an SUO-specific value for the `instance` label. (@rfratto)
 
 # v0.20.0 (2021-10-28)
 

docs/configuration/integrations/_index.md

@@ -16,6 +16,13 @@ agent:
   # collect and send metrics about itself.
   [enabled: <boolean> | default = false]
 
+  # Sets an explicit value for the instance label when the integration is
+  # self-scraped. Overrides inferred values.
+  #
+  # The default value for this integration is inferred from the agent hostname
+  # and HTTP listen port, delimited by a colon.
+  [instance: <string>]
+
   # Automatically collect metrics from this integration. If disabled,
   # the agent integration will be run but not scraped and thus not
   # remote_written. Metrics for the integration will be exposed at
@@ -94,25 +101,6 @@ github_exporter: <github_exporter_config>
 # and can be scraped by an external process.
 [scrape_integrations: <boolean> | default = true]
 
-# When true, replaces the instance label with the hostname of the machine,
-# rather than 127.0.0.1:<server.http_listen_port>. Useful when running multiple
-# Agents with the same integrations and uniquely identifying where metrics are
-# coming from.
-#
-# The value for the instance label can be replaced by providing custom
-# relabel_configs for an integration. The overwritten instance label will be
-# available when relabel_configs run.
-[replace_instance_label: <boolean> | default = true]
-
-# When true, adds an agent_hostname label to all samples coming from
-# integrations. The value of the agent_hostname label will be the
-# value of $HOSTNAME (if available) or the machine's hostname.
-#
-# DEPRECATED. May be removed in a future version. Rely on
-# replace_instance_label instead, since it has better compatability
-# with existing dashboards.
-[use_hostname_label: <boolean> | default = true]
-
 # Extra labels to add to all samples coming from integrations.
 labels:
   { <string>: <string> }

docs/configuration/integrations/consul-exporter-config.md

@@ -16,6 +16,13 @@ Full reference of options:
   # collect system metrics from the configured consul server address
   [enabled: <boolean> | default = false]
 
+  # Sets an explicit value for the instance label when the integration is
+  # self-scraped. Overrides inferred values.
+  #
+  # The default value for this integration is inferred from the hostname portion
+  # of the server URL.
+  [instance: <string>]
+
   # Automatically collect metrics from this integration. If disabled,
   # the consul_exporter integration will be run but not scraped and thus not
   # remote-written. Metrics for the integration will be exposed at

docs/configuration/integrations/dnsmasq-exporter-config.md

@@ -31,6 +31,13 @@ Full reference of options:
   # collect system metrics from the configured dnsmasq server address
   [enabled: <boolean> | default = false]
 
+  # Sets an explicit value for the instance label when the integration is
+  # self-scraped. Overrides inferred values.
+  #
+  # The default value for this integration is inferred from the dnsmasq_address
+  # value.
+  [instance: <string>]
+
   # Automatically collect metrics from this integration. If disabled,
   # the dnsmasq_exporter integration will be run but not scraped and thus not
   # remote-written. Metrics for the integration will be exposed at

docs/configuration/integrations/elasticsearch-exporter-config.md

@@ -12,7 +12,7 @@ the collection of metrics from ElasticSearch servers.
 Note that currently, an Agent can only collect metrics from a single ElasticSearch server.
 However, the exporter is able to collect the metrics from all nodes through that server configured.
 
-We strongly recommend that you configure a separate user for the Agent, and give it only the strictly mandatory 
+We strongly recommend that you configure a separate user for the Agent, and give it only the strictly mandatory
 security privileges necessary for monitoring your node, as per the [official documentation](https://github.com/prometheus-community/elasticsearch_exporter#elasticsearch-7x-security-privileges).
 
 Full reference of options:
@@ -22,6 +22,13 @@ Full reference of options:
   # collect system metrics from the configured ElasticSearch server address
   [enabled: <boolean> | default = false]
 
+  # Sets an explicit value for the instance label when the integration is
+  # self-scraped. Overrides inferred values.
+  #
+  # The default value for this integration is inferred from the hostname portion
+  # of address.
+  [instance: <string>]
+
   # Automatically collect metrics from this integration. If disabled,
   # the elasticsearch_exporter integration will be run but not scraped and thus not
   # remote-written. Metrics for the integration will be exposed at
@@ -57,7 +64,7 @@ Full reference of options:
   #
 
   # HTTP API address of an Elasticsearch node.
-  [ address : <string> | default = "http://localhost:9200" ]
+  [ address: <string> | default = "http://localhost:9200" ]
 
   # Timeout for trying to get stats from Elasticsearch.
   [ timeout: <duration> | default = "5s" ]

docs/configuration/integrations/github-exporter-config.md

@@ -8,7 +8,7 @@ The `github_exporter_config` block configures the `github_exporter` integration,
 which is an embedded version of
 [`github_exporter`](https://github.com/infinityworks/github-exporter). This allows for the collection of metrics from the github api.
 
-We strongly recommend that you configure a separate authentication token for the Agent, and give it only the strictly mandatory 
+We strongly recommend that you configure a separate authentication token for the Agent, and give it only the strictly mandatory
 security privileges necessary for monitoring your repositories, as per the [official documentation](https://docs.github.com/en/rest/reference/permissions-required-for-github-apps).
 We also recommend that you use `api_token_file` parameter, to avoid setting the authentication token directly on the Agent config file.
 
@@ -19,6 +19,13 @@ Full reference of options:
   # collect metrics for the specified github objects.
   [enabled: <boolean> | default = false]
 
+  # Sets an explicit value for the instance label when the integration is
+  # self-scraped. Overrides inferred values.
+  #
+  # The default value for this integration is inferred from the hostname portion
+  # of api_url.
+  [instance: <string>]
+
   # Automatically collect metrics from this integration. If disabled,
   # the github_exporter integration will be run but not scraped and thus not
   # remote-written. Metrics for the integration will be exposed at

docs/configuration/integrations/kafka-exporter-config.md

@@ -8,7 +8,7 @@ The `kafka_exporter_config` block configures the `kafka_exporter`
 integration, which is an embedded version of [`kafka_exporter`](https://github.com/davidmparrott/kafka_exporter).
 This allows for the collection of Kafka Lag metrics and exposing them as Prometheus metrics.
 
-We strongly recommend that you configure a separate user for the Agent, and give it only the strictly mandatory 
+We strongly recommend that you configure a separate user for the Agent, and give it only the strictly mandatory
 security privileges necessary for monitoring your node, as per the [documentation](https://github.com/lightbend/kafka-lag-exporter#required-permissions-for-kafka-acl).
 
 Full reference of options:
@@ -18,6 +18,15 @@ Full reference of options:
   # collect system metrics from the configured dnsmasq server address
   [enabled: <boolean> | default = false]
 
+  # Sets an explicit value for the instance label when the integration is
+  # self-scraped. Overrides inferred values.
+  #
+  # The default value for this integration is inferred from the hostname
+  # portion of the first kafka_uri value. If there is more than one string
+  # in kafka_uri, the integration will fail to load and an instance value
+  # must be manually provided.
+  [instance: <string>]
+
   # Automatically collect metrics from this integration. If disabled,
   # the dnsmasq_exporter integration will be run but not scraped and thus not
   # remote-written. Metrics for the integration will be exposed at
@@ -47,7 +56,7 @@ Full reference of options:
 
   # Monitor the exporter itself and include those metrics in the results.
   [include_exporter_metrics: <bool> | default = false]
-    
+
   # Address array (host:port) of Kafka server
   [kafka_uris: <[]string>]
 
@@ -56,7 +65,7 @@ Full reference of options:
 
   # Only set this to false if using a non-Kafka SASL proxy
   [use_sasl_handshake: <bool> | default = true]
-  
+
   # SASL user name
   [sasl_username: <string>]
 
@@ -93,7 +102,7 @@ Full reference of options:
   # Kafka cluster name
   [kafka_cluster_name: <string>]
 
-  # Metadata refresh interval  
+  # Metadata refresh interval
   [metadata_refresh_interval: <duration> | default = "1m"]
 
   # If true, all scrapes will trigger kafka operations otherwise, they will share results. WARN: This should be disabled on large clusters
@@ -111,4 +120,4 @@ Full reference of options:
   # Regex filter for consumer groups to be monitored
   [groups_filter_regex: <string> | default = ".*"]
 
-```
+```

docs/configuration/integrations/memcached-exporter-config.md

@@ -31,6 +31,13 @@ Full reference of options:
   # collect system metrics from the configured memcached server address
   [enabled: <boolean> | default = false]
 
+  # Sets an explicit value for the instance label when the integration is
+  # self-scraped. Overrides inferred values.
+  #
+  # The default value for this integration is inferred from
+  # memcached_address.
+  [instance: <string>]
+
   # Automatically collect metrics from this integration. If disabled,
   # the memcached_exporter integration will be run but not scraped and thus not
   # remote-written. Metrics for the integration will be exposed at

docs/configuration/integrations/mongodb_exporter-config.md

@@ -23,7 +23,7 @@ relabel_configs:
       replacement: 'prod-cluster'
 ```
 
-We strongly recommend that you configure a separate user for the Agent, and give it only the strictly mandatory 
+We strongly recommend that you configure a separate user for the Agent, and give it only the strictly mandatory
 security privileges necessary for monitoring your node, as per the [official documentation](https://github.com/percona/mongodb_exporter#permissions).
 
 Besides that, there's not much to configure. Please refer to the full reference of options:
@@ -32,6 +32,13 @@ Besides that, there's not much to configure. Please refer to the full reference
   # Enables the mongodb_exporter integration
   [enabled: <boolean> | default = false]
 
+  # Sets an explicit value for the instance label when the integration is
+  # self-scraped. Overrides inferred values.
+  #
+  # The default value for this integration is inferred from the hostname
+  # portion of the mongodb_uri field.
+  [instance: <string>]
+
   # Automatically collect metrics from this integration. If disabled,
   # the mongodb_exporter integration will be run but not scraped and thus not
   # remote-written. Metrics for the integration will be exposed at

docs/configuration/integrations/mysqld-exporter-config.md

@@ -24,7 +24,7 @@ mysqld_exporter:
     replacement: server-a
 ```
 
-We strongly recommend that you configure a separate user for the Agent, and give it only the strictly mandatory 
+We strongly recommend that you configure a separate user for the Agent, and give it only the strictly mandatory
 security privileges necessary for monitoring your node, as per the [official documentation](https://github.com/prometheus/mysqld_exporter#required-grants).
 
 Full reference of options:
@@ -34,6 +34,14 @@ Full reference of options:
   # metrics from a MySQL server.
   [enabled: <boolean> | default = false]
 
+  # Sets an explicit value for the instance label when the integration is
+  # self-scraped. Overrides inferred values.
+  #
+  # The default value for this integration is a truncated version of the
+  # connection DSN, containing only the server and db name. (Credentials
+  # are not included.)
+  [instance: <string>]
+
   # Automatically collect metrics from this integration. If disabled,
   # the mysqld_exporter integration will be run but not scraped and thus not
   # remote-written. Metrics for the integration will be exposed at

docs/configuration/integrations/node-exporter-config.md

@@ -174,6 +174,13 @@ the Agent is running on is a no-op.
   # collect system metrics from the host UNIX system.
   [enabled: <boolean> | default = false]
 
+  # Sets an explicit value for the instance label when the integration is
+  # self-scraped. Overrides inferred values.
+  #
+  # The default value for this integration is inferred from the agent hostname
+  # and HTTP listen port, delimited by a colon.
+  [instance: <string>]
+
   # Automatically collect metrics from this integration. If disabled,
   # the node_exporter integration will be run but not scraped and thus not remote-written. Metrics for the
   # integration will be exposed at /integrations/node_exporter/metrics and can

docs/configuration/integrations/postgres-exporter-config.md

@@ -9,7 +9,7 @@ integration, which is an embedded version of
 [`postgres_exporter`](https://github.com/prometheus-community/postgres_exporter). This
 allows for the collection of metrics from Postgres servers.
 
-We strongly recommend that you configure a separate user for the Agent, and give it only the strictly mandatory 
+We strongly recommend that you configure a separate user for the Agent, and give it only the strictly mandatory
 security privileges necessary for monitoring your node, as per the [official documentation](https://github.com/prometheus-community/postgres_exporter#running-as-non-superuser).
 
 Full reference of options:
@@ -19,6 +19,18 @@ Full reference of options:
   # collect system metrics from the configured postgres server address
   [enabled: <boolean> | default = false]
 
+  # Sets an explicit value for the instance label when the integration is
+  # self-scraped. Overrides inferred values.
+  #
+  # The default value for this integration is inferred from a truncated version of
+  # the first DSN in data_source_names. The truncated DSN includes the hostname
+  # and database name (if used) of the server, but does not include any user
+  # information.
+  #
+  # If data_source_names contains more than one entry, the integration will fail to
+  # load and a value for instance must be manually provided.
+  [instance: <string>]
+
   # Automatically collect metrics from this integration. If disabled,
   # the postgres_exporter integration will be run but not scraped and thus not
   # remote-written. Metrics for the integration will be exposed at

docs/configuration/integrations/process-exporter-config.md

@@ -70,6 +70,13 @@ Full reference of options:
   # collect system metrics from the host UNIX system.
   [enabled: <boolean> | default = false]
 
+  # Sets an explicit value for the instance label when the integration is
+  # self-scraped. Overrides inferred values.
+  #
+  # The default value for this integration is inferred from the agent hostname
+  # and HTTP listen port, delimited by a colon.
+  [instance: <string>]
+
   # Automatically collect metrics from this integration. If disabled,
   # the process_exporter integration will be run but not scraped and thus not
   # remote-written. Metrics for the integration will be exposed at

docs/configuration/integrations/redis-exporter-config.md

@@ -18,7 +18,7 @@ redis_exporter:
     replacement: redis-2
 ```
 
-We strongly recommend that you configure a separate user for the Agent, and give it only the strictly mandatory 
+We strongly recommend that you configure a separate user for the Agent, and give it only the strictly mandatory
 security privileges necessary for monitoring your node, as per the [official documentation](https://github.com/oliver006/redis_exporter#authenticating-with-redis).
 
 Full reference of options:
@@ -27,6 +27,13 @@ Full reference of options:
   # collect system metrics from the configured redis address
   [enabled: <boolean> | default = false]
 
+  # Sets an explicit value for the instance label when the integration is
+  # self-scraped. Overrides inferred values.
+  #
+  # The default value for this integration is inferred from the hostname
+  # portion of redis_addr.
+  [instance: <string>]
+
   # Automatically collect metrics from this integration. If disabled,
   # the redis_exporter integration will be run but not scraped and thus not
   # remote-written. Metrics for the integration will be exposed at

docs/configuration/integrations/statsd-exporter-config.md

@@ -16,6 +16,13 @@ Full reference of options:
   # collect system metrics from the configured statsd server address
   [enabled: <boolean> | default = false]
 
+  # Sets an explicit value for the instance label when the integration is
+  # self-scraped. Overrides inferred values.
+  #
+  # The default value for this integration is inferred from the agent hostname
+  # and HTTP listen port, delimited by a colon.
+  [instance: <string>]
+
   # Automatically collect metrics from this integration. If disabled,
   # the statsd_exporter integration will be run but not scraped and thus not
   # remote-written. Metrics for the integration will be exposed at

docs/configuration/integrations/windows-exporter-config.md

@@ -16,6 +16,13 @@ Full reference of options:
   # collect system metrics from the local windows instance
   [enabled: <boolean> | default = false]
 
+  # Sets an explicit value for the instance label when the integration is
+  # self-scraped. Overrides inferred values.
+  #
+  # The default value for this integration is inferred from the agent hostname
+  # and HTTP listen port, delimited by a colon.
+  [instance: <string>]
+
   # Automatically collect metrics from this integration. If disabled,
   # the consul_exporter integration will be run but not scraped and thus not
   # remote-written. Metrics for the integration will be exposed at

docs/upgrade-guide/_index.md

@@ -12,6 +12,31 @@ releases and how to migrate to newer versions.
 
 These changes will come in a future version.
 
+### Integrations: Change in how instance labels are handled (Breaking change)
+
+Integrations will now use a SUO-specific `instance` label value. Integrations
+that apply to a whole machine or agent will continue to use `<agent machine
+hostname>:<agent listen port>`, but integrations that connect to an external
+system will now infer an appropriate value based on the config for that specific
+integration. Please refer to the documentation for each integration for which
+defaults are used.
+
+*Note:* In some cases, a default value for `instance` cannot be inferred. This
+is the case for mongodb_exporter and postgres_exporter if more than one SUO is
+being connected to. In these cases, the instance value can be manually set by
+configuring the `instance` field on the integration. This can also be useful if
+two agents infer the same value for instance for the same integration.
+
+As part of this change, the `agent_hostname` label is permanently affixed to
+self-scraped integrations and cannot be disabled. This disambigutates multiple
+agents using the same instance label for an integration, and allows users to
+identify which agents need to be updated with an override for `instance`.
+
+Both `use_hostname_label` and `replace_instance_label` are now both deprecated
+and ignored from the YAML file, permanently treated as true. A future release
+will fully remove these fields, causing YAML errors on load instead of being
+silently ignored.
+
 ## v0.20.0
 
 ### Traces: Changes to receiver's TLS config (Breaking change).