Enable cgroup metric collection by default (#3519)

On Linux cgroup metric collection will be enabled by default. The feature is no longer labeled as experimental. The configuration option has been renamed from `cgroups` to `process.cgroups.enabled`.

CHANGELOG.asciidoc

@@ -17,6 +17,9 @@ https://github.com/elastic/beats/compare/v5.1.1...master[Check the HEAD diff]
 - Change beat generator. Use `$GOPATH/src/github.com/elastic/beats/script/generate.py` to generate a beat. {pull}3452[3452]
 
 *Metricbeat*
+- Linux cgroup metrics are now enabled by default for the system process
+  metricset. The configuration option for the feature was renamed from
+  `cgroups` to `process.cgroups.enabled`. {pull}3519[3519]
 
 *Packetbeat*
 

metricbeat/docs/fields.asciidoc

@@ -6305,7 +6305,6 @@ The hard limit on the number of file descriptors opened by the process. The hard
 [float]
 == cgroup Fields
 
-experimental[]
 Metrics and limits from the cgroup of which the task is a member. cgroup metrics are reported when the process has membership in a non-root cgroup. These metrics are only available on Linux.
 
 

metricbeat/docs/metricbeat-in-a-container.asciidoc

@@ -4,7 +4,7 @@
 Elastic does not provide any official container images for Metricbeat. The
 examples on this page assume you are using your own Metricbeat container image.
 
-When executing Metricbeat in a container, there are some important 
+When executing Metricbeat in a container, there are some important
 things to be aware of if you want to monitor the host machine or other
 containers. Let's walk-through some examples using Docker as our container
 orchestration tool.
@@ -18,14 +18,14 @@ work properly inside of a container. This enables Metricbeat to monitor the
 host machine from within the container.
 
 [source,sh]
----
+----
 sudo docker run \
   --volume=/proc:/hostfs/proc:ro \ <1>
   --volume=/sys/fs/cgroup:/hostfs/sys/fs/cgroup:ro \ <2>
   --volume=/:/hostfs:ro \ <3>
   --net=host <4>
   my/metricbeat:latest -system.hostfs=/hostfs
----
+----
 
 <1> Metricbeat's <<metricbeat-module-system,system module>> collects much of its data through the Linux proc
 filesystem, which is normally located at `/proc`. Because containers
@@ -34,10 +34,10 @@ container's `/proc` is different than the host's `/proc`. To account for this, y
 can mount the host's `/proc` filesystem inside of the container and tell
 Metricbeat to look inside the `/hostfs` directory when looking for `/proc` by
 using the `-system.hostfs=/hostfs` CLI flag.
-<2> If you have enabled cgroup reporting (an experimental feature) from the
-<<metricbeat-metricset-system-process,system process metricset>>, then you need to mount the host's cgroup mountpoints
-within the container. They need to be mounted inside the directory specified by
-the `-system.hostfs` CLI flag.
+<2> If cgroup reporting is enabled for the
+<<metricbeat-metricset-system-process,system process metricset>>, then you need
+to mount the host's cgroup mountpoints within the container. They need to be
+mounted inside the directory specified by the `-system.hostfs` CLI flag.
 <3> If you want to be able to monitor filesystems from the host by using the
 <<metricbeat-metricset-system-filesystem,system filesystem metricset>>, then those filesystems need to be mounted inside
 of the container. They can be mounted at any location.
@@ -55,12 +55,12 @@ Next let's look at an example of monitoring a containerized service from a
 Metricbeat container.
 
 [source,sh]
----
+----
 sudo docker run \
   --link some-mysql:mysql \ <1>
   -e MYSQL_PASSWORD=secret \ <2>
   my/metricbeat:latest
----
+----
 
 <1> Linking the containers enables Metricbeat access the exposed ports of the
 mysql container, and it makes the hostname `mysql` resolvable to Metricbeat.
@@ -71,14 +71,14 @@ variables or as command line flags to Metricbeat (see the `-E` CLI flag in <<com
 The mysql module configuration would look like this:
 
 [source,yaml]
----
+----
 metricbeat.modules:
 - module: mysql
   metricsets: ["status"]
   hosts: ["mysql:3306"] <1>
   username: root
   password: ${MYSQL_PASSWORD} <2>
----
+----
 
 <1> The `mysql` hostname will resolve to the `some-mysql` container's address.
 <2> The `MYSQL_PASSWORD` variable will be evaluated at startup. If the variable

metricbeat/docs/modules/system.asciidoc

@@ -26,17 +26,18 @@ metricbeat.modules:
   metricsets: ["process"]
   processes: ['.*']
 ----
-*`cgroups`*:: When the `process` metricset is enabled, you can use the boolean
-`cgroups` option to enable the experimental cgroup metrics on Linux.
+*`process.cgroups.enabled`*:: When the `process` metricset is enabled, you can
+use this boolean configuration option to disable cgroup metrics. By default
+cgroup metrics collection is enabled.
 +
-The following example config enables cgroups metrics on Linux.
+The following example config disables cgroup metrics on Linux.
 +
 [source,yaml]
 ----
 metricbeat.modules:
 - module: system
   metricsets: ["process"]
-  cgroups: true
+  process.cgroups.enabled: false
 ----
 *`cpu_ticks`*:: When the `cpu` or `core` metricset is enabled, you can specify `cpu_ticks: true` to report CPU ticks in addition to CPU percentages stats. For example:
 +

metricbeat/metricbeat.full.yml

@@ -64,8 +64,8 @@ metricbeat.modules:
   # if true, exports the CPU usage in ticks, together with the percentage values
   #cpu_ticks: false
 
-  # EXPERIMENTAL: cgroups can be enabled for the process metricset.
-  #cgroups: false
+  # Enable collection of cgroup metrics from processes on Linux.
+  #process.cgroups.enabled: true
 
   # A list of regular expressions used to whitelist environment variables
   # reported with the process metricset's events. Defaults to empty.

metricbeat/module/system/_meta/config.full.yml

@@ -36,8 +36,8 @@
   # if true, exports the CPU usage in ticks, together with the percentage values
   #cpu_ticks: false
 
-  # EXPERIMENTAL: cgroups can be enabled for the process metricset.
-  #cgroups: false
+  # Enable collection of cgroup metrics from processes on Linux.
+  #process.cgroups.enabled: true
 
   # A list of regular expressions used to whitelist environment variables
   # reported with the process metricset's events. Defaults to empty.

metricbeat/module/system/_meta/docs.asciidoc

@@ -21,17 +21,18 @@ metricbeat.modules:
   metricsets: ["process"]
   processes: ['.*']
 ----
-*`cgroups`*:: When the `process` metricset is enabled, you can use the boolean
-`cgroups` option to enable the experimental cgroup metrics on Linux.
+*`process.cgroups.enabled`*:: When the `process` metricset is enabled, you can
+use this boolean configuration option to disable cgroup metrics. By default
+cgroup metrics collection is enabled.
 +
-The following example config enables cgroups metrics on Linux.
+The following example config disables cgroup metrics on Linux.
 +
 [source,yaml]
 ----
 metricbeat.modules:
 - module: system
   metricsets: ["process"]
-  cgroups: true
+  process.cgroups.enabled: false
 ----
 *`cpu_ticks`*:: When the `cpu` or `core` metricset is enabled, you can specify `cpu_ticks: true` to report CPU ticks in addition to CPU percentages stats. For example:
 +

metricbeat/module/system/process/_meta/docs.asciidoc

@@ -13,8 +13,7 @@ This metricset is available on:
 [float]
 === Control Group (cgroup) Metrics
 
-On Linux this metricset can collect metrics from any cgroups that the process
-is a member of. This feature is currently experimental, and it is not enabled by
-default so you must add `cgroup: true` to the system module configuration to
-enable it.
+On Linux this metricset will collect metrics from any cgroups that the process
+is a member of. This feature is enabled by default and can be disabled by adding
+`process.cgroup.enabled: false` to the system module configuration.
 

metricbeat/module/system/process/_meta/fields.yml

@@ -116,8 +116,6 @@
     - name: cgroup
       type: group
       description: >
-        experimental[]
-
         Metrics and limits from the cgroup of which the task is a member.
         cgroup metrics are reported when the process has membership in a
         non-root cgroup. These metrics are only available on Linux.

metricbeat/module/system/process/process.go

@@ -34,12 +34,11 @@ type MetricSet struct {
 // New creates and returns a new MetricSet.
 func New(base mb.BaseMetricSet) (mb.MetricSet, error) {
 	config := struct {
-		Procs        []string `config:"processes"` // collect all processes by default
-		Cgroups      bool     `config:"cgroups"`
+		Procs        []string `config:"processes"`
+		Cgroups      *bool    `config:"process.cgroups.enabled"`
 		EnvWhitelist []string `config:"process.env.whitelist"`
 	}{
-		Procs:   []string{".*"},
-		Cgroups: false,
+		Procs: []string{".*"}, // collect all processes by default
 	}
 	if err := base.Module().UnpackConfig(&config); err != nil {
 		return nil, err
@@ -63,8 +62,8 @@ func New(base mb.BaseMetricSet) (mb.MetricSet, error) {
 			return nil, fmt.Errorf("unexpected module type")
 		}
 
-		if config.Cgroups {
-			logp.Warn("EXPERIMENTAL: Cgroup is enabled for the system.process MetricSet.")
+		if config.Cgroups == nil || *config.Cgroups {
+			debugf("process cgroup data collection is enabled")
 			m.cgroup, err = cgroup.NewReader(systemModule.HostFS, true)
 			if err != nil {
 				return nil, errors.Wrap(err, "error initializing cgroup reader")

metricbeat/module/system/socket/_meta/docs.asciidoc

@@ -8,20 +8,20 @@ The system `socket` metricset reports an event for each new TCP socket that it
 sees. It does this by polling the kernel periodically to get a dump of all
 sockets. You set the polling interval by configuring the `period` option.
 Specifying a short polling interval with this metricset is important to avoid
-missing short-lived connections. For example: 
+missing short-lived connections. For example:
 
 [source,yaml]
----
+----
 metricbeat.modules:
 - module: system
   metricsets: [cpu, memory]
-- module: system 
+- module: system
   metricsets: [socket] <1>
   period: 1s
----
+----
 
 <1> You can configure the `socket` metricset separately to specify a different
-`period` value than the other metricsets. 
+`period` value than the other metricsets.
 
 The metricset reports the process that has the socket open. In order to provide
 this information, Metricbeat must be running as root. Root access is also
@@ -30,16 +30,16 @@ required to read the file descriptor information of other processes.
 You can configure the metricset to perform a reverse lookup on the remote IP,
 and the returned hostname will be added to the event and cached. If a hostname
 is found, then the eTLD+1 (effective top-level domain plus one level) value will
-also be added to the event. Reverse lookups are disabled by default. 
+also be added to the event. Reverse lookups are disabled by default.
 
 The following example shows the full configuration for the metricset along with
 the defaults.
 
 [source,yaml]
----
+----
 - module: system
   metricsets: [socket]
   socket.reverse_lookup.enabled: false
   socket.reverse_lookup.success_ttl: 60s
   socket.reverse_lookup.failure_ttl: 60s
----
+----

metricbeat/tests/system/test_system.py

@@ -37,8 +37,9 @@
 # cmdline is also part of the system process fields, but it may not be present
 # for some kernel level processes. fd is also part of the system process, but
 # is not available on all OSes and requires root to read for all processes.
+# cgroup is only available on linux.
 SYSTEM_PROCESS_FIELDS = ["cpu", "memory", "name", "pid", "ppid", "pgid",
-                         "state", "username"]
+                         "state", "username", "cgroup"]
 
 
 class SystemTest(metricbeat.BaseTest):
@@ -325,6 +326,9 @@ def test_process(self):
         """
         Test system/process output.
         """
+        if not sys.platform.startswith("linux") and "cgroup" in SYSTEM_PROCESS_FIELDS:
+            SYSTEM_PROCESS_FIELDS.remove("cgroup")
+
         self.render_config_template(modules=[{
             "name": "system",
             "metricsets": ["process"],