Update beats framework to 1c86ec7

NOTICE.txt

@@ -274,7 +274,7 @@ SOFTWARE.
 --------------------------------------------------------------------
 Dependency: github.com/elastic/beats
 Version: master
-Revision: f549cec7f437795bec4f239010d0441bbd75af0b
+Revision: 1c86ec7f0d56ed8e8fb22ac426abd9013e3bbd29
 License type (autodetected): Apache-2.0
 ./vendor/github.com/elastic/beats/LICENSE.txt:
 --------------------------------------------------------------------

_beats/CHANGELOG-developer.next.asciidoc

@@ -38,5 +38,7 @@ The list below covers the major changes between 7.0.0-rc2 and master only.
 - Reduce idxmgmt.Supporter interface and rework export commands to reuse logic. {pull}11777[11777],{pull}12065[12065],{pull}12067[12067],{pull}12160[12160]
 - Update urllib3 version to 1.24.2 {pull}11930[11930]
 - Add libbeat/common/cleanup package. {pull}12134[12134]
+- New helper to check for leaked goroutines on tests. {pull}12106[12106]
 - Only Load minimal template if no fields are provided. {pull}12103[12103]
-- Deprecate setup cmds for `template` and `ilm-policy`. Add new setup cmd for `index-management`. {pull}12132[12132]
+- Add new option `IgnoreAllErrors` to `libbeat.common.schema` for skipping fields that failed while converting. {pull}12089[12089]
+- Deprecate setup cmds for `template` and `ilm-policy`. Add new setup cmd for `index-management`. {pull}12132[12132]

_beats/CHANGELOG.next.asciidoc

@@ -12,6 +12,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 
 - Update to Golang 1.12.1. {pull}11330[11330]
 - Update to Golang 1.12.4. {pull}11782[11782]
+- Update to ECS 1.0.1. {pull}12284[12284]
 
 *Auditbeat*
 
@@ -20,6 +21,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Socket dataset: Exclude localhost by default {pull}11993[11993]
 
 *Filebeat*
+
 - Modify apache/error dataset to follow ECS. {pull}8963[8963]
 - Rename many `traefik.access.*` fields to map to ECS. {pull}9005[9005]
 - Fix parsing of GC entries in elasticsearch server log. {issue}9513[9513] {pull}9810[9810]
@@ -52,6 +54,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 
 *Affecting all Beats*
 
+- Fix typo in TLS renegotiation configuration and setting the option correctly {issue}10871[10871], {pull}12354[12354]
 - Ensure all beat commands respect configured settings. {pull}10721[10721]
 - Add missing fields and test cases for libbeat add_kubernetes_metadata processor. {issue}11133[11133], {pull}11134[11134]
 - decode_json_field: process objects and arrays only {pull}11312[11312]
@@ -63,6 +66,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Not hiding error in case of http failure using elastic fetcher {pull}11604[11604]
 - Relax validation of the X-Pack license UID value. {issue}11640[11640]
 - Fix a parsing error with the X-Pack license check on 32-bit system. {issue}11650[11650]
+- Escape BOM on JsonReader before trying to decode line {pull}11661[11661]
 - Fix ILM policy always being overwritten. {pull}11671[11671]
 - Fix template always being overwritten. {pull}11671[11671]
 - Fix matching of string arrays in contains condition. {pull}11691[11691]
@@ -72,6 +76,9 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Replace wmi queries with win32 api calls as they were consuming CPU resources {issue}3249[3249] and {issue}11840[11840]
 - Fix queue.spool.write.flush.events config type. {pull}12080[12080]
 - Fixed a memory leak when using the add_process_metadata processor under Windows. {pull}12100[12100]
+- Fix of docker json parser for missing "log" jsonkey in docker container's log {issue}11464[11464]
+- Fixed Beat ID being reported by GET / API. {pull}12180[12180]
+- Add host.os.codename to fields.yml. {pull}12261[12261]
 
 *Auditbeat*
 
@@ -81,6 +88,12 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Process dataset: Fixed a memory leak under Windows. {pull}12100[12100]
 - Login dataset: Fix re-read of utmp files. {pull}12028[12028]
 - Package dataset: Fixed a crash inside librpm after Auditbeat has been running for a while. {issue}12147[12147] {pull}12168[12168]
+- Fix formatting of config files on macOS and Windows. {pull}12148[12148]
+- Fix direction of incoming IPv6 sockets. {pull}12248[12248]
+- Package dataset: Close librpm handle. {pull}12215[12215]
+- Package dataset: Auto-detect package directories. {pull}12289[12289]
+- Package dataset: Improve dpkg parsing. {pull}12325[12325]
+- System module: Start system module without host ID. {pull}12373[12373]
 
 *Filebeat*
 
@@ -94,6 +107,10 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Reduce memory usage if long lines are truncated to fit `max_bytes` limit. The line buffer is copied into a smaller buffer now. This allows the runtime to release unused memory earlier. {pull}11524[11524]
 - Fix memory leak in Filebeat pipeline acker. {pull}12063[12063]
 - Fix goroutine leak caused on initialization failures of log input. {pull}12125[12125]
+- Fix goroutine leak on non-explicit finalization of log input. {pull}12164[12164]
+- Skipping unparsable log entries from docker json reader {pull}12268[12268]
+- Require client_auth by default when ssl is enabled for tcp input {pull}12333[12333]
+- Require certificate authorities, certificate file, and key when SSL is enabled for the TCP input. {pull}12355[12355]
 
 *Heartbeat*
 
@@ -117,6 +134,13 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Avoid generating hints-based configuration with empty hosts when no exposed port is suitable for the hosts hint. {issue}8264[8264] {pull}12086[12086]
 - Fixed a socket leak in the postgresql module under Windows when SSL is disabled on the server. {pull}11393[11393]
 - Change some field type from scaled_float to long in aws module. {pull}11982[11982]
+- Fixed RabbitMQ `queue` metricset gathering when `consumer_utilisation` is set empty at the metrics source {pull}12089[12089]
+- Fix direction of incoming IPv6 sockets. {pull}12248[12248]
+- Validate that kibana/status metricset cannot be used when xpack is enabled. {pull}12264[12264]
+- Ignore prometheus metrics when their values are NaN or Inf. {pull}12084[12084] {issue}10849[10849]
+- In the kibana/stats metricset, only log error (don't also index it) if xpack is enabled. {pull}12265[12265]
+- Require client_auth by default when ssl is enabled for module http metricset server{pull}12333[12333]
+- Require certificate authorities, certificate file, and key when SSL is enabled for module http metricset server. {pull}12355[12355]
 
 *Packetbeat*
 
@@ -151,21 +175,26 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Add `convert` processor for converting data types of fields. {issue}8124[8124] {pull}11686[11686]
 - New `extract_array` processor. {pull}11761[11761]
 - Add number of goroutines to reported metrics. {pull}12135[12135]
+- Add `proxy_disable` output flag to explicitly ignore proxy environment variables. {issue}11713[11713] {pull}12243[12243]
+- Processor `add_cloud_metadata` adds fields `cloud.account.id` and `cloud.image.id` for AWS EC2. {pull}12307[12307]
 
 *Auditbeat*
 
 - Auditd module: Add `event.outcome` and `event.type` for ECS. {pull}11432[11432]
 - Package: Enable suse. {pull}11634[11634]
 - Add support to the system package dataset for the SUSE OS family. {pull}11634[11634]
 - Process: Add file hash of process executable. {pull}11722[11722]
+- Socket: Add network.transport and network.community_id. {pull}12231[12231]
+- Host: Fill top-level host fields. {pull}12259[12259]
 
 *Filebeat*
 
 - Add more info to message logged when a duplicated symlink file is found {pull}10845[10845]
 - Add option to configure docker input with paths {pull}10687[10687]
 - Add Netflow module to enrich flow events with geoip data. {pull}10877[10877]
 - Set `event.category: network_traffic` for Suricata. {pull}10882[10882]
-- Add configuration knob for auto-discover hints to control whether log harvesting is enabled for the pod/container. {issue}10811[10811] {pull}10911[10911]
+- Allow custom default settings with autodiscover (for example, use of CRI paths for logs). {pull}12193[12193]
+- Allow to disable hints based autodiscover default behavior (fetching all logs). {pull}12193[12193]
 - Change Suricata module pipeline to handle `destination.domain` being set if a reverse DNS processor is used. {issue}10510[10510]
 - Add the `network.community_id` flow identifier to field to the IPTables, Suricata, and Zeek modules. {pull}11005[11005]
 - New Filebeat coredns module to ingest coredns logs. It supports both native coredns deployment and coredns deployment in kubernetes. {pull}11200[11200]
@@ -175,7 +204,12 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Add Filebeat envoyproxy module. {pull}11700[11700]
 - Add apache2(httpd) log path (`/var/log/httpd`) to make apache2 module work out of the box on Redhat-family OSes. {issue}11887[11887] {pull}11888[11888]
 - Add support to new MongoDB additional diagnostic information {pull}11952[11952]
-- New module `palo_alto` for Palo Alto Networks PAN-OS logs. {pull}11999[11999]
+- New module `panw` for Palo Alto Networks PAN-OS logs. {pull}11999[11999]
+- Add RabbitMQ module. {pull}12032[12032]
+- Add new `container` input. {pull}12162[12162]
+- `container` and `docker` inputs now support reading of labels and env vars written by docker JSON file logging driver. {issue}8358[8358]
+- Add specific date processor to convert timezones so same pipeline can be used when convert_timezone is enabled or disabled. {pull}12253[12253]
+- Add MSSQL module {pull}12079[12079]
 
 *Heartbeat*
 
@@ -199,12 +233,16 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Add AWS cloudwatch metricset. {pull}11798[11798] {issue}11734[11734]
 - Add `regions` in aws module config to specify target regions for querying cloudwatch metrics. {issue}11932[11932] {pull}11956[11956]
 - Keep `etcd` followers members from reporting `leader` metricset events {pull}12004[12004]
+- Add overview dashboard to Consul module {pull}10665[10665]
+- New fields were added in the mysql/status metricset. {pull}12227[12227]
 
 *Packetbeat*
 
 *Functionbeat*
 
 - New options to configure roles and VPC. {pull}11779[11779]
+- Export automation templates used to create functions. {pull}11923[11923]
+- Configurable Amazon endpoint. {pull}12369[12369]
 
 *Winlogbeat*
 
@@ -216,6 +254,8 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 
 *Filebeat*
 
+- `docker` input is deprecated in favour `container`. {pull}12162[12162]
+
 *Heartbeat*
 
 *Journalbeat*

_beats/libbeat/_meta/fields.ecs.yml

@@ -1,5 +1,5 @@
 # WARNING! Do not edit this file directly, it was generated by the ECS project,
-# based on ECS version 1.0.0.
+# based on ECS version 1.0.1.
 # Please visit https://github.com/elastic/ecs to suggest changes to ECS fields.
 
 - key: ecs
@@ -121,7 +121,7 @@
       or requestor in the network transaction. Some systems use the term "originator"
       to refer the client in TCP connections. The client fields describe details about
       the system acting as the client in the network event. Client fields are usually
-      populated in conjunction with server fields.  Client fields are generally not
+      populated in conjunction with server fields. Client fields are generally not
       populated for packet-level events.
 
       Client / server representations can add semantic context to an exchange, which
@@ -223,6 +223,7 @@
     - name: port
       level: core
       type: long
+      format: string
       description: Port of the client.
     - name: user.email
       level: extended
@@ -463,6 +464,7 @@
     - name: port
       level: core
       type: long
+      format: string
       description: Port of the destination.
     - name: user.email
       level: extended
@@ -693,6 +695,7 @@
     - name: severity
       level: core
       type: long
+      format: string
       description: Severity describes the original severity of the event. What the
         different severity values mean can very different between use cases. It's
         up to the implementer to make sure severities are consistent across events.
@@ -1132,6 +1135,7 @@
     - name: response.status_code
       level: extended
       type: long
+      format: string
       description: HTTP response status code.
       example: 404
     - name: version
@@ -1523,11 +1527,15 @@
     - name: pid
       level: core
       type: long
+      format: string
       description: Process id.
+      example: 4242
     - name: ppid
       level: extended
       type: long
-      description: Process parent id.
+      format: string
+      description: Parent process' pid.
+      example: 4241
     - name: start
       level: extended
       type: date
@@ -1536,6 +1544,7 @@
     - name: thread.id
       level: extended
       type: long
+      format: string
       description: Thread ID.
       example: 4242
     - name: title
@@ -1685,6 +1694,7 @@
     - name: port
       level: core
       type: long
+      format: string
       description: Port of the server.
     - name: user.email
       level: extended
@@ -1899,6 +1909,7 @@
     - name: port
       level: core
       type: long
+      format: string
       description: Port of the source.
     - name: user.email
       level: extended
@@ -1996,6 +2007,7 @@
     - name: port
       level: extended
       type: long
+      format: string
       description: Port of the request, such as 443.
       example: 443
     - name: query

_beats/libbeat/processors/add_cloud_metadata/_meta/fields.yml

@@ -8,6 +8,11 @@
       example: project-x
       description: >
         Name of the project in Google Cloud.
+
+    - name: cloud.image.id
+      example: ami-abcd1234
+      description: >
+        Image ID for the cloud instance.
 
     # Alias for old fields
     - name: meta.cloud.provider
@@ -44,3 +49,5 @@
       type: alias
       path: cloud.region
       migration: true
+
+

_beats/libbeat/processors/add_host_metadata/_meta/fields.yml

@@ -21,3 +21,9 @@
           example: "18D109"
           description: >
             OS build information.
+
+        - name: os.codename
+          type: keyword
+          example: "stretch"
+          description: >
+            OS codename, if any.

_beats/libbeat/processors/script/javascript/beatevent_v0_test.go

@@ -19,7 +19,6 @@ package javascript
 
 import (
 	"fmt"
-	"runtime"
 	"strings"
 	"testing"
 	"time"
@@ -29,6 +28,7 @@ import (
 	"github.com/elastic/beats/libbeat/beat"
 	"github.com/elastic/beats/libbeat/common"
 	"github.com/elastic/beats/libbeat/monitoring"
+	"github.com/elastic/beats/libbeat/tests/resources"
 )
 
 const (
@@ -206,15 +206,8 @@ func TestBeatEventV0(t *testing.T) {
 }
 
 func BenchmarkBeatEventV0(b *testing.B) {
-	goroutinesAtStart := runtime.NumGoroutine()
-	defer func() {
-		// Sanity check that timers are not leaking goroutines.
-		goroutinesAtEnd := runtime.NumGoroutine()
-		if goroutinesAtEnd != goroutinesAtStart {
-			b.Errorf("Suspected goroutine leak: atStart=%d, atEnd=%d",
-				goroutinesAtStart, goroutinesAtEnd)
-		}
-	}()
+	goroutines := resources.NewGoroutinesChecker()
+	defer goroutines.Check(b)
 
 	benchTest := func(tc testCase, timeout time.Duration) func(b *testing.B) {
 		return func(b *testing.B) {

_beats/libbeat/tests/system/beat/beat.py

@@ -221,8 +221,6 @@ def start_beat(self,
     def render_config_template(self, template_name=None,
                                output=None, **kargs):
 
-        print("render config")
-
         # Init defaults
         if template_name is None:
             template_name = self.beat_name
@@ -360,6 +358,18 @@ def get_log(self, logfile=None):
 
         return data
 
+    def get_log_lines(self, logfile=None):
+        """
+        Returns the log lines as a list of strings
+        """
+        if logfile is None:
+            logfile = self.beat_name + ".log"
+
+        with open(os.path.join(self.working_dir, logfile), 'r') as f:
+            data = f.readlines()
+
+        return data
+
     def wait_log_contains(self, msg, logfile=None,
                           max_timeout=10, poll_interval=0.1,
                           name="log_contains",

_beats/testing/environments/latest.yml

@@ -3,7 +3,7 @@
 version: '2.3'
 services:
   elasticsearch:
-    image: docker.elastic.co/elasticsearch/elasticsearch:6.6.0
+    image: docker.elastic.co/elasticsearch/elasticsearch:7.1.0
     healthcheck:
       test: ["CMD", "curl", "-f", "http://localhost:9200"]
       retries: 300
@@ -16,7 +16,7 @@ services:
       - "xpack.security.enabled=false"
 
   logstash:
-    image: docker.elastic.co/logstash/logstash:6.6.0
+    image: docker.elastic.co/logstash/logstash:7.1.0
     healthcheck:
       test: ["CMD", "curl", "-f", "http://localhost:9600/_node/stats"]
       retries: 300
@@ -26,7 +26,7 @@ services:
       - ./docker/logstash/pki:/etc/pki:ro
 
   kibana:
-    image: docker.elastic.co/kibana/kibana:6.6.0
+    image: docker.elastic.co/kibana/kibana:7.1.0
     healthcheck:
       test: ["CMD", "curl", "-f", "http://localhost:5601"]
       retries: 300