Filebeat envoyproxy module

CHANGELOG.next.asciidoc

@@ -132,6 +132,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - New module for Cisco ASA logs. {issue}9200[9200] {pull}11171[11171]
 - Added support for Cisco ASA fields to the netflow input. {pull}11201[11201]
 - Configurable line terminator. {pull}11015[11015]
+- Add Filebeat envoyproxy module. {pull}11700[11700]
 
 *Heartbeat*
 

filebeat/docs/fields.asciidoc

@@ -21,6 +21,7 @@ grouped in the following categories:
 * <<exported-fields-docker-processor>>
 * <<exported-fields-ecs>>
 * <<exported-fields-elasticsearch>>
+* <<exported-fields-envoyproxy>>
 * <<exported-fields-haproxy>>
 * <<exported-fields-host-processor>>
 * <<exported-fields-icinga>>
@@ -5355,6 +5356,82 @@ example: doc
 
 Type
 
+--
+
+[[exported-fields-envoyproxy]]
+== Envoyproxy fields
+
+Module for handling logs produced by envoy
+
+
+
+[float]
+== envoyproxy fields
+
+Fields from envoy proxy logs after normalization
+
+
+
+*`envoyproxy.log_type`*::
++
+--
+type: keyword
+
+Envoy log type, normally ACCESS
+
+
+--
+
+*`envoyproxy.response_flags`*::
++
+--
+type: keyword
+
+Response flags
+
+
+--
+
+*`envoyproxy.upstream_service_time`*::
++
+--
+type: long
+
+format: duration
+
+Upstream service time in nanoseconds
+
+
+--
+
+*`envoyproxy.request_id`*::
++
+--
+type: keyword
+
+ID of the request
+
+
+--
+
+*`envoyproxy.authority`*::
++
+--
+type: keyword
+
+Envoy proxy authority field
+
+
+--
+
+*`envoyproxy.proxy_type`*::
++
+--
+type: keyword
+
+Envoy proxy type, tcp or http
+
+
 --
 
 [[exported-fields-haproxy]]

filebeat/docs/modules/envoyproxy.asciidoc

@@ -0,0 +1,35 @@
+////
+This file is generated! See scripts/docs_collector.py
+////
+
+[[filebeat-module-envoyproxy]]
+[role="xpack"]
+
+:modulename: envoyproxy
+:has-dashboards: true
+
+== Envoyproxy Module
+
+This is a filebeat module for Envoy proxy access log (https://www.envoyproxy.io/docs/envoy/v1.10.0/configuration/access_log). It supports both standalone deployment and Envoy proxy deployment in Kubernetes. 
+
+[float]
+=== Compatibility
+
+Although this module has been developed against Envoy proxy 1.10.0 and Kubernetes v1.13.x, it is expected to work
+with other versions of Envoy proxy and Kubernetes.
+
+[float]
+=== Example dashboard
+
+This module comes with a sample dashboard.
+
+[role="screenshot"]
+image::./images/kibana-envoyproxy.jpg[]
+
+
+[float]
+=== Fields
+
+For a description of each field in the module, see the
+<<exported-fields-envoyproxy,exported fields>> section.
+

filebeat/docs/modules_list.asciidoc

@@ -8,6 +8,7 @@ This file is generated! See scripts/docs_collector.py
   * <<filebeat-module-cisco>>
   * <<filebeat-module-coredns>>
   * <<filebeat-module-elasticsearch>>
+  * <<filebeat-module-envoyproxy>>
   * <<filebeat-module-haproxy>>
   * <<filebeat-module-icinga>>
   * <<filebeat-module-iis>>
@@ -38,6 +39,7 @@ include::modules/auditd.asciidoc[]
 include::modules/cisco.asciidoc[]
 include::modules/coredns.asciidoc[]
 include::modules/elasticsearch.asciidoc[]
+include::modules/envoyproxy.asciidoc[]
 include::modules/haproxy.asciidoc[]
 include::modules/icinga.asciidoc[]
 include::modules/iis.asciidoc[]

x-pack/filebeat/filebeat.reference.yml

@@ -156,6 +156,16 @@ filebeat.modules:
     #var.convert_timezone: false
 
 
+#------------------------------ Envoyproxy Module ------------------------------
+- module: envoyproxy
+  # Fileset for native deployment
+  log: 
+    enabled: true
+
+    # Set custom paths for the log files. If left empty,
+    # Filebeat will choose the paths depending on your OS.
+    #var.paths:
+
 #------------------------------- Haproxy Module -------------------------------
 - module: haproxy
   # All logs

x-pack/filebeat/module/envoyproxy/README.md

@@ -0,0 +1,124 @@
+# Envoyproxy Module
+
+This is a filebeat module for Envoy proxy access log. 
+
+## Caveats
+
+* Module is to be considered _beta_.
+
+## Download and install Filebeat
+
+Grab the filebeat binary from elastic.co, and install it by following the instructions.
+
+## Deployment Scenario #1: envoy native deployment
+
+This module assumes that envoy log entries will be written to /var/log/envoy.log. Should it be not the case, please point the module log path to the path of the log file. 
+
+Update filebeat.yml to point to Elasticsearch and Kibana. 
+Setup Filebeat.
+```
+./filebeat setup --modules envoyproxy -e
+```
+
+Enable the Filebeat envoyproxy module
+```
+./filebeat modules enable envoyproxy
+```
+
+Start Filebeat
+```
+./filebeat -e
+```
+
+Now, the Envoy logs and dashboard should appear in Kibana.
+
+
+## Deployment Scenario #2: envoy for kubernetes 
+
+For Kubernetes deployment, the filebeat daemon-set yaml file needs to be deployed to the Kubernetes cluster. Sample configuration files is provided under the `beats/deploy/filebeat` directory (https://github.com/elastic/beats/tree/master/deploy/kubernetes/filebeat), and can be deployed by doing the following:
+```
+kubectl apply -f filebeat
+```
+
+#### Note the following section in the ConfigMap, make changes to the yaml file if necessary
+```
+  filebeat.autodiscover:
+    providers:
+      - type: kubernetes
+        hints.enabled: true
+        default.disable: true
+
+  processors:
+    - add_kubernetes_metadata:
+        in_cluster: true
+```
+
+This enables auto-discovery and hints for filebeat. When default.disable is set to true (default value is false), it will disable log harvesting for the pod/container, unless it has specific annotations enabled. This gives users more granular control on kubernetes log ingestion. The `add_kubernetes_metadata` processor will add enrichment data for Kubernetes to the ingest logs.
+
+#### Note the following section in the DaemonSet, make changes to the yaml file if necessary
+```
+apiVersion: extensions/v1beta1
+kind: DaemonSet
+metadata:
+  name: filebeat
+  namespace: kube-system
+  labels:
+    k8s-app: filebeat
+spec:
+  template:
+    metadata:
+      labels:
+        k8s-app: filebeat
+    spec:
+      serviceAccountName: filebeat
+      terminationGracePeriodSeconds: 30
+      containers:
+      - name: filebeat
+        image: docker.elastic.co/beats/filebeat:%VERSION%
+        args: [
+          "sh", "-c", "filebeat setup -e --modules envoyproxy -c /etc/filebeat.yml && filebeat -e -c /etc/filebeat.yml"
+        ]
+        env:
+        # Edit the following values to reflect your setup accordingly
+        - name: ELASTICSEARCH_HOST
+          value: 192.168.99.1
+        - name: ELASTICSEARCH_USERNAME
+          value: elastic
+        - name: ELASTICSEARCH_PASSWORD
+          value: changeme
+        - name: KIBANA_HOST
+          value: 192.168.99.1
+```
+
+The module setup step can also be done separately without Kubernetes if applicable, and in that case, the args can be simplified to:
+```
+        args: [
+          "sh", "-c", "filebeat -e -c /etc/filebeat.yml"
+        ]
+```
+
+#### Sample Deployment for envoy, using ambassador as an example. Note the annotations.
+
+```
+apiVersion: extensions/v1beta1
+kind: Deployment
+metadata:
+  name: ambassador
+spec:
+  replicas: 3
+  template:
+    metadata:
+      annotations:
+        "co.elastic.logs/module": "envoyproxy"
+        "co.elastic.logs/fileset": "log"
+        "co.elastic.logs/disable": "false"
+      labels:
+        service: ambassador
+    spec:
+      serviceAccountName: ambassador
+      containers:
+      - name: ambassador
+        image: quay.io/datawire/ambassador:0.50.0
+      <snipped>
+```
+

x-pack/filebeat/module/envoyproxy/_meta/config.yml

@@ -0,0 +1,8 @@
+- module: envoyproxy
+  # Fileset for native deployment
+  log: 
+    enabled: true
+
+    # Set custom paths for the log files. If left empty,
+    # Filebeat will choose the paths depending on your OS.
+    #var.paths:

x-pack/filebeat/module/envoyproxy/_meta/docs.asciidoc

@@ -0,0 +1,22 @@
+[role="xpack"]
+
+:modulename: envoyproxy
+:has-dashboards: true
+
+== Envoyproxy Module
+
+This is a filebeat module for Envoy proxy access log (https://www.envoyproxy.io/docs/envoy/v1.10.0/configuration/access_log). It supports both standalone deployment and Envoy proxy deployment in Kubernetes. 
+
+[float]
+=== Compatibility
+
+Although this module has been developed against Envoy proxy 1.10.0 and Kubernetes v1.13.x, it is expected to work
+with other versions of Envoy proxy and Kubernetes.
+
+[float]
+=== Example dashboard
+
+This module comes with a sample dashboard.
+
+[role="screenshot"]
+image::./images/kibana-envoyproxy.jpg[]

x-pack/filebeat/module/envoyproxy/_meta/fields.yml

@@ -0,0 +1,45 @@
+- key: envoyproxy
+  title: Envoyproxy
+  description: >
+    Module for handling logs produced by envoy
+  fields:
+  - name: envoyproxy
+    type: group
+    description: >
+      Fields from envoy proxy logs after normalization
+    fields:
+    - name: log_type
+      type: keyword
+      description: >
+        Envoy log type, normally ACCESS
+
+    - name: response_flags
+      type: keyword
+      description: >
+        Response flags
+
+    - name: upstream_service_time
+      type: long
+      format: duration
+      input_format: nanoseconds
+      description: >
+        Upstream service time in nanoseconds
+
+    - name: request_id
+      type: keyword
+      description: >
+        ID of the request
+
+    - name: authority
+      type: keyword
+      description: >
+        Envoy proxy authority field
+
+    - name: proxy_type
+      type: keyword
+      description: >
+        Envoy proxy type, tcp or http
+
+
+
+