Add support for hints' based autodiscovery in kubernetes provider

[ChrsMark]

What does this PR do?

Elastic Agent implementation to support hints based autodiscovery. See #662 and #613 for more details

Why is it important?

In order to support #613

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

• Feature Implementation
• Add new unit tests for the new functionality

How to test this PR locally

1. Deploy a target redis pod:

apiVersion: v1
kind: Pod
metadata:
  name: redis
  annotations:
    co.elastic.hints/package: redis
    co.elastic.hints/data_streams: info, key
    co.elastic.hints/host: '${kubernetes.pod.ip}:6379'
    co.elastic.hints/info.period: 1m
    co.elastic.hints/key.period: 10m
  labels:
    k8s-app: redis
    app: redis
spec:
  containers:
  - image: redis
    imagePullPolicy: IfNotPresent
    name: redis
    ports:
    - name: redis
      containerPort: 6379
      protocol: TCP

2. Configure Agent properly

elastic-agent.yml

providers:
  kubernetes:
    kube_config: /home/chrismark/.kube/config
    node: "kind-control-plane"
    hints.enabled: true
inputs:
  - name: filestream-redis
    type: filestream
    use_output: default
    streams:
      - condition: ${kubernetes.hints.redis.log.enabled} == true or ${kubernetes.hints.redis.enabled} == true
        data_stream:
          dataset: redis.log
          type: logs
        exclude_files:
          - .gz$
        exclude_lines:
          - ^\s+[\-`('.|_]
        parsers:
          - container:
              format: auto
              stream: ${kubernetes.hints.redis.log.stream|'all'}
        paths:
          - /var/log/containers/*${kubernetes.hints.container_id}.log
        prospector:
          scanner:
            symlinks: true
        tags:
          - redis-log
    data_stream.namespace: default
  - name: redis-redis
    type: redis
    use_output: default
    streams:
      - condition: ${kubernetes.hints.redis.slowlog.enabled} == true or ${kubernetes.hints.redis.enabled} == true
        data_stream:
          dataset: redis.slowlog
          type: logs
        hosts:
          - ${kubernetes.hints.redis.slowlog.host|'127.0.0.1:6379'}
        password: ${kubernetes.hints.redis.slowlog.password|''}
    data_stream.namespace: default
  - name: redis/metrics-redis
    type: redis/metrics
    use_output: default
    streams:
      - condition: ${kubernetes.hints.redis.info.enabled} == true or ${kubernetes.hints.redis.enabled} == true
        data_stream:
          dataset: redis.info
          type: metrics
        hosts:
          - ${kubernetes.hints.redis.info.host|'127.0.0.1:6379'}
        idle_timeout: 20s
        maxconn: 10
        metricsets:
          - info
        network: tcp
        password: ${kubernetes.hints.redis.info.password|''}
        period: ${kubernetes.hints.redis.info.period|'10s'}
      - condition: ${kubernetes.hints.redis.key.enabled} == true or ${kubernetes.hints.redis.enabled} == true
        data_stream:
          dataset: redis.key
          type: metrics
        hosts:
          - ${kubernetes.hints.redis.key.host|'127.0.0.1:6379'}
        idle_timeout: 20s
        key.patterns:
          - limit: 20
            pattern: '*'
        maxconn: 10
        metricsets:
          - key
        network: tcp
        password: ${kubernetes.hints.redis.key.password|''}
        period: ${kubernetes.hints.redis.key.period|'10s'}
      - condition: ${kubernetes.hints.redis.keyspace.enabled} == true or ${kubernetes.hints.redis.enabled} == true
        data_stream:
          dataset: redis.keyspace
          type: metrics
        hosts:
          - ${kubernetes.hints.redis.keyspace.host|'127.0.0.1:6379'}
        idle_timeout: 20s
        maxconn: 10
        metricsets:
          - keyspace
        network: tcp
        password: ${kubernetes.hints.redis.keyspace.password|''}
        period: ${kubernetes.hints.redis.keyspace.period|'10s'}
    data_stream.namespace: default

3. Run inspect command and verify that the configuration is populated properly for the redis Pod
   ./elastic-agent inspect output -o default -p -v -d "*" -c elastic-agent.yml

Example output

[default] filebeat:
filebeat:
  inputs:
  - hosts:
    - 127.0.0.1:6379
    index: logs-redis.slowlog-default
    name: redis-redis
    password: ""
    processors:
    - add_fields:
        fields:
          dataset: redis.slowlog
          namespace: default
          type: logs
        target: data_stream
    - add_fields:
        fields:
          dataset: redis.slowlog
        target: event
    - add_fields:
        fields:
          id: ""
          snapshot: false
          version: 8.5.0
        target: elastic_agent
    - add_fields:
        fields:
          id: ""
        target: agent
    type: redis
  - exclude_files:
    - .gz$
    exclude_lines:
    - ^\s+[\-`('.|_]
    index: logs-redis.log-default
    name: filestream-redis
    parsers:
    - container:
        format: auto
        stream: all
    paths:
    - /var/log/containers/*f5fe0acafa56d55291aec1df632cb33c769c7ad2a1391bfd0abe884f554dfb5e.log
    processors:
    - add_fields:
        fields:
          id: f5fe0acafa56d55291aec1df632cb33c769c7ad2a1391bfd0abe884f554dfb5e
          image:
            name: redis
          runtime: containerd
        target: container
    - add_fields:
        fields:
          container:
            name: redis
          labels:
            app: redis
            k8s-app: redis
          namespace: default
          namespace_labels:
            kubernetes_io/metadata_name: default
          namespace_uid: a22a6d0a-8687-43cd-837c-b567c3e1fac2
          node:
            hostname: kind-control-plane
            labels:
              beta_kubernetes_io/arch: amd64
              beta_kubernetes_io/os: linux
              kubernetes_io/arch: amd64
              kubernetes_io/hostname: kind-control-plane
              kubernetes_io/os: linux
              node-role_kubernetes_io/control-plane: ""
              node_kubernetes_io/exclude-from-external-load-balancers: ""
            name: kind-control-plane
            uid: 42fe39bb-19dd-4382-85be-c8c1d67ab742
          pod:
            ip: 10.244.0.2
            name: redis
            uid: ad62ad95-fa2f-4884-9ba1-7bc34080b860
        target: kubernetes
    - add_fields:
        fields:
          cluster:
            name: kind-kind
            url: https://127.0.0.1:42447
        target: orchestrator
    - add_fields:
        fields:
          dataset: redis.log
          namespace: default
          type: logs
        target: data_stream
    - add_fields:
        fields:
          dataset: redis.log
        target: event
    - add_fields:
        fields:
          id: ""
          snapshot: false
          version: 8.5.0
        target: elastic_agent
    - add_fields:
        fields:
          id: ""
        target: agent
    prospector:
      scanner:
        symlinks: true
    tags:
    - redis-log
    type: filestream
output:
  elasticsearch:
    api-key: example-key
    hosts:
    - 127.0.0.1:9200
---[default] metricbeat:
metricbeat:
  modules:
  - hosts:
    - 10.244.0.2:6379
    idle_timeout: 20s
    index: metrics-redis.info-default
    maxconn: 10
    metricsets:
    - info
    module: redis
    name: redis/metrics-redis
    network: tcp
    password: ""
    period: 1m
    processors:
    - add_fields:
        fields:
          labels:
            app: redis
            k8s-app: redis
          namespace: default
          namespace_labels:
            kubernetes_io/metadata_name: default
          namespace_uid: a22a6d0a-8687-43cd-837c-b567c3e1fac2
          node:
            hostname: kind-control-plane
            labels:
              beta_kubernetes_io/arch: amd64
              beta_kubernetes_io/os: linux
              kubernetes_io/arch: amd64
              kubernetes_io/hostname: kind-control-plane
              kubernetes_io/os: linux
              node-role_kubernetes_io/control-plane: ""
              node_kubernetes_io/exclude-from-external-load-balancers: ""
            name: kind-control-plane
            uid: 42fe39bb-19dd-4382-85be-c8c1d67ab742
          pod:
            ip: 10.244.0.2
            name: redis
            uid: ad62ad95-fa2f-4884-9ba1-7bc34080b860
        target: kubernetes
    - add_fields:
        fields:
          cluster:
            name: kind-kind
            url: https://127.0.0.1:42447
        target: orchestrator
    - add_fields:
        fields:
          dataset: redis.info
          namespace: default
          type: metrics
        target: data_stream
    - add_fields:
        fields:
          dataset: redis.info
        target: event
    - add_fields:
        fields:
          id: ""
          snapshot: false
          version: 8.5.0
        target: elastic_agent
    - add_fields:
        fields:
          id: ""
        target: agent
  - hosts:
    - 10.244.0.2:6379
    idle_timeout: 20s
    index: metrics-redis.key-default
    key.patterns:
    - limit: 20
      pattern: '*'
    maxconn: 10
    metricsets:
    - key
    module: redis
    name: redis/metrics-redis
    network: tcp
    password: ""
    period: 10m
    processors:
    - add_fields:
        fields:
          labels:
            app: redis
            k8s-app: redis
          namespace: default
          namespace_labels:
            kubernetes_io/metadata_name: default
          namespace_uid: a22a6d0a-8687-43cd-837c-b567c3e1fac2
          node:
            hostname: kind-control-plane
            labels:
              beta_kubernetes_io/arch: amd64
              beta_kubernetes_io/os: linux
              kubernetes_io/arch: amd64
              kubernetes_io/hostname: kind-control-plane
              kubernetes_io/os: linux
              node-role_kubernetes_io/control-plane: ""
              node_kubernetes_io/exclude-from-external-load-balancers: ""
            name: kind-control-plane
            uid: 42fe39bb-19dd-4382-85be-c8c1d67ab742
          pod:
            ip: 10.244.0.2
            name: redis
            uid: ad62ad95-fa2f-4884-9ba1-7bc34080b860
        target: kubernetes
    - add_fields:
        fields:
          cluster:
            name: kind-kind
            url: https://127.0.0.1:42447
        target: orchestrator
    - add_fields:
        fields:
          dataset: redis.key
          namespace: default
          type: metrics
        target: data_stream
    - add_fields:
        fields:
          dataset: redis.key
        target: event
    - add_fields:
        fields:
          id: ""
          snapshot: false
          version: 8.5.0
        target: elastic_agent
    - add_fields:
        fields:
          id: ""
        target: agent
  - hosts:
    - 127.0.0.1:6379
    idle_timeout: 20s
    index: metrics-redis.keyspace-default
    maxconn: 10
    metricsets:
    - keyspace
    module: redis
    name: redis/metrics-redis
    network: tcp
    password: ""
    period: 10s
    processors:
    - add_fields:
        fields:
          labels:
            app: redis
            k8s-app: redis
          namespace: default
          namespace_labels:
            kubernetes_io/metadata_name: default
          namespace_uid: a22a6d0a-8687-43cd-837c-b567c3e1fac2
          node:
            hostname: kind-control-plane
            labels:
              beta_kubernetes_io/arch: amd64
              beta_kubernetes_io/os: linux
              kubernetes_io/arch: amd64
              kubernetes_io/hostname: kind-control-plane
              kubernetes_io/os: linux
              node-role_kubernetes_io/control-plane: ""
              node_kubernetes_io/exclude-from-external-load-balancers: ""
            name: kind-control-plane
            uid: 42fe39bb-19dd-4382-85be-c8c1d67ab742
          pod:
            ip: 10.244.0.2
            name: redis
            uid: ad62ad95-fa2f-4884-9ba1-7bc34080b860
        target: kubernetes
    - add_fields:
        fields:
          cluster:
            name: kind-kind
            url: https://127.0.0.1:42447
        target: orchestrator
    - add_fields:
        fields:
          dataset: redis.keyspace
          namespace: default
          type: metrics
        target: data_stream
    - add_fields:
        fields:
          dataset: redis.keyspace
        target: event
    - add_fields:
        fields:
          id: ""
          snapshot: false
          version: 8.5.0
        target: elastic_agent
    - add_fields:
        fields:
          id: ""
        target: agent
output:
  elasticsearch:
    api-key: example-key
    hosts:
    - 127.0.0.1:9200
---[default] FLEET_MONITORING:
monitoring_checksum: d4f14553cdd37ff68bb492de0eea7256
output:
  elasticsearch:
    api-key: example-key
    hosts:
    - 127.0.0.1:9200
    type: elasticsearch
programs:
- filebeat
- metricbeat
---% 

Extra:

Test with various container logs' streams

1. Deploy apache

apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpd-deployment-xfusion
  annotations:
    co.elastic.hints/package: apache
    co.elastic.hints/data_streams: access, error
    #co.elastic.hints/access.stream: stdout
    #co.elastic.hints/error.stream: stderr
  labels:
    app: httpd_app_xfusion
spec :
  replicas: 1
  selector:
    matchLabels:
      app: httpd_app_xfusion
  template:
    metadata:
      annotations:
        co.elastic.hints/package: apache
        co.elastic.hints/data_streams: access, error
        co.elastic.hints/access.stream: stdout
        co.elastic.hints/error.stream: stderr
      labels:
        hints: parse
        app: httpd_app_xfusion
    spec :
      containers:
        - name: httpd-container-xfusion
          image: httpd:latest
          ports:
            - containerPort : 80
---
apiVersion: v1
kind: Service
metadata:
  name: httpd-service-xfusion
spec:
  type: ClusterIP
  selector:
    app: httpd_app_xfusion
  ports:
    - port: 8086
      targetPort: 80

Template for apache

inputs:
  - name: filestream-apache
    type: filestream
    use_output: default
    streams:
      - condition: ${kubernetes.hints.apache.access.enabled} == true or ${kubernetes.hints.apache.enabled} == true
        data_stream:
          dataset: apache.access
          type: logs
        exclude_files:
          - .gz$
        parsers:
          - container:
              format: auto
              stream: ${kubernetes.hints.apache.access.stream|'all'}
        paths:
          - /var/log/containers/*${kubernetes.hints.container_id}.log
        prospector:
          scanner:
            symlinks: true
        tags:
          - apache-access
      - condition: ${kubernetes.hints.apache.error.enabled} == true or ${kubernetes.hints.apache.enabled} == true
        data_stream:
          dataset: apache.error
          type: logs
        exclude_files:
          - .gz$
        parsers:
          - container:
              format: auto
              stream: ${kubernetes.hints.apache.error.stream|'all'}
        paths:
          - /var/log/containers/*${kubernetes.hints.container_id}.log
        processors:
          - add_locale: null
        prospector:
          scanner:
            symlinks: true
        tags:
          - apache-error
    data_stream.namespace: default

3. Verify that streams are assigned properly per data_stream like:

...
name: filestream-apache
    parsers:
    - container:
        format: auto
        stream: stderr
    paths:
    - /var/log/containers/*a5289bb2da8960516a5b3d5c3a484ececc349e7c9a2118209f3c145f73e7cbb3.log
...
name: filestream-apache
    parsers:
    - container:
        format: auto
        stream: stdout
    paths:
    - /var/log/containers/*a5289bb2da8960516a5b3d5c3a484ececc349e7c9a2118209f3c145f73e7cbb3.log

Default container stream

prometheus.yml

inputs:
  - name: filestream-prometheus
    type: filestream
    use_output: default
    streams:
      - condition: ${kubernetes.hints.prometheus.container_logs.enabled} == true
        data_stream:
          dataset: kubernetes.container_logs
          type: logs
        exclude_files: []
        exclude_lines: []
        parsers:
          - container:
              format: auto
              stream: all
        paths:
          - /var/log/containers/*${kubernetes.hints.container_id}.log
        prospector:
          scanner:
            symlinks: true
        tags: []
    data_stream.namespace: default
  - name: prometheus/metrics-prometheus
    type: prometheus/metrics
    use_output: default
    streams:
      - bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        condition: ${kubernetes.hints.prometheus.collector.enabled} == true or ${kubernetes.hints.prometheus.enabled} == true
        data_stream:
          dataset: prometheus.collector
          type: metrics
        hosts:
          - ${kubernetes.hints.prometheus.collector.host|'localhost:9090'}
        metrics_filters.exclude: null
        metrics_filters.include: null
        metrics_path: /metrics
        metricsets:
          - collector
        password: ${kubernetes.hints.prometheus.collector.password|'secret'}
        period: ${kubernetes.hints.prometheus.collector.period|'10s'}
        rate_counters: true
        ssl.certificate_authorities:
          - /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt
        use_types: true
        username: ${kubernetes.hints.prometheus.collector.username|'user'}

Related issues

Implements part 1 of #662

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b hints_based_autodiscovery_mappings upstream/hints_based_autodiscovery_mappings
git merge upstream/main
git push upstream hints_based_autodiscovery_mappings

[mergify]

This pull request does not have a backport label. Could you fix it @ChrsMark? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-v./d./d./d is the label to automatically backport to the 8./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2022-09-14T07:08:29.819+0000

• Duration: 18 min 2 sec

Test stats 🧪

Test	Results
Failed	0
Passed	4935
Skipped	17
Total	4952

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages.

• run integration tests : Run the Elastic Agent Integration tests.

• run end-to-end tests : Generate the packages and run the E2E Tests.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[ChrsMark]

@blakerouse does this interact with the changes on the feature-arch-v2 branch at all? I don't think providers changed at all but I want to double check.

@ChrsMark FYI we have re-implemented large portions of the agent internals related to starting and controlling processes on a separate feature branch while we migrate the affected inputs/beats to the new APIs. My hope is this change is mostly unaffected by what is on that branch assuming you want this merged for 8.5.

Thanks @cmacknz for raising this. This PR only extends the existent provider's mechanism so I hope we are safe here. Let's wait for @blakerouse to confirm though:) .

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b hints_based_autodiscovery_mappings upstream/hints_based_autodiscovery_mappings
git merge upstream/main
git push upstream hints_based_autodiscovery_mappings

[gizas]

Put a comment here what the managed boolean flag provides. Repeat it in all rest of references

[gizas]

Why is this used?

[ChrsMark]

Within the hint's string you might have sth like ${kubernetes.pod.ip} which you want to be retrieved dynamically from the Pod's meta. So here we have a regexp to extract the actual value of the variable we target out of a "${kubernetes.*}" hint.

To be more specific imagine that you have a hint like '${kubernetes.pod.ip}:6379', then you want to replace the ${kubernetes.pod.ip} part with value from the Pod's metadata so as to have sth like 10.28.90.345:6379.

[ChrsMark]

added this as comment within the code too;) : 8b50e76

[gizas]

Do you like it like this? Maybe inlined?

[blakerouse]

This should work with v2 just fine. Nothing changed in the providers or the variable substitution.

[blakerouse]

Shouldn't this be debug level?

[blakerouse]

Shouldn't this be debug level?

[elasticmachine]

🌐 Coverage report

Name	Metrics % (covered/total)	Diff
Packages	97.333% (73/75)	👍
Files	68.085% (160/235)	👍 0.564
Classes	68.478% (315/460)	👍 0.356
Methods	52.826% (888/1681)	👍 0.631
Lines	38.879% (9556/24579)	👍 0.373
Conditionals	100.0% (0/0)	💚

[ChrsMark]

/test