Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add support for hints' based autodiscovery in kubernetes provider #662

Closed
3 tasks done
Tracked by #613
ChrsMark opened this issue Jul 4, 2022 · 5 comments
Closed
3 tasks done
Tracked by #613

Add support for hints' based autodiscovery in kubernetes provider #662

ChrsMark opened this issue Jul 4, 2022 · 5 comments
Assignees
@ChrsMark
Labels
8.5-candidate Team:Cloudnative-Monitoring Label for the Cloud Native Monitoring team

Comments

@ChrsMark
Copy link
Member

ChrsMark commented Jul 4, 2022 •
edited
Loading

As discussed at #613 (comment), kubernetes provider can be enhanced in order to support hint's based autodiscovery.

This can be achieved using the already supported dynamic variable resolution mechanism by constructing and emitting hint's specific mappings similar to elastic-agent/internal/pkg/composable/providers/kubernetes/pod.go.

Find below a full example of how we can reach from k8s annotations to an Agent input.

Hint's as annotations

annotations:
  co.elastic.hints/package: redis
  co.elastic.hints/data_streams: info, key
  co.elastic.hints/host: '${kubernetes.pod.ip}:6379'
  co.elastic.hints/info.period: 1m
  co.elastic.hints/key.period: 10m

Emitted mapping by k8s provider

{
  "kubernetes": {
    "hints": {
      "redis": {
        "info": {
          "enabled": true,
          "period": "1m",
          "host": "152.10.67.976:6379"
        },
        "key": {
          "enabled": true,
          "period": "10m",
          "host": "152.10.67.976:6379"
        }
      }
    }
  }
}

Defined template constructed by Fleet UI

providers:
  kubernetes:
    kube_config: /Users/chrismark/.kube/config
    node: "kind-control-plane"
    hints.enabled: true

inputs:
  - name: templates.d/redis/0.3.6
    type: redis/metrics
    data_stream.namespace: default
    use_output: default
    streams:
      - data_stream:
          dataset: redis.info
          type: metrics
        metricsets:
         - info
        hosts:
          - "${kubernetes.hints.redis.info.host|kubernetes.hints.redis.host|'127.0.0.1:6379'}"
        idle_timeout: 20s
        maxconn: 10
        network: tcp
        period: "${kubernetes.hints.redis.info.period|kubernetes.hints.redis.period|'10s'}"
        condition: ${kubernetes.hints.redis.info.enabled} == true or ${kubernetes.hints.redis.enabled} == true //enabled even if only the integration is enabled, AKA enabled by default
      - data_stream:
          dataset: redis.key
          type: metrics
        metricsets:
          - key
        hosts:
          - "${kubernetes.hints.redis.key.host|kubernetes.hints.redis.host|'127.0.0.1'}:${kubernetes.hints.redis.info.port|'6379'}"
        idle_timeout: 20s
        key.patterns:
          - limit: 20
            pattern: '*'
        maxconn: 10
        network: tcp
        period: "${kubernetes.hints.redis.key.period|kubernetes.hints.redis.period|'10s'}"
        condition: ${kubernetes.hints.redis.key.enabled} == true and ${kubernetes.hints.redis.enabled} == true //enabled only if explicitely enabled, AKA disabled by default

The above will be provided as a ConfigMap mounted into the Agent Pod:

apiVersion: v1
kind: ConfigMap
metadata:
  name: elastic-agent-standalone-inputs
data:
  redis.yml: |-
    inputs:
     - name: templates.d/redis/0.3.6
       type: redis/metrics
       data_stream.namespace: default
       use_output: default
       streams:
         - data_stream:
             dataset: redis.info
             type: metrics
           metricsets:
             ...

The scope of this issue is summarised into 2 parts and can be implemented already regardless of Fleet's UI implementation part:

Known issues/dependencies:

Closes #274
@ChrsMark ChrsMark added the Team:Cloudnative-Monitoring Label for the Cloud Native Monitoring team label Jul 4, 2022
This was referenced Jul 4, 2022
Closed
Closed
@ChrsMark ChrsMark self-assigned this Jul 4, 2022
@ChrsMark
Copy link
Member Author

ChrsMark commented Jul 4, 2022 •
edited
Loading

It seems that with Metrics collection the current mechanism could cover our needs without issues with the proper additions. Made some tests to figure out how we will handle container logs and the results are good. My main concern here was about handling the container input providing the option to define stream like what we do in Filebeat with https://www.elastic.co/guide/en/beats/filebeat/current/configuration-autodiscover-hints.html#_co_elastic_logsfileset

For the testing I use an apache webserver as the target container and I label it properly so as to collect its logs using the existing kubernetes provider.

Sample ConfigMap for Agent:

agent-configmap.yml 
apiVersion: v1
kind: ConfigMap
metadata:
  name: agent-node-datastreams
  namespace: kube-system
  labels:
    k8s-app: elastic-agent-standalone
data:
  agent.yml: |-
    outputs:
      default:
        type: elasticsearch
        hosts:
          - >-
            ${ES_HOST}
        username: ${ES_USERNAME}
        password: ${ES_PASSWORD}
    agent:
      monitoring:
        enabled: true
        use_output: default
        logs: true
        metrics: true
    providers.kubernetes:
      node: ${NODE_NAME}
      scope: node
    inputs:
      - name: apache-1
        type: filestream
        use_output: default
        condition: ${kubernetes.labels.hints} == "parse"
        meta:
          package:
            name: apache
            version: 1.3.5
        data_stream:
          namespace: default
        streams:
          - data_stream:
              dataset: apache.access
              type: logs
            paths:
              - "/var/log/containers/*${kubernetes.container.id}.log"
            tags:
              - apache-access
            exclude_files:
              - .gz$  # Is this needed ???
            prospector.scanner.symlinks: true
            parsers:
                - container:
                  stream: stdout
                  format: auto
          - data_stream:
              dataset: apache.error
              type: logs
            paths:
              - "/var/log/containers/*${kubernetes.container.id}.log"
            exclude_files:
              - .gz$
            tags:
              - apache-error
            processors:
              - add_locale: null
            prospector.scanner.symlinks: true
            parsers:
              - container:
                stream: stderr # Here users would use co.elastic.hints/access.stream: stderr
                format: auto

Sample Apache:

apache.yml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpd-deployment-xfusion
  labels:
    app: httpd_app_xfusion
spec :
  replicas: 2
  selector:
    matchLabels:
      app: httpd_app_xfusion
  template:
    metadata:
      labels:
        hints: parse
        app: httpd_app_xfusion
    spec :
      containers:
        - name: httpd-container-xfusion
          image: httpd:latest
          ports:
            - containerPort : 80
---
apiVersion: v1
kind: Service
metadata:
  name: httpd-service-xfusion
spec:
  type: ClusterIP
  selector:
    app: httpd_app_xfusion
  ports:
    - port: 8086
      targetPort: 80

Then Elastic Agent starts collecting logs from the proper container stream and sends them to ES.
After installing the assets of the apache integration are manually installed based on https://www.elastic.co/guide/en/fleet/current/install-standalone-elastic-agent.html and https://www.elastic.co/guide/en/fleet/current/install-uninstall-integration-assets.html#install-integration-assets the logs are parsed using the proper pipeline. A sample document ingested is the following:

apache.doc.json 
{
  "_index": ".ds-logs-apache.access-default-2022.07.04-000001",
  "_id": "muNSyYEBdZFFax9TmqND",
  "_version": 1,
  "_score": 1,
  "_source": {
    "container": {
      "image": {
        "name": "httpd:latest"
      },
      "runtime": "containerd",
      "id": "ea36b297cd0d1e576eb358f93ec7a2489b9741daeea1ca0af233053b1f6b529a"
    },
    "kubernetes": {
      "container": {
        "name": "httpd-container-xfusion"
      },
      "node": {
        "uid": "f6f42049-3f43-427e-8b43-dbd2cecc1f46",
        "hostname": "kind-control-plane",
        "name": "kind-control-plane",
        "labels": {
          "node_kubernetes_io/exclude-from-external-load-balancers": "",
          "node-role_kubernetes_io/master": "",
          "kubernetes_io/hostname": "kind-control-plane",
          "node-role_kubernetes_io/control-plane": "",
          "beta_kubernetes_io/os": "linux",
          "kubernetes_io/arch": "amd64",
          "kubernetes_io/os": "linux",
          "beta_kubernetes_io/arch": "amd64"
        }
      },
      "pod": {
        "uid": "bbb0368f-91fe-4af2-9fda-05503d02bef3",
        "ip": "10.244.0.6",
        "name": "httpd-deployment-xfusion-5546996bfc-48znw"
      },
      "namespace": "default",
      "namespace_uid": "81609706-b8f8-4327-a981-f7a1e0805147",
      "replicaset": {
        "name": "httpd-deployment-xfusion-5546996bfc"
      },
      "namespace_labels": {
        "kubernetes_io/metadata_name": "default"
      },
      "labels": {
        "app": "httpd_app_xfusion",
        "hints": "parse",
        "pod-template-hash": "5546996bfc"
      },
      "deployment": {
        "name": "httpd-deployment-xfusion"
      }
    },
    "agent": {
      "name": "kind-control-plane",
      "id": "ded3358e-655a-45f3-adcf-618b068e90d5",
      "type": "filebeat",
      "ephemeral_id": "c619f972-34de-403d-ac1f-1dc188a952f5",
      "version": "8.3.0"
    },
    "log": {
      "file": {
        "path": "/var/log/containers/httpd-deployment-xfusion-5546996bfc-48znw_default_httpd-container-xfusion-ea36b297cd0d1e576eb358f93ec7a2489b9741daeea1ca0af233053b1f6b529a.log"
      },
      "offset": 2188
    },
    "elastic_agent": {
      "id": "ded3358e-655a-45f3-adcf-618b068e90d5",
      "version": "8.3.0",
      "snapshot": false
    },
    "source": {
      "address": "127.0.0.1",
      "ip": "127.0.0.1"
    },
    "apache": {
      "access": {}
    },
    "ecs": {
      "version": "1.12.0"
    },
    "stream": "stdout",
    "host": {
      "hostname": "kind-control-plane",
      "os": {
        "kernel": "4.9.184-linuxkit",
        "codename": "focal",
        "name": "Ubuntu",
        "family": "debian",
        "type": "linux",
        "version": "20.04.4 LTS (Focal Fossa)",
        "platform": "ubuntu"
      },
      "ip": [
        "10.244.0.1",
        "10.244.0.1",
        "10.244.0.1",
        "10.244.0.1",
        "172.18.0.2",
        "fc00:f853:ccd:e793::2",
        "fe80::42:acff:fe12:2",
        "10.244.0.1",
        "10.244.0.1"
      ],
      "containerized": true,
      "name": "kind-control-plane",
      "mac": [
        "02:42:ac:12:00:02",
        "2a:a1:f4:21:45:db",
        "4e:a6:5a:2c:bd:0e",
        "72:0a:5b:02:39:10",
        "b6:54:ec:dc:ae:fd",
        "be:e3:32:16:ff:7e",
        "fa:33:ca:2f:3c:d7"
      ],
      "architecture": "x86_64"
    },
    "event": {
      "agent_id_status": "auth_metadata_missing",
      "ingested": "2022-07-04T13:07:22Z",
      "kind": "event",
      "created": "2022-07-04T13:07:18.842Z",
      "category": "web",
      "dataset": "apache.access",
      "outcome": "success"
    },
    "url": {
      "path": "/",
      "original": "/"
    },
    "tags": [
      "apache-access"
    ],
    "input": {
      "type": "filestream"
    },
    "orchestrator": {
      "cluster": {
        "name": "kind",
        "url": "kind-control-plane:6443"
      }
    },
    "@timestamp": "2022-07-04T13:07:18.000Z",
    "data_stream": {
      "namespace": "default",
      "type": "logs",
      "dataset": "apache.access"
    },
    "http": {
      "request": {
        "method": "GET"
      },
      "response": {
        "status_code": 200,
        "body": {
          "bytes": 45
        }
      },
      "version": "1.1"
    },
    "user": {
      "name": "-"
    }
  },
  "fields": {
    "orchestrator.cluster.name": [
      "kind"
    ],
    "kubernetes.node.uid": [
      "f6f42049-3f43-427e-8b43-dbd2cecc1f46"
    ],
    "event.category": [
      "web"
    ],
    "elastic_agent.version": [
      "8.3.0"
    ],
    "kubernetes.namespace_uid": [
      "81609706-b8f8-4327-a981-f7a1e0805147"
    ],
    "kubernetes.deployment.name": [
      "httpd-deployment-xfusion"
    ],
    "host.os.name.text": [
      "Ubuntu"
    ],
    "host.hostname": [
      "kind-control-plane"
    ],
    "kubernetes.node.labels.kubernetes_io/os": [
      "linux"
    ],
    "host.mac": [
      "02:42:ac:12:00:02",
      "2a:a1:f4:21:45:db",
      "4e:a6:5a:2c:bd:0e",
      "72:0a:5b:02:39:10",
      "b6:54:ec:dc:ae:fd",
      "be:e3:32:16:ff:7e",
      "fa:33:ca:2f:3c:d7"
    ],
    "container.id": [
      "ea36b297cd0d1e576eb358f93ec7a2489b9741daeea1ca0af233053b1f6b529a"
    ],
    "kubernetes.node.labels.node-role_kubernetes_io/master": [
      ""
    ],
    "kubernetes.labels.pod-template-hash": [
      "5546996bfc"
    ],
    "container.image.name": [
      "httpd:latest"
    ],
    "http.request.method": [
      "GET"
    ],
    "kubernetes.labels.app": [
      "httpd_app_xfusion"
    ],
    "host.os.version": [
      "20.04.4 LTS (Focal Fossa)"
    ],
    "kubernetes.node.labels.beta_kubernetes_io/os": [
      "linux"
    ],
    "kubernetes.namespace": [
      "default"
    ],
    "host.os.name": [
      "Ubuntu"
    ],
    "source.ip": [
      "127.0.0.1"
    ],
    "agent.name": [
      "kind-control-plane"
    ],
    "host.name": [
      "kind-control-plane"
    ],
    "event.agent_id_status": [
      "auth_metadata_missing"
    ],
    "http.response.status_code": [
      200
    ],
    "http.version": [
      "1.1"
    ],
    "event.kind": [
      "event"
    ],
    "event.outcome": [
      "success"
    ],
    "host.os.type": [
      "linux"
    ],
    "input.type": [
      "filestream"
    ],
    "log.offset": [
      2188
    ],
    "data_stream.type": [
      "logs"
    ],
    "tags": [
      "apache-access"
    ],
    "host.architecture": [
      "x86_64"
    ],
    "container.runtime": [
      "containerd"
    ],
    "url.path": [
      "/"
    ],
    "agent.id": [
      "ded3358e-655a-45f3-adcf-618b068e90d5"
    ],
    "host.containerized": [
      true
    ],
    "ecs.version": [
      "1.12.0"
    ],
    "event.created": [
      "2022-07-04T13:07:18.842Z"
    ],
    "kubernetes.node.labels.node-role_kubernetes_io/control-plane": [
      ""
    ],
    "agent.version": [
      "8.3.0"
    ],
    "host.os.family": [
      "debian"
    ],
    "kubernetes.node.name": [
      "kind-control-plane"
    ],
    "kubernetes.node.hostname": [
      "kind-control-plane"
    ],
    "user.name": [
      "-"
    ],
    "kubernetes.pod.uid": [
      "bbb0368f-91fe-4af2-9fda-05503d02bef3"
    ],
    "source.address": [
      "127.0.0.1"
    ],
    "host.ip": [
      "10.244.0.1",
      "10.244.0.1",
      "10.244.0.1",
      "10.244.0.1",
      "172.18.0.2",
      "fc00:f853:ccd:e793::2",
      "fe80::42:acff:fe12:2",
      "10.244.0.1",
      "10.244.0.1"
    ],
    "orchestrator.cluster.url": [
      "kind-control-plane:6443"
    ],
    "agent.type": [
      "filebeat"
    ],
    "event.module": [
      "apache"
    ],
    "stream": [
      "stdout"
    ],
    "host.os.kernel": [
      "4.9.184-linuxkit"
    ],
    "kubernetes.pod.name": [
      "httpd-deployment-xfusion-5546996bfc-48znw"
    ],
    "elastic_agent.snapshot": [
      false
    ],
    "kubernetes.pod.ip": [
      "10.244.0.6"
    ],
    "kubernetes.container.name": [
      "httpd-container-xfusion"
    ],
    "elastic_agent.id": [
      "ded3358e-655a-45f3-adcf-618b068e90d5"
    ],
    "data_stream.namespace": [
      "default"
    ],
    "kubernetes.replicaset.name": [
      "httpd-deployment-xfusion-5546996bfc"
    ],
    "host.os.codename": [
      "focal"
    ],
    "kubernetes.namespace_labels.kubernetes_io/metadata_name": [
      "default"
    ],
    "http.response.body.bytes": [
      45
    ],
    "kubernetes.node.labels.kubernetes_io/hostname": [
      "kind-control-plane"
    ],
    "kubernetes.node.labels.beta_kubernetes_io/arch": [
      "amd64"
    ],
    "event.ingested": [
      "2022-07-04T13:07:22.000Z"
    ],
    "url.original": [
      "/"
    ],
    "kubernetes.labels.hints": [
      "parse"
    ],
    "@timestamp": [
      "2022-07-04T13:07:18.000Z"
    ],
    "host.os.platform": [
      "ubuntu"
    ],
    "log.file.path": [
      "/var/log/containers/httpd-deployment-xfusion-5546996bfc-48znw_default_httpd-container-xfusion-ea36b297cd0d1e576eb358f93ec7a2489b9741daeea1ca0af233053b1f6b529a.log"
    ],
    "data_stream.dataset": [
      "apache.access"
    ],
    "kubernetes.node.labels.kubernetes_io/arch": [
      "amd64"
    ],
    "agent.ephemeral_id": [
      "c619f972-34de-403d-ac1f-1dc188a952f5"
    ],
    "kubernetes.node.labels.node_kubernetes_io/exclude-from-external-load-balancers": [
      ""
    ],
    "event.dataset": [
      "apache.access"
    ]
  }
}

So it seems that for logs we need something like the following in the templates:

  - name: apache-1
    type: filestream
    use_output: default
    meta:
      package:
        name: apache
        version: 1.3.5
    data_stream:
      namespace: default
    streams:
      - data_stream:
          dataset: apache.access
          type: logs
        paths:
          - "/var/log/containers/*${hints.container.id}.log"
        tags:
          - apache-access
        exclude_files:
          - .gz$  # Is this needed ???
        prospector.scanner.symlinks: true
        parsers:
            - container:
              stream: "${hints.redis.access.stream|'all'}" # Here users would use co.elastic.hints/access.stream: stdout
              format: auto
      - data_stream:
          dataset: apache.error
          type: logs
        paths:
          - "/var/log/containers/*${hints.container.id}.log"
        exclude_files:
          - .gz$
        tags:
          - apache-error
        processors:
          - add_locale: null
        prospector.scanner.symlinks: true
        parsers:
          - container:
            stream: "${hints.redis.error.stream|'all'}" # Here users would use co.elastic.hints/access.stream: stderr
            format: auto

Note that we add the container parser and we set the path accordingly.

@gizas @ruflin let me know what you think.

@ChrsMark ChrsMark changed the title Add support for hint's based autodiscovery in kubernetes provider Add support for hints' based autodiscovery in kubernetes provider Jul 4, 2022
This was referenced Jul 4, 2022
Open
Closed
Closed
Merged
@ChrsMark
Copy link
Member Author

At the moment inputs with defined processors will not be able to receive extra processors coming from the hints because of a known limitation: #735

@ChrsMark ChrsMark mentioned this issue Aug 29, 2022
Merged
1 task
@gizas gizas mentioned this issue Sep 14, 2022
Open
3 tasks
This was referenced Sep 14, 2022
Merged
Merged
@ChrsMark
Copy link
Member Author

3/3 tasks completed. Closing this one.

@gizas gizas mentioned this issue Sep 28, 2022
Open
@mlunadia
Copy link

@ChrsMark where can I find a list of the supported packages for this feature?

@ChrsMark
Copy link
Member Author

@ChrsMark where can I find a list of the supported packages for this feature?

https://www.elastic.co/guide/en/fleet/current/hints-annotations-autodiscovery.html#_available_packages_that_support_hints_autodiscovery

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ChrsMark ChrsMark

Labels
8.5-candidate Team:Cloudnative-Monitoring Label for the Cloud Native Monitoring team
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@ChrsMark @mlunadia