[metribeat] Failed to parse kubernetes.labels.app

[nerddelphi]

Hi everyone. Based on this closed issue 8773 - Filebeat and Kubernetes pods with labels (helm deployments, etc), I guess it wasn't fixed at all. I'm getting the same behavior, even using the latest version of metricbeat (7.1.1) and config:

processors:
 -  add_kubernetes_metadata:
      labels.dedot: true
      annotations.dedot: true

When metricbeat tries to push out events like these below to Elasticsearch:
{ "@timestamp": "2019-06-21T14:30:06.333Z", "@metadata": { "beat": "metricbeat", "type": "_doc", "version": "7.1.1" }, "ecs": { "version": "1.0.0" }, "cloud": { "availability_zone": "us-east1-b", "instance": { "id": "123123213213", "name": "gke-staging-preemptible-pool-xxxxxxx" }, "machine": { "type": "n1-standard-8" }, "project": { "id": "kubernetes-staging-220222" }, "provider": "gcp" }, "kubernetes": { "deployment": { "name": "shared-queue-preemptive", "replicas": { "unavailable": 0, "updated": 1, "desired": 1, "available": 1 }, "paused": false }, "namespace": "shared-queue", **"labels": { "app": { "kubernetes": { "io/instance": "shared-queue", "io/managed-by": "Tiller", "io/name": "shared-queue", "io/version": "1.3.25" } }**, "helm": { "sh/chart": "microservice-0.1.2" } } }, "metricset": { "name": "state_deployment" }, "service": { "address": "kube-state-metrics.kube-system.svc.cluster.local:8080", "type": "kubernetes" }, "event": { "module": "kubernetes", "duration": 384732698, "dataset": "kubernetes.deployment" }, "host": { "name": "gke-staging-preemptible-pool-xxxxxx" }, "agent": { "hostname": "gke-staging-preemptible-pool-xxxxxxx", "id": "d6e948a1-419e-440c-bdd3-c110209e5942", "version": "7.1.1", "type": "metricbeat", "ephemeral_id": "8810147c-0e44-4583-a663-b91c788309c4" } }

Obviously because these labels:
"kubernetes": {
"deployment": {
"name": "shared-queue-preemptive",
"replicas": {
"unavailable": 0,
"updated": 1,
"desired": 1,
"available": 1
},
"paused": false
},
"namespace": "shared-queue",
"labels": {
"app": {
"kubernetes": {
"io/instance": "shared-queue",
"io/managed-by": "Tiller",
"io/name": "shared-queue",
"io/version": "1.3.25"
}
},
"helm": {
"sh/chart": "microservice-0.1.2"
}
}
}

[kaiyan-sheng]

Thanks @nerddelphi, can you also paste the error message you saw here please?

[nerddelphi]

@kaiyan-sheng These errors:

{"type":"mapper_parsing_exception","reason":"failed to parse field [kubernetes.labels.app] of type [keyword] in document with id '6OPyamsBGJCb4HgtJtQp'","caused_by":{"type":"illegal_state_exception","reason":"Can't get text on a START_OBJECT at 1:264"}}

and

{"type":"mapper_parsing_exception","reason":"object mapping for [kubernetes.labels.app] tried to parse field [app] as object, but found a concrete value"}

[kaiyan-sheng]

@nerddelphi Thank you! I suspect it's because how we define kubernetes.labels https://github.com/elastic/beats/blob/7.1/libbeat/processors/add_kubernetes_metadata/_meta/fields.yml#L31 here. Will take it into investigation and update with whatever I find.

[nerddelphi]

@kaiyan-sheng ok. I appreciate it and I hope we can fix that :)

[jserviceorg]

got the same, is there any fix / workaround available yet? the filebeat fixes dont work for metricbeat (drop fields, rename)