[filebeat] Failed to parse kubernetes.labels.app

[dmitryzykov]

I'm using helm/stable/filebeat which is based on docker.elastic.co/beats/filebeat-oss:6.4.2
all other components are also oss:6.4.2

I'm having a problem when filebeat failed to parse kubernetes.labels.app for many containers.

This is filebeat log when I'm sending directly to elasticsearch:

2018-10-27T09:42:23.923Z	WARN	elasticsearch/client.go:520	Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0x12a6762a, ext:63676230135, loc:(*time.Location)(nil)}, Meta:common.MapStr(nil), Fields:common.MapStr{"beat":common.MapStr{"version":"6.4.2", "name":"filebeat-tx7fz", "hostname":"filebeat-tx7fz"}, "offset":3227, "stream":"stderr", "message":"   at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngine.GetService(Type serviceType, ServiceProviderEngineScope serviceProviderEngineScope)", "kubernetes":common.MapStr{"container":common.MapStr{"name":"sink"}, "namespace":"default", "replicaset":common.MapStr{"name":"test-propertysite-sink-78549f4bb7"}, "labels":common.MapStr{"pod-template-hash":"3410590663", "app":common.MapStr{"kubernetes":common.MapStr{"io/name":"propertysite", "io/component":"sink", "io/instance":"test"}}}, "pod":common.MapStr{"name":"test-propertysite-sink-78549f4bb7-rzj84"}, "node":common.MapStr{"name":"aks-nodepool1-38193062-2"}}, "host":common.MapStr{"name":"filebeat-tx7fz"}, "meta":common.MapStr{"cloud":common.MapStr{"provider":"az", "instance_id":"0fdf45d7-7bfa-4f34-b5e6-9edc8dde83e6", "instance_name":"aks-nodepool1-38193062-2", "machine_type":"Standard_DS2_v2", "region":"westeurope"}}, "source":"/var/lib/docker/containers/408f7e34fa197c31708354997cbbac77b1deb460c6ab7d2bb62ae59dae8a2231/408f7e34fa197c31708354997cbbac77b1deb460c6ab7d2bb62ae59dae8a2231-json.log", "prospector":common.MapStr{"type":"docker"}, "input":common.MapStr{"type":"docker"}}, Private:file.State{Id:"", Finished:false, Fileinfo:(*os.fileStat)(0xc4203b3040), Source:"/var/lib/docker/containers/408f7e34fa197c31708354997cbbac77b1deb460c6ab7d2bb62ae59dae8a2231/408f7e34fa197c31708354997cbbac77b1deb460c6ab7d2bb62ae59dae8a2231-json.log", Offset:3464, Timestamp:time.Time{wall:0xbeed2a9fb6313090, ext:224243617813, loc:(*time.Location)(0x1f61860)}, TTL:-1, Type:"docker", Meta:map[string]string(nil), FileStateOS:file.StateOS{Inode:0xfe57e, Device:0x801}}}, Flags:0x1} (status=400): {"type":"mapper_parsing_exception","reason":"failed to parse [kubernetes.labels.app]","caused_by":{"type":"illegal_state_exception","reason":"Can't get text on a START_OBJECT at 1:419"}}

and this is the same error from logstash, when I'm sending from filebeat to logstash

[2018-10-26T19:38:14,673][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"filebeat-2018.10.26", :_type=>"doc", :_routing=>nil}, #<LogStash::Event:0x21da5026>], :response=>{"index"=>{"_index"=>"filebeat-2018.10.26", "_type"=>"doc", "_id"=>"Ch3isWYBrdoKcExWXgp1", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [kubernetes.labels.app]", "caused_by"=>{"type"=>"illegal_state_exception", "reason"=>"Can't get text on a START_OBJECT at 1:695"}}}}}

This is a part of my filebeat config file:

    filebeat.prospectors:
    - enabled: true
      paths:
      - /var/log/*.log
      - /var/log/messages
      - /var/log/syslog
      type: log
    - containers.ids:
      - '*'
      processors:
      - add_kubernetes_metadata:
          in_cluster: true
      - drop_event:
          when:
            equals:
              kubernetes.container.name: filebeat
      type: docker

I noticed that parsing problems appear only for pods with such labels format(which is recommended for helm charts:

  labels:
    app.kubernetes.io/name: foo
    helm.sh/chart: foo-chart
    app.kubernetes.io/instance: foo
    app.kubernetes.io/managed-by: Tiller
    app.kubernetes.io/component: foo

and the problem doesn't appear for more simpler label format like:

  labels:
    name: bar
    chart: bar-chart
    instance: bar
    managed-by: Tiller
    component: bar

Expected behavior from filebeat is a successful parse of such labels and successful shipment to elasticsearch instead parse error and failure to send log.

For example Fluentd parse app.kubernetes.io/component and it appears in elasticsearch as kubernetes.labels.app_kubernetes_io/component

During bug research, I found similar problems

• https://discuss.elastic.co/t/mapping-for-kubernetes-labels-app-tried-to-parse-field-app-as-object-but-found-a-concrete-value/146490
• https://stackoverflow.com/questions/52760676/elasticsearch-error-kubernetes-labels-app

[lesaux]

I'm seeing this as well.

[wwyhy]

me too, is there any solutions? thanks

[exekias]

Hi everyone, thank you for your detailed report

This issue is caused by label/annotation dots (.) creating hierarchy in Elasticsearch documents. If an annotation has an structure like this: example.annotation.data: some-data and a different one contains example.anottation: value their mapping will conflict, as example.annotation is both an object and a keyword.

We have some counter measures in place to detect these situations when the annotations belong to the same object, but it seems it's not enough.

My thinking goes toward allowing replacing . with _. I will open a PR to do so (as a config flag initially)

In the meanwhile, the workaround would be to drop the offending annotations. You can use the drop_fields processor to do so:

processors:
  - drop_fields:
      when:
        has_fields: ['kubernetes.labels.app']
      fields:
        - 'kubernetes.labels.app'

[lesaux]

Thank you! this fix works in the meantime!

[kuzm1ch]

Hello, this workaround is also helped me, but I have almost the same problem with basic kubernetes component:
quay.io/calico/node:v2.6.7
quay.io/calico/cni:v1.11.2
quay.io/calico/kube-controllers:v1.0.3
and etc.

metricbeat: 6.4.2
kubernetes: v1.10.6
kube-state-metrics: k8s.gcr.io/addon-resizer:1.8.3

kubernetes.labels.role: 
{
  "kubernetes": {
    "io/networking": "1"
  }
}

(I think this metric should be like "kubernetes.labels.role.kubernetes.io/networking: 1" )
and therefore in some case it is parse as object, in anather as text
also create new issue related this problem
#8987

[choppedpork]

Oddly enough, I have two separate clusters with the same version (6.4.2) and identical (as far as filebeat.yml is concerned) config and this issue is only occurring in one of them.

In my case, this issue is only occurring for logs which are being parsed as JSON using json.keys_under_root.

[justinfiore]

We see this issue also with labels like:

labels:
    app.kubernetes.io/name: foo
    app.kubernetes.io/instance: foo
    app.kubernetes.io/component: foo

Is there a fix for this other than dropping the labels?

[sco11morgan]

@exekias Looking forward to your fix for label dots. Maybe like the Docker fix with labels.dedot

[dpankros]

Also getting this, but I'm seeing it primarily in metricbeat.

I just want to add that this label format is what is recommended by the Kubernetes team.