Make use of consumer_lag in dashboard

[ChrsMark]

This PR updates the Kafka Consumer Group Lag vs Time [Metricbeat Kafka] ECS visualisation to make use of the consumer_lag field added on #14822.

This may resolve_ #3608

Manual Testing

Note: Manual testing can be tricky since this requires an actual Kafka setup with consumer/producers.

An easy way to manual test this is to make use of the container we provide for the integration tests.

Start a Kafka service using the testing container: https://github.com/elastic/beats/blob/master/metricbeat/module/kafka/_meta/Dockerfile

One can run the tests in order to bring up the container like go test -v -tags=integration github.com/elastic/beats/metricbeat/module/kafka/partition.

Verify that tests are successful and you can go on to manual testing. After the tests have finished the container is still alive so we can use it for the manual testing too.

Check what are the exported ports from Docker with a docker ps:

5da9631cfa25        docker.elastic.co/observability-ci/beats-integration-kafka:2.1.1-1   "/run.sh"           23 minutes ago      Up 23 minutes (healthy)   2181/tcp, 0.0.0.0:32771->8774/tcp, 0.0.0.0:32770->8775/tcp, 0.0.0.0:32769->8779/tcp, 0.0.0.0:32768->9092/tcp   metricbeat_kafka_1

Kafka Module configuration

In Kafka module configuration make use of the testing creds -> https://github.com/elastic/beats/blob/master/metricbeat/module/kafka/consumergroup/consumergroup_integration_test.go#L38

The config should look like:

- module: kafka
  metricsets:
    - partition
    - consumergroup
  period: 10s
  hosts: ["0.0.0.0:32784"]
  username: stats
  password: test-secret

Where 0.0.0.0:32784 comes from the port mapping of Docker like: 0.0.0.0:32784->9092/tcp, use docker ps to see which the mapped port to the host network.

Sample producer/consumer pairs

In order to populate consumer lag data exec into the Kafka container and spawn 2 pairs of consumer/producer like:
Producer:
{ while sleep 1; do echo message; done } | KAFKA_OPTS="-Djava.security.auth.login.config=/kafka/bin/jaas-kafka-client-producer.conf" /kafka/bin/kafka-console-producer.sh --topic test2 --broker-list localhost:9091 --producer.config /kafka/bin/sasl-producer.properties
https://github.com/elastic/beats/blob/master/metricbeat/module/kafka/_meta/run.sh#L76
Consumer:
KAFKA_OPTS="-Djava.security.auth.login.config=/kafka/bin/jaas-kafka-client-consumer.conf" /kafka/bin/kafka-console-consumer.sh --topic=test2 --bootstrap-server=localhost:9091 --consumer.config /kafka/bin/sasl-producer.properties
https://github.com/elastic/beats/blob/master/metricbeat/module/kafka/_meta/run.sh#L82

After that the data in the dashboard is expected to be populated accordingly.

[mtojek]

There is a typo in the PR title and description. I'm raising it just to prevent from any copy-paste bugs.

BTW. do you plan to update PNG screenshots?

[jsoriano]

Thanks!