[Kafka] [tsdb] Add metric_type for consumer_group datastream

packages/kafka/changelog.yml

@@ -1,4 +1,9 @@
 # newer versions go on top
+- version: "1.5.1"
+  changes:
+    - description: Add  metric_type mapping for consumer_group datastream.
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/5932
 - version: "1.5.0"
   changes:
     - description: Add metric_type mapping for broker datastream.

packages/kafka/data_stream/consumergroup/fields/fields.yml

@@ -6,12 +6,14 @@
       description: Consumer Group ID
     - name: offset
       type: long
+      metric_type: gauge
       description: consumer offset into partition being read
     - name: meta
       type: keyword
       description: custom consumer meta data string
     - name: consumer_lag
       type: long
+      metric_type: gauge
       description: consumer lag for partition/topic calculated as the difference between the partition offset and consumer offset
     - name: error.code
       type: long

packages/kafka/docs/README.md

@@ -259,62 +259,62 @@ An example event for `consumergroup` looks as following:
 
 **Exported fields**
 
-| Field | Description | Type |
-|---|---|---|
-| @timestamp | Event timestamp. | date |
-| cloud.account.id | The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier. | keyword |
-| cloud.availability_zone | Availability zone in which this host is running. | keyword |
-| cloud.image.id | Image ID for the cloud instance. | keyword |
-| cloud.instance.id | Instance ID of the host machine. | keyword |
-| cloud.instance.name | Instance name of the host machine. | keyword |
-| cloud.machine.type | Machine type of the host machine. | keyword |
-| cloud.project.id | Name of the project in Google Cloud. | keyword |
-| cloud.provider | Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean. | keyword |
-| cloud.region | Region in which this host is running. | keyword |
-| container.id | Unique container id. | keyword |
-| container.image.name | Name of the image the container was built on. | keyword |
-| container.labels | Image labels. | object |
-| container.name | Container name. | keyword |
-| data_stream.dataset | Data stream dataset. | constant_keyword |
-| data_stream.namespace | Data stream namespace. | constant_keyword |
-| data_stream.type | Data stream type. | constant_keyword |
-| ecs.version | ECS version this event conforms to. `ecs.version` is a required field and must exist in all events. When querying across multiple indices -- which may conform to slightly different ECS versions -- this field lets integrations adjust to the schema version of the events. | keyword |
-| event.dataset | Event dataset | constant_keyword |
-| event.module | Event module | constant_keyword |
-| host.architecture | Operating system architecture. | keyword |
-| host.containerized | If the host is a container. | boolean |
-| host.domain | Name of the domain of which the host is a member. For example, on Windows this could be the host's Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host's LDAP provider. | keyword |
-| host.hostname | Hostname of the host. It normally contains what the `hostname` command returns on the host machine. | keyword |
-| host.id | Unique host id. As hostname is not always unique, use values that are meaningful in your environment. Example: The current usage of `beat.name`. | keyword |
-| host.ip | Host ip addresses. | ip |
-| host.mac | Host mac addresses. | keyword |
-| host.name | Name of the host. It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use. | keyword |
-| host.os.build | OS build information. | keyword |
-| host.os.codename | OS codename, if any. | keyword |
-| host.os.family | OS family (such as redhat, debian, freebsd, windows). | keyword |
-| host.os.kernel | Operating system kernel version as a raw string. | keyword |
-| host.os.name | Operating system name, without the version. | keyword |
-| host.os.name.text | Multi-field of `host.os.name`. | text |
-| host.os.platform | Operating system platform (such centos, ubuntu, windows). | keyword |
-| host.os.version | Operating system version as a raw string. | keyword |
-| host.type | Type of host. For Cloud providers this can be the machine type like `t2.medium`. If vm, this could be the container, for example, or other information meaningful in your environment. | keyword |
-| kafka.broker.address | Broker advertised address | keyword |
-| kafka.broker.id | Broker id | long |
-| kafka.consumergroup.client.host | Client host | keyword |
-| kafka.consumergroup.client.id | Client ID (kafka setting client.id) | keyword |
-| kafka.consumergroup.client.member_id | internal consumer group member ID | keyword |
-| kafka.consumergroup.consumer_lag | consumer lag for partition/topic calculated as the difference between the partition offset and consumer offset | long |
-| kafka.consumergroup.error.code | kafka consumer/partition error code. | long |
-| kafka.consumergroup.id | Consumer Group ID | keyword |
-| kafka.consumergroup.meta | custom consumer meta data string | keyword |
-| kafka.consumergroup.offset | consumer offset into partition being read | long |
-| kafka.partition.id | Partition id. | long |
-| kafka.partition.topic_broker_id | Unique id of the partition in the topic and the broker. | keyword |
-| kafka.partition.topic_id | Unique id of the partition in the topic. | keyword |
-| kafka.topic.error.code | Topic error code. | long |
-| kafka.topic.name | Topic name | keyword |
-| service.address | Address where data about this service was collected from. This should be a URI, network address (ipv4:port or [ipv6]:port) or a resource path (sockets). | keyword |
-| service.type | The type of the service data is collected from. The type can be used to group and correlate logs and metrics from one service type. Example: If logs or metrics are collected from Elasticsearch, `service.type` would be `elasticsearch`. | keyword |
+| Field | Description | Type | Metric Type |
+|---|---|---|---|
+| @timestamp | Event timestamp. | date |  |
+| cloud.account.id | The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier. | keyword |  |
+| cloud.availability_zone | Availability zone in which this host is running. | keyword |  |
+| cloud.image.id | Image ID for the cloud instance. | keyword |  |
+| cloud.instance.id | Instance ID of the host machine. | keyword |  |
+| cloud.instance.name | Instance name of the host machine. | keyword |  |
+| cloud.machine.type | Machine type of the host machine. | keyword |  |
+| cloud.project.id | Name of the project in Google Cloud. | keyword |  |
+| cloud.provider | Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean. | keyword |  |
+| cloud.region | Region in which this host is running. | keyword |  |
+| container.id | Unique container id. | keyword |  |
+| container.image.name | Name of the image the container was built on. | keyword |  |
+| container.labels | Image labels. | object |  |
+| container.name | Container name. | keyword |  |
+| data_stream.dataset | Data stream dataset. | constant_keyword |  |
+| data_stream.namespace | Data stream namespace. | constant_keyword |  |
+| data_stream.type | Data stream type. | constant_keyword |  |
+| ecs.version | ECS version this event conforms to. `ecs.version` is a required field and must exist in all events. When querying across multiple indices -- which may conform to slightly different ECS versions -- this field lets integrations adjust to the schema version of the events. | keyword |  |
+| event.dataset | Event dataset | constant_keyword |  |
+| event.module | Event module | constant_keyword |  |
+| host.architecture | Operating system architecture. | keyword |  |
+| host.containerized | If the host is a container. | boolean |  |
+| host.domain | Name of the domain of which the host is a member. For example, on Windows this could be the host's Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host's LDAP provider. | keyword |  |
+| host.hostname | Hostname of the host. It normally contains what the `hostname` command returns on the host machine. | keyword |  |
+| host.id | Unique host id. As hostname is not always unique, use values that are meaningful in your environment. Example: The current usage of `beat.name`. | keyword |  |
+| host.ip | Host ip addresses. | ip |  |
+| host.mac | Host mac addresses. | keyword |  |
+| host.name | Name of the host. It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use. | keyword |  |
+| host.os.build | OS build information. | keyword |  |
+| host.os.codename | OS codename, if any. | keyword |  |
+| host.os.family | OS family (such as redhat, debian, freebsd, windows). | keyword |  |
+| host.os.kernel | Operating system kernel version as a raw string. | keyword |  |
+| host.os.name | Operating system name, without the version. | keyword |  |
+| host.os.name.text | Multi-field of `host.os.name`. | text |  |
+| host.os.platform | Operating system platform (such centos, ubuntu, windows). | keyword |  |
+| host.os.version | Operating system version as a raw string. | keyword |  |
+| host.type | Type of host. For Cloud providers this can be the machine type like `t2.medium`. If vm, this could be the container, for example, or other information meaningful in your environment. | keyword |  |
+| kafka.broker.address | Broker advertised address | keyword |  |
+| kafka.broker.id | Broker id | long |  |
+| kafka.consumergroup.client.host | Client host | keyword |  |
+| kafka.consumergroup.client.id | Client ID (kafka setting client.id) | keyword |  |
+| kafka.consumergroup.client.member_id | internal consumer group member ID | keyword |  |
+| kafka.consumergroup.consumer_lag | consumer lag for partition/topic calculated as the difference between the partition offset and consumer offset | long | gauge |
+| kafka.consumergroup.error.code | kafka consumer/partition error code. | long |  |
+| kafka.consumergroup.id | Consumer Group ID | keyword |  |
+| kafka.consumergroup.meta | custom consumer meta data string | keyword |  |
+| kafka.consumergroup.offset | consumer offset into partition being read | long | gauge |
+| kafka.partition.id | Partition id. | long |  |
+| kafka.partition.topic_broker_id | Unique id of the partition in the topic and the broker. | keyword |  |
+| kafka.partition.topic_id | Unique id of the partition in the topic. | keyword |  |
+| kafka.topic.error.code | Topic error code. | long |  |
+| kafka.topic.name | Topic name | keyword |  |
+| service.address | Address where data about this service was collected from. This should be a URI, network address (ipv4:port or [ipv6]:port) or a resource path (sockets). | keyword |  |
+| service.type | The type of the service data is collected from. The type can be used to group and correlate logs and metrics from one service type. Example: If logs or metrics are collected from Elasticsearch, `service.type` would be `elasticsearch`. | keyword |  |
 
 
 ### partition

packages/kafka/manifest.yml

@@ -1,7 +1,7 @@
 format_version: 1.0.0
 name: kafka
 title: Kafka
-version: 1.5.0
+version: 1.5.1
 license: basic
 description: Collect logs and metrics from Kafka servers with Elastic Agent.
 type: integration