Skip to content

Commit 4dee550

Browse files
constanca-msodhikirti07
authored andcommitted
[AWS] Add metric type to SNS, SQS and Billing data streams (#6319)
* Add metric type to SNS. Signed-off-by: constanca-m <[email protected]> * Add metric type to SQS. Signed-off-by: constanca-m <[email protected]> * Add metric type to Billing. Signed-off-by: constanca-m <[email protected]> * Update changelog. Signed-off-by: constanca-m <[email protected]> * Update files. Signed-off-by: constanca-m <[email protected]> --------- Signed-off-by: constanca-m <[email protected]>
1 parent 854a638 commit 4dee550

File tree

8 files changed

+227
-196
lines changed

8 files changed

+227
-196
lines changed

packages/aws/changelog.yml

+5
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,9 @@
11
# newer versions go on top
2+
- version: "1.40.0"
3+
changes:
4+
- description: Add metric type to SNS, SQS and Billing data streams.
5+
type: enhancement
6+
link: https://github.com/elastic/integrations/pull/6319
27
- version: "1.39.0"
38
changes:
49
- description: Add AWS API Gateway data stream.

packages/aws/data_stream/billing/fields/fields.yml

+6
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@
66
fields:
77
- name: EstimatedCharges
88
type: long
9+
metric_type: gauge
910
description: Maximum estimated charges for AWS acccount.
1011
- name: Currency
1112
type: keyword
@@ -19,6 +20,7 @@
1920
fields:
2021
- name: amount
2122
type: double
23+
metric_type: gauge
2224
description: Amortized cost amount.
2325
- name: unit
2426
type: keyword
@@ -28,6 +30,7 @@
2830
fields:
2931
- name: amount
3032
type: double
33+
metric_type: gauge
3134
description: Blended cost amount.
3235
- name: unit
3336
type: keyword
@@ -37,6 +40,7 @@
3740
fields:
3841
- name: amount
3942
type: double
43+
metric_type: gauge
4044
description: Normalized usage amount.
4145
- name: unit
4246
type: keyword
@@ -46,6 +50,7 @@
4650
fields:
4751
- name: amount
4852
type: double
53+
metric_type: gauge
4954
description: Unblended cost amount.
5055
- name: unit
5156
type: keyword
@@ -54,6 +59,7 @@
5459
type: group
5560
fields:
5661
- name: amount
62+
metric_type: gauge
5763
type: double
5864
description: Usage quantity amount.
5965
- name: unit

packages/aws/data_stream/sns/fields/fields.yml

+11
Original file line numberDiff line numberDiff line change
@@ -32,35 +32,46 @@
3232
fields:
3333
- name: PublishSize.avg
3434
type: double
35+
metric_type: gauge
3536
description: The size of messages published.
3637
- name: SMSSuccessRate.avg
3738
type: double
39+
metric_type: gauge
3840
description: The rate of successful SMS message deliveries.
3941
- name: NumberOfMessagesPublished.sum
4042
type: long
43+
metric_type: gauge
4144
description: The number of messages published to your Amazon SNS topics.
4245
- name: NumberOfNotificationsDelivered.sum
4346
type: long
47+
metric_type: gauge
4448
description: The number of messages successfully delivered from your Amazon SNS topics to subscribing endpoints.
4549
- name: NumberOfNotificationsFailed.sum
4650
type: long
51+
metric_type: gauge
4752
description: The number of messages that Amazon SNS failed to deliver.
4853
- name: NumberOfNotificationsFilteredOut.sum
4954
type: long
55+
metric_type: gauge
5056
description: The number of messages that were rejected by subscription filter policies.
5157
- name: NumberOfNotificationsFilteredOut-InvalidAttributes.sum
5258
type: long
59+
metric_type: gauge
5360
description: The number of messages that were rejected by subscription filter policies because the messages' attributes are invalid - for example, because the attribute JSON is incorrectly formatted.
5461
- name: NumberOfNotificationsFilteredOut-NoMessageAttributes.sum
5562
type: long
63+
metric_type: gauge
5664
description: The number of messages that were rejected by subscription filter policies because the messages have no attributes.
5765
- name: NumberOfNotificationsRedrivenToDlq.sum
5866
type: long
67+
metric_type: gauge
5968
description: The number of messages that have been moved to a dead-letter queue.
6069
- name: NumberOfNotificationsFailedToRedriveToDlq.sum
6170
type: long
71+
metric_type: gauge
6272
description: The number of messages that couldn't be moved to a dead-letter queue.
6373
- name: SMSMonthToDateSpentUSD.sum
74+
metric_type: gauge
6475
type: long
6576
description: The charges you have accrued since the start of the current calendar month for sending SMS messages.
6677
- name: cloudwatch

packages/aws/data_stream/sqs/fields/fields.yml

+9
Original file line numberDiff line numberDiff line change
@@ -13,39 +13,48 @@
1313
fields:
1414
- name: oldest_message_age.sec
1515
type: long
16+
metric_type: gauge
1617
format: duration
1718
description: |
1819
The approximate age of the oldest non-deleted message in the queue.
1920
- name: messages.delayed
2021
type: long
22+
metric_type: gauge
2123
description: |
2224
TThe number of messages in the queue that are delayed and not available for reading immediately.
2325
- name: messages.not_visible
2426
type: long
27+
metric_type: gauge
2528
description: |
2629
The number of messages that are in flight.
2730
- name: messages.visible
2831
type: long
32+
metric_type: gauge
2933
description: |
3034
The number of messages available for retrieval from the queue.
3135
- name: messages.deleted
3236
type: long
37+
metric_type: gauge
3338
description: |
3439
The number of messages deleted from the queue.
3540
- name: messages.received
3641
type: long
42+
metric_type: gauge
3743
description: |
3844
The number of messages returned by calls to the ReceiveMessage action.
3945
- name: messages.sent
4046
type: long
47+
metric_type: gauge
4148
description: |
4249
The number of messages added to a queue.
4350
- name: empty_receives
4451
type: long
52+
metric_type: gauge
4553
description: |
4654
The number of ReceiveMessage API calls that did not return a message.
4755
- name: sent_message_size.bytes
4856
type: long
57+
metric_type: gauge
4958
format: bytes
5059
description: |
5160
The size of messages added to a queue.

packages/aws/docs/billing.md

+70-70
Original file line numberDiff line numberDiff line change
@@ -111,74 +111,74 @@ An example event for `billing` looks as following:
111111

112112
**Exported fields**
113113

114-
| Field | Description | Type |
115-
|---|---|---|
116-
| @timestamp | Event timestamp. | date |
117-
| agent.id | Unique identifier of this agent (if one exists). Example: For Beats this would be beat.id. | keyword |
118-
| aws.\*.metrics.\*.\* | Metrics that returned from Cloudwatch API query. | object |
119-
| aws.billing.AmortizedCost.amount | Amortized cost amount. | double |
120-
| aws.billing.AmortizedCost.unit | Amortized cost unit. | keyword |
121-
| aws.billing.BlendedCost.amount | Blended cost amount. | double |
122-
| aws.billing.BlendedCost.unit | Blended cost unit. | keyword |
123-
| aws.billing.Currency | Currency name. | keyword |
124-
| aws.billing.EstimatedCharges | Maximum estimated charges for AWS acccount. | long |
125-
| aws.billing.NormalizedUsageAmount.amount | Normalized usage amount. | double |
126-
| aws.billing.NormalizedUsageAmount.unit | Normalized usage amount unit. | keyword |
127-
| aws.billing.ServiceName | AWS service name. | keyword |
128-
| aws.billing.UnblendedCost.amount | Unblended cost amount. | double |
129-
| aws.billing.UnblendedCost.unit | Unblended cost unit. | keyword |
130-
| aws.billing.UsageQuantity.amount | Usage quantity amount. | double |
131-
| aws.billing.UsageQuantity.unit | Usage quantity unit. | keyword |
132-
| aws.billing.end_date | End date for retrieving AWS costs. | keyword |
133-
| aws.billing.group_by | Cost explorer group by key values. | object |
134-
| aws.billing.group_definition.key | The string that represents a key for a specified group. | keyword |
135-
| aws.billing.group_definition.type | The string that represents the type of group. | keyword |
136-
| aws.billing.start_date | Start date for retrieving AWS costs. | keyword |
137-
| aws.cloudwatch.namespace | The namespace specified when query cloudwatch api. | keyword |
138-
| aws.dimensions.\* | Metric dimensions. | object |
139-
| aws.linked_account.id | ID used to identify linked account. | keyword |
140-
| aws.linked_account.name | Name or alias used to identify linked account. | keyword |
141-
| aws.s3.bucket.name | Name of a S3 bucket. | keyword |
142-
| aws.tags.\* | Tag key value pairs from aws resources. | object |
143-
| cloud | Fields related to the cloud or infrastructure the events are coming from. | group |
144-
| cloud.account.id | The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier. | keyword |
145-
| cloud.account.name | The cloud account name or alias used to identify different entities in a multi-tenant environment. Examples: AWS account name, Google Cloud ORG display name. | keyword |
146-
| cloud.availability_zone | Availability zone in which this host, resource, or service is located. | keyword |
147-
| cloud.image.id | Image ID for the cloud instance. | keyword |
148-
| cloud.instance.id | Instance ID of the host machine. | keyword |
149-
| cloud.instance.name | Instance name of the host machine. | keyword |
150-
| cloud.machine.type | Machine type of the host machine. | keyword |
151-
| cloud.project.id | The cloud project identifier. Examples: Google Cloud Project id, Azure Project id. | keyword |
152-
| cloud.provider | Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean. | keyword |
153-
| cloud.region | Region in which this host, resource, or service is located. | keyword |
154-
| container.id | Unique container id. | keyword |
155-
| container.image.name | Name of the image the container was built on. | keyword |
156-
| container.labels | Image labels. | object |
157-
| container.name | Container name. | keyword |
158-
| data_stream.dataset | Data stream dataset. | constant_keyword |
159-
| data_stream.namespace | Data stream namespace. | constant_keyword |
160-
| data_stream.type | Data stream type. | constant_keyword |
161-
| ecs.version | ECS version this event conforms to. `ecs.version` is a required field and must exist in all events. When querying across multiple indices -- which may conform to slightly different ECS versions -- this field lets integrations adjust to the schema version of the events. | keyword |
162-
| error | These fields can represent errors of any kind. Use them for errors that happen while fetching events or in cases where the event itself contains an error. | group |
163-
| error.message | Error message. | match_only_text |
164-
| event.dataset | Event dataset | constant_keyword |
165-
| event.module | Event module | constant_keyword |
166-
| host.architecture | Operating system architecture. | keyword |
167-
| host.containerized | If the host is a container. | boolean |
168-
| host.domain | Name of the domain of which the host is a member. For example, on Windows this could be the host's Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host's LDAP provider. | keyword |
169-
| host.hostname | Hostname of the host. It normally contains what the `hostname` command returns on the host machine. | keyword |
170-
| host.id | Unique host id. As hostname is not always unique, use values that are meaningful in your environment. Example: The current usage of `beat.name`. | keyword |
171-
| host.ip | Host ip addresses. | ip |
172-
| host.mac | Host MAC addresses. The notation format from RFC 7042 is suggested: Each octet (that is, 8-bit byte) is represented by two [uppercase] hexadecimal digits giving the value of the octet as an unsigned integer. Successive octets are separated by a hyphen. | keyword |
173-
| host.name | Name of the host. It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use. | keyword |
174-
| host.os.build | OS build information. | keyword |
175-
| host.os.codename | OS codename, if any. | keyword |
176-
| host.os.family | OS family (such as redhat, debian, freebsd, windows). | keyword |
177-
| host.os.kernel | Operating system kernel version as a raw string. | keyword |
178-
| host.os.name | Operating system name, without the version. | keyword |
179-
| host.os.name.text | Multi-field of `host.os.name`. | match_only_text |
180-
| host.os.platform | Operating system platform (such centos, ubuntu, windows). | keyword |
181-
| host.os.version | Operating system version as a raw string. | keyword |
182-
| host.type | Type of host. For Cloud providers this can be the machine type like `t2.medium`. If vm, this could be the container, for example, or other information meaningful in your environment. | keyword |
183-
| service.type | The type of the service data is collected from. The type can be used to group and correlate logs and metrics from one service type. Example: If logs or metrics are collected from Elasticsearch, `service.type` would be `elasticsearch`. | keyword |
114+
| Field | Description | Type | Metric Type |
115+
|---|---|---|---|
116+
| @timestamp | Event timestamp. | date | |
117+
| agent.id | Unique identifier of this agent (if one exists). Example: For Beats this would be beat.id. | keyword | |
118+
| aws.\*.metrics.\*.\* | Metrics that returned from Cloudwatch API query. | object | |
119+
| aws.billing.AmortizedCost.amount | Amortized cost amount. | double | gauge |
120+
| aws.billing.AmortizedCost.unit | Amortized cost unit. | keyword | |
121+
| aws.billing.BlendedCost.amount | Blended cost amount. | double | gauge |
122+
| aws.billing.BlendedCost.unit | Blended cost unit. | keyword | |
123+
| aws.billing.Currency | Currency name. | keyword | |
124+
| aws.billing.EstimatedCharges | Maximum estimated charges for AWS acccount. | long | gauge |
125+
| aws.billing.NormalizedUsageAmount.amount | Normalized usage amount. | double | gauge |
126+
| aws.billing.NormalizedUsageAmount.unit | Normalized usage amount unit. | keyword | |
127+
| aws.billing.ServiceName | AWS service name. | keyword | |
128+
| aws.billing.UnblendedCost.amount | Unblended cost amount. | double | gauge |
129+
| aws.billing.UnblendedCost.unit | Unblended cost unit. | keyword | |
130+
| aws.billing.UsageQuantity.amount | Usage quantity amount. | double | gauge |
131+
| aws.billing.UsageQuantity.unit | Usage quantity unit. | keyword | |
132+
| aws.billing.end_date | End date for retrieving AWS costs. | keyword | |
133+
| aws.billing.group_by | Cost explorer group by key values. | object | |
134+
| aws.billing.group_definition.key | The string that represents a key for a specified group. | keyword | |
135+
| aws.billing.group_definition.type | The string that represents the type of group. | keyword | |
136+
| aws.billing.start_date | Start date for retrieving AWS costs. | keyword | |
137+
| aws.cloudwatch.namespace | The namespace specified when query cloudwatch api. | keyword | |
138+
| aws.dimensions.\* | Metric dimensions. | object | |
139+
| aws.linked_account.id | ID used to identify linked account. | keyword | |
140+
| aws.linked_account.name | Name or alias used to identify linked account. | keyword | |
141+
| aws.s3.bucket.name | Name of a S3 bucket. | keyword | |
142+
| aws.tags.\* | Tag key value pairs from aws resources. | object | |
143+
| cloud | Fields related to the cloud or infrastructure the events are coming from. | group | |
144+
| cloud.account.id | The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier. | keyword | |
145+
| cloud.account.name | The cloud account name or alias used to identify different entities in a multi-tenant environment. Examples: AWS account name, Google Cloud ORG display name. | keyword | |
146+
| cloud.availability_zone | Availability zone in which this host, resource, or service is located. | keyword | |
147+
| cloud.image.id | Image ID for the cloud instance. | keyword | |
148+
| cloud.instance.id | Instance ID of the host machine. | keyword | |
149+
| cloud.instance.name | Instance name of the host machine. | keyword | |
150+
| cloud.machine.type | Machine type of the host machine. | keyword | |
151+
| cloud.project.id | The cloud project identifier. Examples: Google Cloud Project id, Azure Project id. | keyword | |
152+
| cloud.provider | Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean. | keyword | |
153+
| cloud.region | Region in which this host, resource, or service is located. | keyword | |
154+
| container.id | Unique container id. | keyword | |
155+
| container.image.name | Name of the image the container was built on. | keyword | |
156+
| container.labels | Image labels. | object | |
157+
| container.name | Container name. | keyword | |
158+
| data_stream.dataset | Data stream dataset. | constant_keyword | |
159+
| data_stream.namespace | Data stream namespace. | constant_keyword | |
160+
| data_stream.type | Data stream type. | constant_keyword | |
161+
| ecs.version | ECS version this event conforms to. `ecs.version` is a required field and must exist in all events. When querying across multiple indices -- which may conform to slightly different ECS versions -- this field lets integrations adjust to the schema version of the events. | keyword | |
162+
| error | These fields can represent errors of any kind. Use them for errors that happen while fetching events or in cases where the event itself contains an error. | group | |
163+
| error.message | Error message. | match_only_text | |
164+
| event.dataset | Event dataset | constant_keyword | |
165+
| event.module | Event module | constant_keyword | |
166+
| host.architecture | Operating system architecture. | keyword | |
167+
| host.containerized | If the host is a container. | boolean | |
168+
| host.domain | Name of the domain of which the host is a member. For example, on Windows this could be the host's Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host's LDAP provider. | keyword | |
169+
| host.hostname | Hostname of the host. It normally contains what the `hostname` command returns on the host machine. | keyword | |
170+
| host.id | Unique host id. As hostname is not always unique, use values that are meaningful in your environment. Example: The current usage of `beat.name`. | keyword | |
171+
| host.ip | Host ip addresses. | ip | |
172+
| host.mac | Host MAC addresses. The notation format from RFC 7042 is suggested: Each octet (that is, 8-bit byte) is represented by two [uppercase] hexadecimal digits giving the value of the octet as an unsigned integer. Successive octets are separated by a hyphen. | keyword | |
173+
| host.name | Name of the host. It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use. | keyword | |
174+
| host.os.build | OS build information. | keyword | |
175+
| host.os.codename | OS codename, if any. | keyword | |
176+
| host.os.family | OS family (such as redhat, debian, freebsd, windows). | keyword | |
177+
| host.os.kernel | Operating system kernel version as a raw string. | keyword | |
178+
| host.os.name | Operating system name, without the version. | keyword | |
179+
| host.os.name.text | Multi-field of `host.os.name`. | match_only_text | |
180+
| host.os.platform | Operating system platform (such centos, ubuntu, windows). | keyword | |
181+
| host.os.version | Operating system version as a raw string. | keyword | |
182+
| host.type | Type of host. For Cloud providers this can be the machine type like `t2.medium`. If vm, this could be the container, for example, or other information meaningful in your environment. | keyword | |
183+
| service.type | The type of the service data is collected from. The type can be used to group and correlate logs and metrics from one service type. Example: If logs or metrics are collected from Elasticsearch, `service.type` would be `elasticsearch`. | keyword | |
184184

0 commit comments

Comments
 (0)