Skip to content

[AWS] Add common variables to all SQS-based AWS inputs #9865

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
kpollich merged 15 commits into from
Jun 4, 2024
Merged

[AWS] Add common variables to all SQS-based AWS inputs #9865

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
21dbabc
Add endpoint var to AWS integration + data streams
kpollich May 14, 2024
9a97542
Update PR in changelog
kpollich May 14, 2024
f3dc2a6
Merge branch 'main' into aws/add-endpoint-var
kpollich May 20, 2024
a881ffa
Add endpoint + region var to all SQS based policy templates
kpollich May 23, 2024
8d7bb24
Remove endpoint from non-SQS inputs
kpollich May 23, 2024
d9ef5eb
fixup! Remove endpoint from non-SQS inputs
kpollich May 23, 2024
52e0d49
fixup! fixup! Remove endpoint from non-SQS inputs
kpollich May 23, 2024
572297b
(wip) Add custom_configurations variable
kpollich May 23, 2024
8e7687e
Add custom configurations to S3 inputs
kpollich May 28, 2024
31f1d68
Remove endpoint from ecs_metrics
kpollich May 28, 2024
fbd458b
Remove top-level integration custom settings for now
kpollich May 28, 2024
b30738d
Fix ec2 logs
kpollich May 28, 2024
9b4314a
Run elastic-package check
kpollich May 28, 2024
52b0ff7
Fix cloudtrail
kpollich May 28, 2024
c5f7f53
Remove blank line from cloudtrail manifest
kpollich May 30, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions packages/aws/_dev/benchmark/rally/billing-benchmark/config.yml
View file
Open in desktop

@kpollich kpollich May 14, 2024

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems like this file just got changed by the linter 馃し

@strawgate strawgate May 14, 2024

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Haiku:
Code repository,
Today, not the time to lint,
Tomorrow, perhaps.

Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,6 @@ fields:
- name: aws.billing.group_definition.type
value: "DIMENSION"
- name: aws.billing.group_by.INSTANCE_TYPE
enum: ["NoInstanceType", "a1.large", "c5.2xlarge", "c5.xlarge", "c6i.2xlarge", "db.r6g.2xlarge", "db.t2.micro", "dc2.large", "m5.large", "t1.micro", "t2.medium", "t2.micro", "t2.small", "t2.xlarge", "t3.2xlarge", "t3.medium", "t3.xlarge","t3.xlarge"]
enum: ["NoInstanceType", "a1.large", "c5.2xlarge", "c5.xlarge", "c6i.2xlarge", "db.r6g.2xlarge", "db.t2.micro", "dc2.large", "m5.large", "t1.micro", "t2.medium", "t2.micro", "t2.small", "t2.xlarge", "t3.2xlarge", "t3.medium", "t3.xlarge", "t3.xlarge"]
- name: aws.billing.group_by.SERVICE
enum: ["Amazon Simple Storage Service", "Amazon Elastic Compute Cloud - Compute", "EC2 - Other", "Amazon Kinesis", "Amazon Relational Database Service", "Amazon Elastic Load Balancing", "AmazonCloudWatch", "AWS CloudTrail", "AWS Config", "AWS Key Management Service", "AWS Lambda", "AWS Secrets Manager", "AWS Service Catalog", "Amazon API Gateway", "Amazon DynamoDB", "Amazon EC2 Container Registry (ECR)", "Amazon Elastic Container Service for Kubernetes", "Amazon Kinesis Firehose", "Amazon Redshift", "Amazon Simple Notification Service", "Amazon Simple Queue Service", "Amazon Virtual Private Cloud"]
enum: ["Amazon Simple Storage Service", "Amazon Elastic Compute Cloud - Compute", "EC2 - Other", "Amazon Kinesis", "Amazon Relational Database Service", "Amazon Elastic Load Balancing", "AmazonCloudWatch", "AWS CloudTrail", "AWS Config", "AWS Key Management Service", "AWS Lambda", "AWS Secrets Manager", "AWS Service Catalog", "Amazon API Gateway", "Amazon DynamoDB", "Amazon EC2 Container Registry (ECR)", "Amazon Elastic Container Service for Kubernetes", "Amazon Kinesis Firehose", "Amazon Redshift", "Amazon Simple Notification Service", "Amazon Simple Queue Service", "Amazon Virtual Private Cloud"]
2 changes: 1 addition & 1 deletion packages/aws/_dev/benchmark/rally/billing-benchmark/fields.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,4 +48,4 @@
- name: aws.billing.group_by.INSTANCE_TYPE
type: keyword
- name: aws.billing.group_by.SERVICE
type: keyword
type: keyword
2 changes: 1 addition & 1 deletion packages/aws/_dev/benchmark/rally/ec2_metrics-benchmark.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,4 +11,4 @@ corpora:
config:
path: ./ec2metrics-benchmark/config.yml
fields:
path: ./ec2metrics-benchmark/fields.yml
path: ./ec2metrics-benchmark/fields.yml
2 changes: 1 addition & 1 deletion packages/aws/_dev/benchmark/rally/sqs-benchmark.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,4 +11,4 @@ corpora:
config:
path: ./sqs-benchmark/config.yml
fields:
path: ./sqs-benchmark/fields.yml
path: ./sqs-benchmark/fields.yml
5 changes: 5 additions & 0 deletions packages/aws/changelog.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,9 @@
# newer versions go on top
- version: "2.15.2-preview01"
changes:
- description: Port endpoint package-level variable from AWS Custom Logs integration to AWS integration
type: enhancement
link: https://github.com/elastic/integrations/pull/9865
- version: "2.15.1"
changes:
- description: Don't index empty AWS Security Hub responses.
Expand Down
3 changes: 3 additions & 0 deletions packages/aws/data_stream/apigateway_logs/agent/stream/aws-cloudwatch.yml.hbs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,9 @@ shared_credential_file: {{shared_credential_file}}
{{#if api_timeout}}
api_timeout: {{api_timeout}}
{{/if}}
{{#if endpoint}}
endpoint: {{endpoint}}
{{/if}}
{{#if default_region}}
default_region: {{default_region}}
{{/if}}
Expand Down
5 changes: 4 additions & 1 deletion packages/aws/data_stream/apigateway_logs/agent/stream/aws-s3.yml.hbs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,9 @@ credential_profile_name: {{credential_profile_name}}
{{#if shared_credential_file}}
shared_credential_file: {{shared_credential_file}}
{{/if}}
{{#if endpoint}}
endpoint: {{endpoint}}
{{/if}}
{{#if default_region}}
default_region: {{default_region}}
{{/if}}
Expand Down Expand Up @@ -69,4 +72,4 @@ publisher_pipeline.disable_host: true
{{#if processors}}
processors:
{{processors}}
{{/if}}
{{/if}}
1 change: 1 addition & 0 deletions packages/aws/data_stream/apigateway_logs/fields/agent.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@
example: "stretch"
description: >
OS codename, if any.

- name: log.offset
type: long
description: Log offset
5 changes: 4 additions & 1 deletion packages/aws/data_stream/apigateway_metrics/agent/stream/stream.yml.hbs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,9 @@ shared_credential_file: {{shared_credential_file}}
{{#if role_arn}}
role_arn: {{role_arn}}
{{/if}}
{{#if endpoint}}
endpoint: {{endpoint}}
{{/if}}
{{#if default_region}}
default_region: {{default_region}}
{{/if}}
Expand Down Expand Up @@ -63,4 +66,4 @@ metrics:
name:
- Latency
- IntegrationLatency
- DataProcessed
- DataProcessed
3 changes: 3 additions & 0 deletions packages/aws/data_stream/billing/agent/stream/stream.yml.hbs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,9 @@ shared_credential_file: {{shared_credential_file}}
{{#if role_arn}}
role_arn: {{role_arn}}
{{/if}}
{{#if endpoint}}
endpoint: {{endpoint}}
{{/if}}
{{#if default_region}}
default_region: {{default_region}}
{{/if}}
Expand Down
2 changes: 2 additions & 0 deletions packages/aws/data_stream/billing/manifest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,7 @@ streams:
default: false
description: >
Enable leaderelection between a set of Elastic Agents running on Kubernetes.

- name: processors
type: yaml
title: Processors
Expand All @@ -66,5 +67,6 @@ streams:
show_user: false
description: >
Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. This executes in the agent before the events are shipped. See [Processors](https://www.elastic.co/guide/en/fleet/current/elastic-agent-processor-configuration.html) for details.

title: AWS Billing Metrics
description: Collect billing metrics from Amazon Web Services with Elastic Agent.
5 changes: 4 additions & 1 deletion packages/aws/data_stream/cloudfront_logs/agent/stream/aws-s3.yml.hbs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,9 @@ api_timeout: {{api_timeout}}
{{#if max_number_of_messages}}
max_number_of_messages: {{max_number_of_messages}}
{{/if}}
{{#if endpoint}}
endpoint: {{endpoint}}
{{/if}}
{{#if default_region}}
default_region: {{default_region}}
{{/if}}
Expand Down Expand Up @@ -52,4 +55,4 @@ processors:
message: "^#.*"
{{#if processors}}
{{processors}}
{{/if}}
{{/if}}
1 change: 1 addition & 0 deletions packages/aws/data_stream/cloudfront_logs/fields/agent.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@
example: "stretch"
description: >
OS codename, if any.

- name: log.offset
type: long
description: Log offset
Expand Down
3 changes: 3 additions & 0 deletions packages/aws/data_stream/cloudtrail/agent/stream/aws-cloudwatch.yml.hbs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,9 @@ shared_credential_file: {{shared_credential_file}}
{{#if api_timeout}}
api_timeout: {{api_timeout}}
{{/if}}
{{#if endpoint}}
endpoint: {{endpoint}}
{{/if}}
{{#if default_region}}
default_region: {{default_region}}
{{/if}}
Expand Down
6 changes: 4 additions & 2 deletions packages/aws/data_stream/cloudtrail/agent/stream/aws-s3.yml.hbs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,9 @@ credential_profile_name: {{credential_profile_name}}
shared_credential_file: {{shared_credential_file}}
{{/if}}


{{#if endpoint}}
endpoint: {{endpoint}}
{{/if}}
{{#if default_region}}
default_region: {{default_region}}
{{/if}}
Expand Down Expand Up @@ -86,4 +88,4 @@ publisher_pipeline.disable_host: true
{{#if processors}}
processors:
{{processors}}
{{/if}}
{{/if}}
1 change: 1 addition & 0 deletions packages/aws/data_stream/cloudtrail/fields/agent.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@
example: "stretch"
description: >
OS codename, if any.

- name: log.offset
type: long
description: Log offset
3 changes: 3 additions & 0 deletions packages/aws/data_stream/cloudwatch_logs/agent/stream/aws-cloudwatch.yml.hbs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,9 @@ shared_credential_file: {{shared_credential_file}}
{{#if api_timeout}}
api_timeout: {{api_timeout}}
{{/if}}
{{#if endpoint}}
endpoint: {{endpoint}}
{{/if}}
{{#if default_region}}
default_region: {{default_region}}
{{/if}}
Expand Down
3 changes: 3 additions & 0 deletions packages/aws/data_stream/cloudwatch_metrics/agent/stream/stream.yml.hbs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,9 @@ shared_credential_file: {{shared_credential_file}}
{{#if role_arn}}
role_arn: {{role_arn}}
{{/if}}
{{#if endpoint}}
endpoint: {{endpoint}}
{{/if}}
{{#if default_region}}
default_region: {{default_region}}
{{/if}}
Expand Down
2 changes: 1 addition & 1 deletion packages/aws/data_stream/cloudwatch_metrics/fields/base-fields.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,4 +13,4 @@
- name: event.module
type: constant_keyword
description: Event module
value: aws
value: aws
5 changes: 4 additions & 1 deletion packages/aws/data_stream/dynamodb/agent/stream/stream.yml.hbs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,9 @@ shared_credential_file: {{shared_credential_file}}
{{#if role_arn}}
role_arn: {{role_arn}}
{{/if}}
{{#if endpoint}}
endpoint: {{endpoint}}
{{/if}}
{{#if default_region}}
default_region: {{default_region}}
{{/if}}
Expand Down Expand Up @@ -81,4 +84,4 @@ metrics:
- AccountMaxTableLevelWrites
- AccountMaxWrites
- MaxProvisionedTableReadCapacityUtilization
- MaxProvisionedTableWriteCapacityUtilization
- MaxProvisionedTableWriteCapacityUtilization
5 changes: 4 additions & 1 deletion packages/aws/data_stream/ebs/agent/stream/stream.yml.hbs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,9 @@ shared_credential_file: {{shared_credential_file}}
{{#if role_arn}}
role_arn: {{role_arn}}
{{/if}}
{{#if endpoint}}
endpoint: {{endpoint}}
{{/if}}
{{#if default_region}}
default_region: {{default_region}}
{{/if}}
Expand All @@ -50,4 +53,4 @@ metrics:
- namespace: AWS/EBS
statistic: ["Sum"]
name: ["VolumeTotalReadTime", "VolumeTotalWriteTime", "VolumeIdleTime"]
resource_type: ec2
resource_type: ec2
3 changes: 3 additions & 0 deletions packages/aws/data_stream/ec2_logs/agent/stream/aws-cloudwatch.yml.hbs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,9 @@ shared_credential_file: {{shared_credential_file}}
{{#if api_timeout}}
api_timeout: {{api_timeout}}
{{/if}}
{{#if endpoint}}
endpoint: {{endpoint}}
{{/if}}
{{#if default_region}}
default_region: {{default_region}}
{{/if}}
Expand Down
5 changes: 4 additions & 1 deletion packages/aws/data_stream/ec2_logs/agent/stream/aws-s3.yml.hbs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,9 @@ api_timeout: {{api_timeout}}
{{#if max_number_of_messages}}
max_number_of_messages: {{max_number_of_messages}}
{{/if}}
{{#if endpoint}}
endpoint: {{endpoint}}
{{/if}}
{{#if default_region}}
default_region: {{default_region}}
{{/if}}
Expand Down Expand Up @@ -48,4 +51,4 @@ publisher_pipeline.disable_host: true
{{#if processors}}
processors:
{{processors}}
{{/if}}
{{/if}}
1 change: 1 addition & 0 deletions packages/aws/data_stream/ec2_logs/fields/agent.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@
example: "stretch"
description: >
OS codename, if any.

- name: log.offset
type: long
description: Log offset
Expand Down
3 changes: 3 additions & 0 deletions packages/aws/data_stream/ec2_metrics/agent/stream/stream.yml.hbs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,9 @@ shared_credential_file: {{shared_credential_file}}
{{#if role_arn}}
role_arn: {{role_arn}}
{{/if}}
{{#if endpoint}}
endpoint: {{endpoint}}
{{/if}}
{{#if default_region}}
default_region: {{default_region}}
{{/if}}
Expand Down
6 changes: 6 additions & 0 deletions packages/aws/data_stream/ecs_metrics/agent/stream/stream.yml.hbs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,12 @@ shared_credential_file: {{shared_credential_file}}
{{#if role_arn}}
role_arn: {{role_arn}}
{{/if}}
{{#if endpoint}}
endpoint: {{endpoint}}
{{/if}}

@efd6 efd6 May 28, 2024

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see endpoint in the manifest. Also, is region intentionally not added here?

{{#if endpoint}}
endpoint: {{endpoint}}
{{/if}}
{{#if default_region}}
default_region: {{default_region}}
{{/if}}
Expand Down
6 changes: 6 additions & 0 deletions packages/aws/data_stream/elb_logs/agent/stream/aws-cloudwatch.yml.hbs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,12 @@ shared_credential_file: {{shared_credential_file}}
{{#if api_timeout}}
api_timeout: {{api_timeout}}
{{/if}}
{{#if endpoint}}
endpoint: {{endpoint}}
{{/if}}
{{#if endpoint}}
endpoint: {{endpoint}}
{{/if}}
{{#if default_region}}
default_region: {{default_region}}
{{/if}}
Expand Down
8 changes: 7 additions & 1 deletion packages/aws/data_stream/elb_logs/agent/stream/aws-s3.yml.hbs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,12 @@ api_timeout: {{api_timeout}}
{{#if max_number_of_messages}}
max_number_of_messages: {{max_number_of_messages}}
{{/if}}
{{#if endpoint}}
endpoint: {{endpoint}}
{{/if}}
{{#if endpoint}}
endpoint: {{endpoint}}
{{/if}}
{{#if default_region}}
default_region: {{default_region}}
{{/if}}
Expand Down Expand Up @@ -48,4 +54,4 @@ publisher_pipeline.disable_host: true
{{#if processors}}
processors:
{{processors}}
{{/if}}
{{/if}}
1 change: 1 addition & 0 deletions packages/aws/data_stream/elb_logs/fields/agent.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@
example: "stretch"
description: >
OS codename, if any.

- name: log.offset
type: long
description: Log offset
Expand Down
5 changes: 4 additions & 1 deletion packages/aws/data_stream/elb_metrics/agent/stream/stream.yml.hbs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,9 @@ shared_credential_file: {{shared_credential_file}}
{{#if role_arn}}
role_arn: {{role_arn}}
{{/if}}
{{#if endpoint}}
endpoint: {{endpoint}}
{{/if}}
{{#if default_region}}
default_region: {{default_region}}
{{/if}}
Expand Down Expand Up @@ -140,4 +143,4 @@ metrics:
statistic: ["Maximum"]
name:
- HealthyHostCount
- UnHealthyHostCount
- UnHealthyHostCount
3 changes: 3 additions & 0 deletions packages/aws/data_stream/emr_logs/agent/stream/aws-cloudwatch.yml.hbs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,9 @@ shared_credential_file: {{shared_credential_file}}
{{#if api_timeout}}
api_timeout: {{api_timeout}}
{{/if}}
{{#if endpoint}}
endpoint: {{endpoint}}
{{/if}}
{{#if default_region}}
default_region: {{default_region}}
{{/if}}
Expand Down
6 changes: 4 additions & 2 deletions packages/aws/data_stream/emr_logs/agent/stream/aws-s3.yml.hbs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,9 @@ file_selectors:

{{/if}}


{{#if endpoint}}
endpoint: {{endpoint}}
{{/if}}
{{#if default_region}}
default_region: {{default_region}}
{{/if}}
Expand Down Expand Up @@ -75,4 +77,4 @@ publisher_pipeline.disable_host: true
{{#if processors}}
processors:
{{processors}}
{{/if}}
{{/if}}
1 change: 1 addition & 0 deletions packages/aws/data_stream/emr_logs/fields/agent.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@
example: "stretch"
description: >
OS codename, if any.

- name: log.offset
type: long
description: Log offset
Loading