Skip to content

[8.x](backport #41022) [AWS] Use namespace for GetListMetrics when exists #41176

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
kaiyan-sheng merged 1 commit into from
Oct 8, 2024
Merged

[8.x](backport #41022) [AWS] Use namespace for GetListMetrics when exists #41176

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.next.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -203,6 +203,7 @@ https://github.com/elastic/beats/compare/v8.8.1\...main[Check the HEAD diff]
- Remove excessive info-level logs in cgroups setup {pull}40491[40491]
- Add missing ECS Cloud fields in GCP `metrics` metricset when using `exclude_labels: true` {issue}40437[40437] {pull}40467[40467]
- Add AWS OwningAccount support for cross account monitoring {issue}40570[40570] {pull}40691[40691]
- Use namespace for GetListMetrics when exists in AWS {pull}41022[41022]
- Fix http server helper SSL config. {pull}39405[39405]

*Osquerybeat*
Expand Down
3 changes: 2 additions & 1 deletion metricbeat/docs/modules/aws.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -329,7 +329,8 @@ GetMetricData max page size: 100, based on https://docs.aws.amazon.com/AmazonClo
| IAM ListAccountAliases | 1 | Once on startup
| STS GetCallerIdentity | 1 | Once on startup
| EC2 DescribeRegions| 1 | Once on startup
| CloudWatch ListMetrics | Total number of results / ListMetrics max page size | Per region per collection period
| CloudWatch ListMetrics without specifying namespace in configuration | Total number of results / ListMetrics max page size | Per region per collection period
| CloudWatch ListMetrics with specific namespaces in configuration | Total number of results / ListMetrics max page size * number of unique namespaces | Per region per collection period
| CloudWatch GetMetricData | Total number of results / GetMetricData max page size | Per region per namespace per collection period
|===
`billing`, `ebs`, `elb`, `sns`, `usage` and `lambda` are the same as `cloudwatch` metricset.
Expand Down
3 changes: 2 additions & 1 deletion x-pack/metricbeat/module/aws/_meta/docs.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -317,7 +317,8 @@ GetMetricData max page size: 100, based on https://docs.aws.amazon.com/AmazonClo
| IAM ListAccountAliases | 1 | Once on startup
| STS GetCallerIdentity | 1 | Once on startup
| EC2 DescribeRegions| 1 | Once on startup
| CloudWatch ListMetrics | Total number of results / ListMetrics max page size | Per region per collection period
| CloudWatch ListMetrics without specifying namespace in configuration | Total number of results / ListMetrics max page size | Per region per collection period
| CloudWatch ListMetrics with specific namespaces in configuration | Total number of results / ListMetrics max page size * number of unique namespaces | Per region per collection period
| CloudWatch GetMetricData | Total number of results / GetMetricData max page size | Per region per namespace per collection period
|===
`billing`, `ebs`, `elb`, `sns`, `usage` and `lambda` are the same as `cloudwatch` metricset.
Expand Down
20 changes: 16 additions & 4 deletions x-pack/metricbeat/module/aws/cloudwatch/cloudwatch.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -179,10 +179,22 @@ func (m *MetricSet) Fetch(report mb.ReporterV2) error {
continue
}

// retrieve all the details for all the metrics available in the current region
listMetricsOutput, err := aws.GetListMetricsOutput("*", regionName, m.Period, m.IncludeLinkedAccounts, m.OwningAccount, m.MonitoringAccountID, svcCloudwatch)
if err != nil {
m.Logger().Errorf("Error while retrieving the list of metrics for region %s: %w", regionName, err)
// retrieve all the details for all the metrics available in the current region when no namespace is specified
// otherwise only retrieve metrics from the specific namespaces from the config
var listMetricsOutput []aws.MetricWithID
if len(namespaceDetailTotal) == 0 {
listMetricsOutput, err = aws.GetListMetricsOutput("*", regionName, m.Period, m.IncludeLinkedAccounts, m.OwningAccount, m.MonitoringAccountID, svcCloudwatch)
if err != nil {
m.Logger().Errorf("Error while retrieving the list of metrics for region %s and namespace %s: %w", regionName, "*", err)
}
} else {
for namespace := range namespaceDetailTotal {
listMetricsOutputPerNamespace, err := aws.GetListMetricsOutput(namespace, regionName, m.Period, m.IncludeLinkedAccounts, m.OwningAccount, m.MonitoringAccountID, svcCloudwatch)
if err != nil {
m.Logger().Errorf("Error while retrieving the list of metrics for region %s and namespace %s: %w", regionName, namespace, err)
}
listMetricsOutput = append(listMetricsOutput, listMetricsOutputPerNamespace...)
}
}

if len(listMetricsOutput) == 0 {
Expand Down