Skip to content

[Metricbeat] Add latency config option into aws module #20875

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 6 commits into from
Oct 5, 2020
Merged

[Metricbeat] Add latency config option into aws module #20875

merged 6 commits into from
Oct 5, 2020

Conversation

@kaiyan-sheng
Copy link
Contributor

@kaiyan-sheng kaiyan-sheng commented Aug 31, 2020 •
edited
Loading

What does this PR do?

This PR adds the latency config parameter for aws module. With this config parameter, users can collect CloudWatch metrics with a latency larger than Metricbeat collection period. For example, for S3 request metrics, the latency can be 3 minutes. The recommended collection period is 1min. Without the latency config parameter, no data points will be collected ever.

In this screenshot: current timestamp is 14:21 but we only see the latest data at 14:18 with a 3-minute delay. In this case, without this fix, our Metricbeat module will keep trying to collect data from the last 2-minute time frame, which will always be empty:
Screenshot 2020-09-03 at 16 22 19

(@gbanasiak Thank you for your screenshot!)

Why is it important?

latency config sets CloudWatch API start time and end time with a delay. For example, if the current time is 2020-08-31T18:30:00.000Z with collection period 1m.

Without latency:
startTime = 2020-08-31T18:29:00.000Z
endTime = 2020-08-31T18:30:00.000Z

With latency set to 5min:
startTime = 2020-08-31T18:24:00.000Z
endTime = 2020-08-31T18:25:00.000Z

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

  1. Enable aws module: ./metricbeat modules enable aws
  2. Change modules.d/aws.yml to:
- module: aws
  period: 5m
  credential_profile_name: elastic-beats
  latency: 5m
  metricsets:
    - ec2
  1. Start Metricbeat with debug level logging: ./metricbeat -e -d "*"
  2. You should see the startTime and endTime in debug log with a latency of 5 minutes applied.

@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Aug 31, 2020
@kaiyan-sheng kaiyan-sheng self-assigned this Aug 31, 2020
@kaiyan-sheng kaiyan-sheng added review Team:Platforms Label for the Integrations - Platforms team labels Aug 31, 2020
@elasticmachine
Copy link
Collaborator

elasticmachine commented Aug 31, 2020

Pinging @elastic/integrations-platforms (Team:Platforms)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Aug 31, 2020
@elasticmachine
Copy link
Collaborator

elasticmachine commented Aug 31, 2020 •
edited by jenkins-beats-ci bot
Loading

💚 Build Succeeded

Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: [Pull request #20875 updated]

  • Start Time: 2020-09-14T03:57:17.800+0000

  • Duration: 76 min 52 sec

Test stats 🧪

Test Results
Failed 0
Passed 3560
Skipped 795
Total 4355

ChrsMark
ChrsMark reviewed Sep 4, 2020
View reviewed changes
Copy link
Member

@ChrsMark ChrsMark left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm! just one comment regarding docs.

@kaiyan-sheng kaiyan-sheng added the needs_backport PR is waiting to be backported to other branches. label Oct 3, 2020
ChrsMark
ChrsMark approved these changes Oct 5, 2020
View reviewed changes
Copy link
Member

@ChrsMark ChrsMark left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@kaiyan-sheng kaiyan-sheng merged commit 7400f43 into elastic:master Oct 5, 2020
@kaiyan-sheng kaiyan-sheng deleted the add_latency_aws branch October 5, 2020 13:30
@kaiyan-sheng kaiyan-sheng mentioned this pull request Oct 5, 2020
Merged
v1v added a commit to v1v/beats that referenced this pull request Oct 5, 2020
@v1v
Merge remote-tracking branch 'upstream/master' into feature/windows-2016
07afe0b 
* upstream/master: (26 commits)
  [Ingest Manager] Send updating state (elastic#21461)
  [Filebeat][New Fileset] Cisco Umbrella support (elastic#21504)
  [Ingest Manager] Download asc from artifact store specified in spec (elastic#21488)
  Implementation of fileProspector (elastic#21479)
  [Metricbeat] Add latency config option into aws module (elastic#20875)
  Skip filestream flaky tests (elastic#21490)
  Ignore unsupported metrics in the azure module (elastic#21486)
  Do not run symlink tests on Windows (elastic#21472)
  Map `cloud.account.id` to azure sub id (elastic#21483)
  Add support for app_state metricset (elastic#20639)
  Include original error when metricbeat fails to connect with Kafka (elastic#21484)
  Prompt only when agent is already enrolled (elastic#21473)
  Fix leftover delpoyment example (elastic#21474)
  Bump version to ECS 1.6 in modules without ECS updates (elastic#21455)
  Clarify input type configuration options (elastic#19284)
  Increase index pattern size check to 10MiB (elastic#21487)
  Migrate S3 Input to Filebeat Input V2 (elastic#20005)
  [libbeat] Add configurable exponential backoff for disk queue write errors (elastic#21493)
  Revert "Revert "[JJBB] Set shallow cloning to 10 (elastic#21409)" (elastic#21447)" (elastic#21467)
  Fix format of debug messages in tlscommon (elastic#21482)
  ...
kvch pushed a commit that referenced this pull request Oct 5, 2020
@kaiyan-sheng
Fix billing.go aws.GetStartTimeEndTime (#21531)
931e3ca 
This PR is to fix this error introduced in #20875:
```
# github.com/elastic/beats/v7/x-pack/metricbeat/module/aws/billing
../x-pack/metricbeat/module/aws/billing/billing.go:119:47: not enough arguments in call to "github.com/elastic/beats/v7/x-pack/metricbeat/module/aws".GetStartTimeEndTime
	have (time.Duration)
	want (time.Duration, time.Duration)
Error: error getting default metricsets: Error running subcommand to get metricsets: running "go run /home/travis/gopath/src/github.com/elastic/beats/x-pack/metricbeat/scripts/msetlists/main.go" failed with exit code 2
```
@kaiyan-sheng kaiyan-sheng mentioned this pull request Oct 5, 2020
Merged
6 tasks
@kaiyan-sheng kaiyan-sheng added v7.10.0 and removed needs_backport PR is waiting to be backported to other branches. labels Oct 5, 2020
kaiyan-sheng added a commit that referenced this pull request Oct 5, 2020
@kaiyan-sheng
Cherry-pick #20875 to 7.x: [Metricbeat] Add latency config option int…
e75e3a4 
…o aws module (#21537)

(cherry picked from commit 7400f43)

* cherrypick #21531 to fix billing.go
@kaiyan-sheng kaiyan-sheng mentioned this pull request Oct 13, 2020
Merged
2 tasks
@gbanasiak gbanasiak mentioned this pull request Sep 27, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ChrsMark ChrsMark ChrsMark approved these changes

+1 more reviewer

@gbanasiak gbanasiak gbanasiak left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

@kaiyan-sheng kaiyan-sheng

Labels

review Team:Platforms Label for the Integrations - Platforms team v7.10.0

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@kaiyan-sheng @elasticmachine @ChrsMark @gbanasiak