Migrate metricsets in aws module to use cloudwatch input as lightweight module

[kaiyan-sheng]

In aws module we have several metricsets that are not lightweight modules. This issue is to track the work for converting them into lightweight modules. There are some problems we are facing:

1. Some metricsets are not only calling CloudWatch API. For example, billing metricset is also making CostExplorer API and ec2 metricset is also making DescribeInstances API.

• * billing: also uses CostExplorer API
• * ec2: also uses EC2 DescribeInstances API Migrate ec2 metricset to use cloudwatch input as light weight module #25924
• * rds: also uses RDS DescribeDBInstances API Migrate rds metricset to use cloudwatch input as light weight module #26077
• * s3_daily_storage: easy migration Move s3_daily_storage and s3_request metricsets to use cloudwatch input #21703
• * s3_request: easy migration Move s3_daily_storage and s3_request metricsets to use cloudwatch input #21703
• * sqs: also uses SQS ListQueues API Migrate sqs metricset to use cloudwatch input as light weight module #26117

2. How to keep backwards compatibility with metric names. For example, after using lightweight module, aws.ec2.cpu.total.pct will be aws.ec2. CPUUtilization.avg. @jsoriano suggested that fields can be renamed using processors. This will be a good option to keep both names instead of making a breaking change.

This is a meta issue to track the work for migrating all listed metricsets above to use lightweight modules.

cc: @exekias @jsoriano @sorantis

[elasticmachine]

Pinging @elastic/integrations-platforms (Team:Platforms)

[kaiyan-sheng]

Migrate ec2 metricset: #25924

[kaiyan-sheng]

Migrate rds metricset: #26077

[kaiyan-sheng]

Migrate sqs metricset: #26117

[kaiyan-sheng]

Update: For ec2, rds and sqs, we are adding metadata from separate API calls into the events that are populated from CloudWatch API response. billing metricset in aws module is not the same as other metricsets because it is not adding metadata into the same events. Instead, billing is creating two sets of events: from CloudWatch and from Cost Explorer.
I will keep billing metricset as it is for now. Maybe we should think about separating billing into its separate module in the future.