Add host inventory metrics to ec2 metricset

[kaiyan-sheng]

What does this PR do?

This PR is to add proposed host common fields into ec2 metricset:

• host.id
• host.name
• host.cpu.pct
• host.network.in.bytes
• host.network.in.packets
• host.network.out.bytes
• host.network.out.packets
• host.disk.read.bytes
• host.disk.write.bytes

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

1. Start Metricbeat AWS module: ./metricbeat modules enable aws
2. Edit modules.d/aws.yml to only include ec2 metricset:

- module: aws
  period: 5m
  credential_profile_name: elastic-beats
  metricsets:
    - ec2

3. Change add_host_metadata processor config in metricbeat.yml file:

processors:
  - add_host_metadata:
      replace_fields: false

4. Start metricbeat
5. You should see metrics from ec2 metricset and includes fields listed above.
   (host.name only equals to EC2 instance name when EC2 instance name is available.)

Event Example

{
    "@timestamp": "2017-10-12T08:05:34.853Z",
    "aws": {
        "ec2": {
            "cpu": {
                "credit_balance": 144,
                "credit_usage": 0.05477,
                "surplus_credit_balance": 0,
                "surplus_credits_charged": 0,
                "total": {
                    "pct": 1.166484417714406
                }
            },
            "diskio": {
                "read": {
                    "bytes": 0,
                    "bytes_per_sec": 0,
                    "count": 0,
                    "count_per_sec": 0
                },
                "write": {
                    "bytes": 0,
                    "bytes_per_sec": 0,
                    "count": 0,
                    "count_per_sec": 0
                }
            },
            "instance": {
                "core": {
                    "count": 1
                },
                "image": {
                    "id": "ami-04bc3da8f14823e88"
                },
                "monitoring": {
                    "state": "disabled"
                },
                "private": {
                    "dns_name": "ip-172-31-9-119.us-west-1.compute.internal",
                    "ip": "172.31.9.119"
                },
                "public": {
                    "dns_name": "ec2-13-52-163-56.us-west-1.compute.amazonaws.com",
                    "ip": "13.52.163.56"
                },
                "state": {
                    "code": 16,
                    "name": "running"
                },
                "threads_per_core": 1
            },
            "network": {
                "in": {
                    "bytes": 6644.4,
                    "bytes_per_sec": 22.148,
                    "packets": 44.6,
                    "packets_per_sec": 0.14866666666666667
                },
                "out": {
                    "bytes": 6016.6,
                    "bytes_per_sec": 20.055333333333333,
                    "packets": 35.4,
                    "packets_per_sec": 0.118
                }
            },
            "status": {
                "check_failed": 0,
                "check_failed_instance": 0,
                "check_failed_system": 0
            }
        },
        "tags": {
            "Name": "mysql-test",
            "created-by": "ks"
        }
    },
    "cloud": {
        "account": {
            "id": "428152502467",
            "name": "elastic-beats"
        },
        "availability_zone": "us-west-1b",
        "instance": {
            "id": "i-0516ddaca5c1d231f",
            "name": "mysql-test"
        },
        "machine": {
            "type": "t2.micro"
        },
        "provider": "aws",
        "region": "us-west-1"
    },
    "event": {
        "dataset": "aws.ec2",
        "duration": 115000,
        "module": "aws"
    },
    "host": {
        "cpu": {
            "pct": 1.166484417714406
        },
        "disk": {
            "read": {
                "bytes": 0
            }
        },
        "diskio": {
            "write": {
                "bytes": 0
            }
        },
        "id": "i-0516ddaca5c1d231f",
        "name": "mysql-test",
        "network": {
            "in": {
                "bytes": 6644.4,
                "packets": 44.6
            },
            "out": {
                "bytes": 6016.6,
                "packets": 35.4
            }
        }
    },
    "metricset": {
        "name": "ec2",
        "period": 10000
    },
    "service": {
        "type": "aws"
    }
}

[elasticmachine]

💚 Build Succeeded

Expand to view the summary

Build stats

• Build Cause: [Pull request #20171 updated]

• Start Time: 2020-08-18T16:35:20.865+0000

• Duration: 69 min 42 sec

Test stats 🧪

Test	Results
Failed	0
Passed	3541
Skipped	794
Total	4335

[elasticmachine]

Pinging @elastic/integrations-platforms (Team:Platforms)

[jsoriano]

This is looking good.

[jsoriano]

Is this a counter? Why is it a decimal number?

[kaiyan-sheng]

It's a gauge actually and it's an average value among 5 minutes:

[jsoriano]

There are some issues with documented fields, for the rest it is looking good!

Issues fixed

[jsoriano]

LGTM if CI gets green.

[sorantis]

@kaiyan-sheng, @jsoriano, in the following configuration

processors:
  - add_host_metadata:
      replace_fields: false

what does the replace_fields flag do?

[jsoriano]

@kaiyan-sheng, @jsoriano, in the following configuration

processors:
  - add_host_metadata:
      replace_fields: false

what does the replace_fields flag do?

It can be used to control if add_host_metadata should replace existing host fields. It defaults to true, so this processor overwrites existing host fields, and can be set to false to avoid this overwrite, that is what we are going to want with inventory fields added by modules. We added this option to support inventory fields, while avoiding the introduction of a breaking change by now.

Full discussion and change to add this flag can be found here: #20490

[sorantis]

ok, my expectation was that this field would either map or remove the respective ec2.* fields in favor of host.* fields.
Because the resulting document contains duplicate metrics (e.g.: ec2.cpu.total.pct = host.cpu.pct), which increases the doc size by 10-11%

[kaiyan-sheng]

@sorantis Currently we are duplicating the fields so yes we have both ec2.* fields and host.* fields. But I'm planning to create a PR to remove the duplicated ec2.* fields and use alias to keep backward compatibility. For example:

- name: aws.ec2
  type: group
  fields:
    - name: cpu.total.pct
      type: alias
      path: host.cpu.pct
      migration: true
    - name: network.in.bytes
      type: alias
      path: host.network.in.bytes
      migration: true

[sorantis]

Great! Thanks @kaiyan-sheng .