[Metricbeat] Fix CPU/disk/network visualizations in EC2 overview dashboard

[zmoog]

Proposed commit message

Updates the CPU/disk/network metric field names in the "[Metricbeat AWS] EC2 Overview" dashboard.

The "[Metricbeat AWS] EC2 Overview" dashboard that comes with the AWS modules was still using the old metric field names that are no longer in use, resulting in a dashboard with multiple empty visualizations.

Here are the current changes:

Old	New	Formatter
aws.ec2.cpu.total.pct	host.cpu.usage	percent
aws.ec2.diskio.read.bytes	host.disk.read.bytes	bytes
aws.ec2.diskio.write.bytes	host.disk.write.bytes	bytes
aws.ec2.network.in.bytes	host.network.ingress.bytes	bytes
aws.ec2.network.out.bytes	host.network.egress.bytes	bytes

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

• Update the CPU usage visualization
• Update the disk visualization
• Update the network visualization

How to test this PR locally

Build Metricbeat

cd x-pack/metricbeat/

mage build

Enable AWS module

./metricbeat modules enable aws

And update modules.d/aws.yml to only collect EC2 metrics.

Set up cloud.id and cloud.auth to access Elastic Cloud, and deploy the dashboards

./metricbeat setup -E setup.dashboards.directory=build/kibana

Start Metricbeat to collect metric values to visualize in the dashboard.

Missing host.disk.*.bytes values?

If the disk read|write visualizations are empty, it means there are not EC2 instances with instance store volumes running, so we're not getting the fields host.disk.read.bytes and host.disk.write.bytes.

If you need to set up an EC2 instance for getting, you have to:

1. Launch and EC2 instance with an instance store volume (I used an m5d.large, see https://docs.aws.amazon.com/ec2/latest/instancetypes/gp.html#gp_instance-store for more options.
2. SSH into the EC2 instance
3. Identify, format and mount the disk.
4. Write and read data.

Here's what I did following https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/making-instance-stores-available-on-your-instances.html

$ lsblk
NAME          MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
nvme1n1       259:0    0 69.8G  0 disk
nvme0n1       259:1    0    8G  0 disk
├─nvme0n1p1   259:2    0    8G  0 part /
├─nvme0n1p127 259:3    0    1M  0 part
└─nvme0n1p128 259:4    0   10M  0 part /boot/efi

sudo mkfs -t xfs /dev/nvme1n1
sudo mkdir /data
sudo mount /dev/nvme1n1 /data

# Generate many small files
for i in {1..1000}; do
    echo "Test data $i" > /data/file_$i.txt
done

# Read them back
cat /data/file_* > /dev/null

Related issues

• N/A

Screenshots

Here are the CPU usage, disk, network visualization displaying data.

[github-actions]

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

• run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @zmoog? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit
• backport-active-all is the label that automatically backports to all active branches.
• backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
• backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

[elasticmachine]

Pinging @elastic/obs-ds-hosted-services (Team:obs-ds-hosted-services)

[Kavindu-Dodan]

LGTM & thank you for working on this :)

[kaiyan-sheng]

Tested the dashboards and they do work now. Thanks for fixing it!

[github-actions]

@Mergifyio backport 8.17 8.18 8.19 9.0 9.1

[mergify]

backport 8.17 8.18 8.19 9.0 9.1

✅ Backports have been created

Details

• #45246 [8.17](backport #45117) [Metricbeat] Fix CPU/disk/network visualizations in EC2 overview dashboard has been created for branch 8.17
• #45247 [8.18](backport #45117) [Metricbeat] Fix CPU/disk/network visualizations in EC2 overview dashboard has been created for branch 8.18
• #45248 [8.19](backport #45117) [Metricbeat] Fix CPU/disk/network visualizations in EC2 overview dashboard has been created for branch 8.19
• #45249 [9.0](backport #45117) [Metricbeat] Fix CPU/disk/network visualizations in EC2 overview dashboard has been created for branch 9.0
• #45250 [9.1](backport #45117) [Metricbeat] Fix CPU/disk/network visualizations in EC2 overview dashboard has been created for branch 9.1