elastic-agent: update monitor fields to include new TBS metrics

[ericywl]

Proposed commit message

Add back apm-server monitoring metrics removed in #12793, plus new TBS metrics outlined in elastic/apm-server#15533

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

• Verify mappings are present in the metrics-elastic_agent.apm_server* datastream.

How to test this PR locally

Step 1: Ensure Elasticsearch & Kibana is running

1. Spin up a stack in QA: https://console.qa.cld.elstc.co/

> cd integrations/packages/elastic_agent

> export ELASTIC_PACKAGE_KIBANA_HOST=https://alice-863450.kb.us-central1.gcp.qa.cld.elstc.co
> export ELASTIC_PACKAGE_ELASTICSEARCH_HOST=https://alice-863450.es.us-central1.gcp.qa.cld.elstc.co
> export ELASTIC_PACKAGE_ELASTICSEARCH_USERNAME="elastic"
> export ELASTIC_PACKAGE_ELASTICSEARCH_PASSWORD="xxx"

> elastic-package stack up -v -d --provider environment

2. Check stack health

> elastic-package stack status

Status of Elastic stack services:
╭───────────────┬─────────┬───────────────────╮
│    SERVICE    │ VERSION │      STATUS       │
├───────────────┼─────────┼───────────────────┤
│ elasticsearch │ 9.2.1   │ healthy           │
│ kibana        │ 9.2.1   │ healthy           │
│ fleet-server  │ unknown │ healthy           │
│ elastic-agent │ 9.2.1   │ running (healthy) │
╰───────────────┴─────────┴───────────────────╯

Step 2: Apply Integration Changes

• Install the elastic_agent integration with the updated mappings in this PR

> cd integrations/packages/elastic_agent

> elastic-package build
> elastic-package stack up -v -d --services package-registry
> elastic-package install -v

Step 3: Enroll Elastic Agent in Fleet

1. Go to the QA Kibana UI
2. Data management -> Integrations -> Installed integrations -> Elastic APM -> Add Elastic APM
3. Add Elastic Agent to your hosts -> Add agent -> Go to Step 4

Step 4: Install EA Locally in a Container

1. Create docker container to run EA in.

> docker run --rm -it -p 8220:8220 -p 8200:8200 buildpack-deps:24.04 /bin/bash

2. From within the container, update and add neovim to edit apm-server.yml later on.

> apt update -y && apt install -y neovim

4. Install elastic-agent with commands from Kibana UI listed in Step 3.

> curl -L -O https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-9.2.1-linux-arm64.tar.gz 
> tar xzvf elastic-agent-9.2.1-linux-arm64.tar.gz
> cd elastic-agent-9.2.1-linux-arm64

> ./elastic-agent install --url=<REDACTED> --enrollment-token=<REDACTED> --install-servers

5. Check the Elastic Agent health

> elastic-agent status

┌─ fleet
│  └─ status: (HEALTHY) Connected
└─ elastic-agent
   └─ status: (HEALTHY) Running

Step 5: Verify Data

Verify mappings in the metrics-elastic_agent.apm_server* datastream from Dev Tools.

1. Get the index name by running the cmd below and copying the data_streams.indices.index_name field.

GET _data_stream/metrics-elastic_agent.apm_server*

2. Contains the proper mappings due to the changes made in this PR.

GET .ds-metrics-elastic_agent.apm_server-default-2025.11.19-000001/_mapping?filter_path=**.apm_server.**.storage

{
  ".ds-metrics-elastic_agent.apm_server-default-2025.11.19-000002": {
    "mappings": {
      "properties": {
        "beat": {
          "properties": {
            "stats": {
              "properties": {
                "apm_server": {
                  "properties": {
                    "sampling": {
                      "properties": {
                        "tail": {
                          "properties": {
                            "storage": {
                              "properties": {
                                "disk_total": {
                                  "type": "long",
                                  "meta": {
                                    "unit": "byte"
                                  },
                                  "time_series_metric": "gauge"
                                },
                                "disk_usage_threshold": {
                                  "type": "long",
                                  "time_series_metric": "gauge"
                                },
                                "disk_used": {
                                  "type": "long",
                                  "meta": {
                                    "unit": "byte"
                                  },
                                  "time_series_metric": "gauge"
                                },
                                "lsm_size": {
                                  "type": "long",
                                  "meta": {
                                    "unit": "byte"
                                  },
                                  "time_series_metric": "gauge"
                                },
                                "storage_limit": {
                                  "type": "long",
                                  "meta": {
                                    "unit": "byte"
                                  },
                                  "time_series_metric": "gauge"
                                },
                                "value_log_size": {
                                  "type": "long",
                                  "meta": {
                                    "unit": "byte"
                                  },
                                  "time_series_metric": "gauge"
                                }
                              }
                            }
                          }
                        }
                      }
                    }
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}

Related issues

• Relates monitoring: expose storage limit and disk metrics for TBS monitoring apm-server#15533

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[elastic-vault-github-plugin-prod]

🚀 Benchmarks report

Package elastic_agent 👍(0) 💚(1) 💔(1)

Expand to view

Data stream	Previous EPS	New EPS	Diff (%)	Result
elastic_agent_logs	37037.04	31250	-5787.04 (-15.63%)	💔

To see the full report comment with /test benchmark fullreport

[belimawr]

@ericywl, it seems you forgot to update the version in manifest.yml and the changelog (changelog.yml) as added as belonging to the currently released version.

Could you update those two files?

[ericywl]

@ericywl, it seems you forgot to update the version in manifest.yml and the changelog (changelog.yml) as added as belonging to the currently released version.

My bad, not familiar with this, let me update them

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 7002a7e

History

• 💚 Build #35663 succeeded 96253c2
• 💚 Build #35491 succeeded 7f1bc91

cc @ericywl