[Infra] Fix Hosts filter options to match selected schema by rmyz · Pull Request #259825 · elastic/kibana

rmyz · 2026-03-26T16:23:23Z

Summary

Fixes the filter controls on the Hosts page so that suggested values match the selected data schema (ECS vs semconv), and adds a semconv+APM synthtrace scenario for testing.

Problem

The Options List filter controls (Operating System, Cloud Provider, Service Name) showed suggestions from all documents in the metrics data view regardless of the selected schema. Selecting a filter value from the wrong schema produced zero results.

Additionally, the semconv OS filter used os.name which doesn't exist in OTel data, and there was no synthtrace scenario to test semconv hosts with APM services.

Changes

Filter scoping (controls_content.tsx)

Adds a schemaFilters memo that injects a schema-specific Elasticsearch filter into the ControlGroupRenderer
The filter is a bool.should combining:
- The schema's metrics node filter (e.g., event.module: system for ECS, data_stream.dataset: hostmetricsreceiver.otel for semconv)
- The schema's APM transaction dataset filter (e.g., data_stream.dataset: apm.transaction.* for ECS, data_stream.dataset: transaction.*.otel for semconv)
This ensures OS and Cloud Provider suggestions come from schema-specific metrics, and Service Name from schema-specific APM data

OS field fix (constants.ts, control_panels_config.ts)

Changed the semconv OS filter field from os.name (non-existent) to os.type, which is the actual field populated by the OTel collector

New synthtrace scenario (infra_hosts_semconv_with_apm_hosts.ts)

Generates semconv host metrics (CPU, memory, disk, network) combined with OTel APM traces
Half the instances have host metrics, the rest are APM-only (matching the ECS variant's structure)
Uses ApmSynthtracePipelineSchema.Otel for APM data so service.name appears under semconv filters

Semconv metric field alignment

Added metrics.*-prefixed fields alongside non-prefixed ones in both semconv scenarios (e.g., metrics.system.cpu.utilization alongside system.cpu.utilization). In real OTel indices, the metrics.* prefix is added automatically via passthrough mappings, but synthtrace bypasses those templates
Fixed disk metrics from a single metrics.system.filesystem.utilization value to state-based metrics.system.filesystem.usage docs (matching what the Lens formula expects)

APM service name collision fix

Changed synth-node-${index % 3} to synth-node-${index} in the ECS scenario to avoid version_conflict_engine_exception on service_summary metrics

Demo

Before

Kapture.2026-03-27.at.11.02.04.mp4

After

ECS

Kapture.2026-03-27.at.10.30.57.mp4

SEMCONV

Kapture.2026-03-27.at.10.58.22.mp4

Testing

Start Kibana and ingest both scenarios (separately, not at the same time):

node scripts/synthtrace infra_hosts_with_apm_hosts --live --clean
node scripts/synthtrace infra_hosts_semconv_with_apm_hosts --live --clean

Navigate to Infrastructure → Hosts
Try both ECS and semconv schemas
Verify:
- Filter dropdown options match the selected schema
- Applying a filter returns matching results

The Options List filter controls on the Hosts page showed suggestions from all documents in the metrics data view regardless of the selected schema (ECS vs semconv), leading to filter values that produced zero results when applied. This change: - Fixes the semconv OS filter field from `os.name` (non-existent) to `os.type` which is the actual field populated by the OTel collector - Adds a schema-scoped filter to the ControlGroupRenderer that restricts suggestions to documents matching the selected schema's metrics (nodeFilter) or APM transaction data (dataset-based filter) - ECS controls query: system integration docs OR `apm.transaction.*` docs - semconv controls query: hostmetricsreceiver docs OR `transaction.*.otel` docs Closes elastic#256158 Made-with: Cursor

The infra routing transform routed all docs with `metricset.name: cpu` (etc.) to `metrics-system.*-default` regardless of schema. When both ECS and semconv scenarios were ingested, semconv hosts leaked into the ECS view because they shared indices. - Add early return in routing transform for docs with pre-set `_index` - Set `_index: metrics-generic.otel-default` on semconv scenario docs - Add `metrics-generic.otel*` to infraEsClient cleanup data streams - Fix semconv metric field names (add `metrics.*` prefix for Lens) - Fix disk metrics to use state-based `metrics.system.filesystem.usage` Made-with: Cursor

rmyz · 2026-03-27T10:03:59Z

/ci

rmyz · 2026-03-27T10:04:23Z

@coderabbitai full review

coderabbitai · 2026-03-27T10:04:28Z

✅ Actions performed

Full review triggered.

Copilot

Pull request overview

Fixes Hosts page filter suggestions so Options List controls are scoped to the selected schema (ECS vs semconv), and adds synthtrace coverage for semconv hosts with OTel APM services.

Changes:

Injects a schema-specific ES filter into the Hosts Control Group to scope suggestions to schema-appropriate metrics/APM datasets.
Updates semconv OS control to use os.type instead of the non-existent os.name.
Adds a new semconv + OTel APM synthtrace scenario and adjusts existing ECS scenario to avoid APM service name collisions.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
x-pack/solutions/observability/plugins/metrics_data_access/common/index.ts	Re-exports the semconv OS field constant as `OS_TYPE`.
x-pack/solutions/observability/plugins/metrics_data_access/common/constants.ts	Changes semconv OS field constant from `os.name` to `os.type`.
x-pack/solutions/observability/plugins/infra/public/pages/metrics/hosts/components/search_bar/controls_content.tsx	Adds schema-scoped filters to ControlGroupRenderer so suggestions align with ECS/semconv selection.
x-pack/solutions/observability/plugins/infra/public/pages/metrics/hosts/components/search_bar/control_panels_config.ts	Switches semconv OS control to `OS_TYPE` and updates replacement mapping for schema switching.
src/platform/packages/shared/kbn-synthtrace/src/scenarios/infra_hosts_with_apm_hosts.ts	Avoids APM service-name collisions by making service names unique per instance.
src/platform/packages/shared/kbn-synthtrace/src/scenarios/infra_hosts_semconv_with_apm_hosts.ts	Adds a new semconv scenario combining host metrics with OTel APM traces.
src/platform/packages/shared/kbn-synthtrace/src/scenarios/infra_hosts_semconv.ts	Aligns semconv host docs by adding `os.type`.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

...vability/plugins/infra/public/pages/metrics/hosts/components/search_bar/controls_content.tsx

src/platform/packages/shared/kbn-synthtrace/src/scenarios/infra_hosts_semconv_with_apm_hosts.ts

rmyz · 2026-03-27T10:19:46Z

/ci

coderabbitai · 2026-03-27T10:34:29Z

✅ Actions performed

Full review triggered.

coderabbitai · 2026-03-27T10:38:06Z

📝 Walkthrough

Walkthrough

This pull request modifies Synthtrace data generation scenarios and updates the infrastructure hosts metrics filter UI to use OpenTelemetry semantic convention fields more consistently. Changes include adding os.type metadata to generated documents, introducing a new scenario that correlates infrastructure metrics with APM traces, updating control panel configurations from OS_NAME to OS_TYPE constants, and implementing schema-aware filtering logic in the search control component. Supporting updates include new package dependencies, TypeScript configuration adjustments, and APM service naming modifications for instance uniqueness.

🚥 Pre-merge checks | ✅ 2

✅ Passed checks (2 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	PR addresses all core requirements from `#256158`: scoped filters to selected schema, fixed semconv OS field from os.name to os.type, added schema-filtered APM dataset clauses, and new synthtrace scenario for testing.
Out of Scope Changes check	✅ Passed	All changes align with issue objectives: filter scoping, semconv schema fixes, synthtrace scenarios for validation, and supporting config updates in tsconfig/constants.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

🛠️ Update Documentation: Commit on current branch
🛠️ Update Documentation: Create PR

Warning

Tools execution failed with the following error:

Failed to run tools: 13 INTERNAL: Received RST_STREAM with code 2 (Internal server error)

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

rmyz · 2026-03-27T10:42:11Z

/ci

coderabbitai

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In
`@src/platform/packages/shared/kbn-synthtrace/src/scenarios/infra_hosts_semconv_with_apm_hosts.ts`:
- Around line 49-50: The os.type fields are set to the distro 'ubuntu' but must
use the OS family 'linux'; update both occurrences of the key names shown in
this diff ('resource.attributes.os.type' and 'os.type') in the
infra_hosts_semconv_with_apm_hosts scenario so their values are 'linux'
(matching infra_hosts_semconv.ts) to avoid filter mismatches.
- Around line 133-147: The networkDocs entries currently only set
'system.network.io' but should also include the Lens-compatible prefixed field;
update the mapping that creates networkDocs (the array mapped into objects with
...base, ...net, 'metricset.name': 'network', 'device.keyword': 'eth0') to also
set 'metrics.system.network.io' to the same value as 'system.network.io' for
both transmit and receive docs so they match CPU/memory/disk patterns.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository YAML (base), Organization UI (inherited)

Review profile: CHILL

Plan: Pro

Run ID: b8dffe3d-e8fb-4bdd-aacf-bf5d471d6f79

📥 Commits

Reviewing files that changed from the base of the PR and between c63ea72 and 554a996.

📒 Files selected for processing (9)

src/platform/packages/shared/kbn-synthtrace/src/scenarios/infra_hosts_semconv.ts
src/platform/packages/shared/kbn-synthtrace/src/scenarios/infra_hosts_semconv_with_apm_hosts.ts
src/platform/packages/shared/kbn-synthtrace/src/scenarios/infra_hosts_with_apm_hosts.ts
x-pack/solutions/observability/plugins/infra/moon.yml
x-pack/solutions/observability/plugins/infra/public/pages/metrics/hosts/components/search_bar/control_panels_config.ts
x-pack/solutions/observability/plugins/infra/public/pages/metrics/hosts/components/search_bar/controls_content.tsx
x-pack/solutions/observability/plugins/infra/tsconfig.json
x-pack/solutions/observability/plugins/metrics_data_access/common/constants.ts
x-pack/solutions/observability/plugins/metrics_data_access/common/index.ts

src/platform/packages/shared/kbn-synthtrace/src/scenarios/infra_hosts_semconv_with_apm_hosts.ts

The Lens formulas for semconv network KPIs use the metrics.* prefixed field. Without it, TX/RX tiles would show blank — same pattern already applied for CPU, memory, and disk metrics. Made-with: Cursor

rmyz · 2026-03-27T12:01:30Z

/ci

elasticmachine · 2026-03-30T07:30:56Z

Pinging @elastic/obs-presentation-team (Team:obs-presentation)

macroscopeapp · 2026-03-30T07:32:44Z

Approvability

Verdict: Needs human review

This PR introduces new runtime filtering logic in the hosts search bar that changes what filter options are displayed based on schema selection. While framed as a fix, the new schemaFilters logic in controls_content.tsx is a behavioral change that affects user-facing functionality. Additionally, the author does not own any of the modified files, which are all owned by @elastic/obs-presentation-team.

^{You can customize Macroscope's approvability policy. Learn more.}

elasticmachine · 2026-03-30T08:44:43Z

💚 Build Succeeded

Buildkite Build
Commit: 996e4f1

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`infra`	1.3MB	1.3MB	+536.0B

History

💛 Build #417825 was flaky 25eac36
💔 Build #417736 failed 72bd5e9

cc @rmyz

flash1293

synthtrace change only, LGTM

iblancof

Obs-exploration code changes (synthtrace) lgtm

sbelastic

code LGTM, also tested locally and everything seems to be working :)

kibanamachine · 2026-03-31T08:33:18Z

Starting backport for target branches: 9.2, 9.3

https://github.com/elastic/kibana/actions/runs/23788098684

kibanamachine · 2026-03-31T08:41:49Z

💔 All backports failed

Status	Branch	Result
❌	9.2	Backport failed because of merge conflicts
❌	9.3	Backport failed because of merge conflicts

Manual backport

To create the backport manually run:

node scripts/backport --pr 259825

Questions ?

Please refer to the Backport tool documentation

rmyz · 2026-03-31T08:54:05Z

💚 All backports created successfully

Status	Branch	Result
✅	9.3
✅	9.2

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

…9825) Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> (cherry picked from commit 988eba6) # Conflicts: # src/platform/packages/shared/kbn-apm-synthtrace/src/scenarios/infra_hosts_semconv_with_apm_hosts.ts # src/platform/packages/shared/kbn-synthtrace/src/scenarios/infra_hosts_semconv.ts # x-pack/solutions/observability/plugins/infra/moon.yml # x-pack/solutions/observability/plugins/infra/public/pages/metrics/hosts/components/search_bar/control_panels_config.ts # x-pack/solutions/observability/plugins/infra/public/pages/metrics/hosts/components/search_bar/controls_content.tsx # x-pack/solutions/observability/plugins/infra/tsconfig.json

…9825) Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> (cherry picked from commit 988eba6) # Conflicts: # x-pack/solutions/observability/plugins/infra/moon.yml # x-pack/solutions/observability/plugins/infra/public/pages/metrics/hosts/components/search_bar/control_panels_config.ts # x-pack/solutions/observability/plugins/infra/public/pages/metrics/hosts/components/search_bar/controls_content.tsx # x-pack/solutions/observability/plugins/infra/tsconfig.json

…e_for_children6 * commit '3402744f63ca1196e97b11ffac4e7f7efab240df': (80 commits) [PerUserAuth] Add EARS auth type for Connectors V2 (elastic#253695) Fix `@elastic/eui/require-aria-label-for-modals` lint violations across `@elastic/kibana-core` files (elastic#259757) [Entity Analytics][Leads generation][4] Add API routes, LeadDataClient, and async generation (elastic#257046) [Agent Builder] Agent-centric UX redesign (elastic#258005) fix query streams failing test (elastic#260277) [Lens as code] Add list layout to the new API (elastic#259967) [FTR] Add warning comments to deployment-agnostic FTR base configs (elastic#260018) [Discover][Logs profile] Fix missing search highlights (elastic#260056) Plugin system: safe deletion (elastic#259038) [Infra] Fix Hosts filter options to match selected schema (elastic#259825) Manual Entity Resolution and flyout representation (elastic#260162) [Cascade] Handle grouping on fields with unset values (elastic#260033) [Fleet] generate OTel config for integration packages with otelcol inputs (elastic#259968) [Search] Switch over to V2 index management details (elastic#259866) [inference] increase timeout for ES inference calls (elastic#260382) [ES|QL] Enable subqueries (elastic#257455) [ES|QL] Change Point order free options (elastic#260282) [Auth] Added authentication strategy for UIAM OAuth (elastic#256182) [Security Solution] Add "alerts_candidate_count" rule execution metric (elastic#259917) [api-docs] 2026-03-31 Daily api_docs build (elastic#260380) ...

) (#260399) # Backport This will backport the following commits from `main` to `9.3`: - [[Infra] Fix Hosts filter options to match selected schema (#259825)](#259825)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport)  --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

) (#260403) # Backport This will backport the following commits from `main` to `9.2`: - [[Infra] Fix Hosts filter options to match selected schema (#259825)](#259825)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport)

…9825) Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

rmyz added release_note:fix backport:version Backport to applied version labels v9.3.0 v9.4.0 Team:obs-presentation Focus: APM UI, Infra UI, Hosts UI, Universal Profiling, Obs Overview and left Navigation labels Mar 26, 2026

rmyz self-assigned this Mar 26, 2026

rmyz removed the v9.4.0 label Mar 26, 2026

rmyz changed the title ~~fix: [Infra] Scope Hosts filter options to match selected schema~~ [Infra] Fix Hosts filter options to match selected schema Mar 26, 2026

rmyz added 2 commits March 27, 2026 09:53

improve scenarios

7e1383e

rmyz added v9.3.3 v9.2.8 and removed v9.3.0 labels Mar 27, 2026

revert changes

5456464

rmyz requested a review from Copilot March 27, 2026 10:03

Merge branch 'main' into fix/hosts-filter-schema-scoping

7e35371

Copilot AI reviewed Mar 27, 2026

View reviewed changes

PR comment fixes

72bd5e9

Changes from node scripts/lint_ts_projects --fix

56a3778

Changes from node scripts/regenerate_moon_projects.js --update

554a996

coderabbitai bot reviewed Mar 27, 2026

View reviewed changes

src/platform/packages/shared/kbn-synthtrace/src/scenarios/infra_hosts_semconv_with_apm_hosts.ts Show resolved Hide resolved

src/platform/packages/shared/kbn-synthtrace/src/scenarios/infra_hosts_semconv_with_apm_hosts.ts Show resolved Hide resolved

add metrics.system.network.io prefixed field to semconv scenarios

25eac36

The Lens formulas for semconv network KPIs use the metrics.* prefixed field. Without it, TX/RX tiles would show blank — same pattern already applied for CPU, memory, and disk metrics. Made-with: Cursor

Merge branch 'main' into fix/hosts-filter-schema-scoping

996e4f1

rmyz marked this pull request as ready for review March 30, 2026 07:30

rmyz requested review from a team as code owners March 30, 2026 07:30

flash1293 approved these changes Mar 30, 2026

View reviewed changes

iblancof approved these changes Mar 31, 2026

View reviewed changes

sbelastic approved these changes Mar 31, 2026

View reviewed changes

rmyz merged commit 988eba6 into elastic:main Mar 31, 2026
22 checks passed

kibanamachine added the v9.4.0 label Mar 31, 2026

rmyz deleted the fix/hosts-filter-schema-scoping branch March 31, 2026 08:36

This was referenced Mar 31, 2026

[9.3] [Infra] Fix Hosts filter options to match selected schema (#259825) #260399

Merged

[9.2] [Infra] Fix Hosts filter options to match selected schema (#259825) #260403

Merged

jeramysoucy pushed a commit to jeramysoucy/kibana that referenced this pull request Apr 1, 2026

[Infra] Fix Hosts filter options to match selected schema (elastic#25…

257d099

…9825) Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

Conversation

rmyz commented Mar 26, 2026 • edited by kibanamachine Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Problem

Changes

Demo

Before

After

Testing

Uh oh!

rmyz commented Mar 27, 2026

Uh oh!

rmyz commented Mar 27, 2026

Uh oh!

coderabbitai bot commented Mar 27, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rmyz commented Mar 27, 2026

Uh oh!

coderabbitai bot commented Mar 27, 2026

Uh oh!

coderabbitai bot commented Mar 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Uh oh!

rmyz commented Mar 27, 2026

Uh oh!

coderabbitai bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

rmyz commented Mar 27, 2026

Uh oh!

elasticmachine commented Mar 30, 2026

Uh oh!

macroscopeapp bot commented Mar 30, 2026

Approvability

Uh oh!

elasticmachine commented Mar 30, 2026

💚 Build Succeeded

Metrics [docs]

Async chunks

History

Uh oh!

flash1293 left a comment

Choose a reason for hiding this comment

Uh oh!

iblancof left a comment

Choose a reason for hiding this comment

Uh oh!

sbelastic left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kibanamachine commented Mar 31, 2026

Uh oh!

kibanamachine commented Mar 31, 2026

💔 All backports failed

Manual backport

Questions ?

Uh oh!

rmyz commented Mar 31, 2026

💚 All backports created successfully

Questions ?

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

rmyz commented Mar 26, 2026 •

edited by kibanamachine

Loading

coderabbitai bot commented Mar 27, 2026 •

edited

Loading