fix(streams): update default_pipeline on write index when adding processing to a second OTEL metrics stream by flash1293 · Pull Request #269988 · elastic/kibana

flash1293 · 2026-05-19T15:42:52Z

Summary

When a second classic stream (e.g. a second OTEL metrics data stream) sharing the same template-level ingest pipeline receives its first processing step, updateExistingStreamsManagedPipeline was only updating the shared pipeline's processors. It never pushed update_default_ingest_pipeline actions for the new stream's current write index, so index.default_pipeline was never set on it and documents indexed into that stream bypassed processing entirely.

This did not affect logs data streams because their write indices already have default_pipeline set by built-in templates. OTEL metrics data streams do not, so they depend entirely on Streams to set the index.default_pipeline setting.

Root cause

In translate_classic_stream_pipeline_actions.ts, the updateExistingStreamsManagedPipeline path (taken when the template pipeline already exists as a streams-managed pipeline) was missing the update_default_ingest_pipeline pushes that createStreamsManagedPipeline correctly performs for the first stream.

Fix

For every append_processor_to_ingest_pipeline action processed by updateExistingStreamsManagedPipeline, emit a matching update_default_ingest_pipeline action — mirroring what createStreamsManagedPipeline already does.

Testing

Updated the existing unit test (translate_classic_stream_pipeline_actions.test.ts) to assert update_default_ingest_pipeline is emitted when updateExistingStreamsManagedPipeline appends a processor.
Added a new FTR API integration test (otel_metrics_processing.ts) that:
1. Creates two metrics-*.otel-default data streams via the ES API
2. Adds a set processor to the first stream via the Streams API (creates the template pipeline)
3. Adds a set processor to the second stream via the Streams API (exercises the previously-broken path)
4. Indexes a document into the second stream and verifies that the processor was actually applied

Made with Cursor

…essing to a second OTEL metrics stream When a second classic stream sharing the same template-level ingest pipeline receives its first processing step, `updateExistingStreamsManagedPipeline` was only updating the shared pipeline's processors. It never pushed `update_default_ingest_pipeline` actions for the new stream's current write index, so `index.default_pipeline` was never set on it and documents indexed into that stream bypassed processing entirely. The fix mirrors what `createStreamsManagedPipeline` already does: for every `append_processor_to_ingest_pipeline` action handled by the `updateExistingStreamsManagedPipeline` path, emit a matching `update_default_ingest_pipeline` action so the write index setting is kept in sync. Closes elastic#269984 Co-authored-by: Cursor <cursoragent@cursor.com>

… runs - Switch from PUT /api/streams/{name} to PUT /api/streams/{name}/_ingest - Fix assertion to use nested object access (source.attributes?.stream_origin) instead of flat dotted key access, matching how the set processor stores data - Add after-hook cleanup for the Streams-managed ingest pipelines so that subsequent local runs start with a clean template pipeline and correctly exercise the update_default_ingest_pipeline code path Co-authored-by: Cursor <cursoragent@cursor.com>

infra-vault-gh-plugin-prod · 2026-05-20T09:03:09Z

Pinging @elastic/obs-onboarding-team (Team:obs-onboarding)

nikita-lavrov

Tested locally and works as expected

kibanamachine · 2026-05-22T10:04:02Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: 4e5af69

Failed CI Steps

Test Failures

[job] [logs] FTR Configs #11 / Entity Analytics - Risk Score Maintainer @ess @serverless @serverlessQA Asset Criticality CSV Upload V2 "before all" hook for "uploads CSV and updates asset criticality for a matched entity"
[job] [logs] Scout Lane #6 - stateful-classic / default / local-stateful-classic - Hosts Page - Empty State - should show onboarding page when no data is present
[job] [logs] Scout Lane #4 - stateful-classic / default / local-stateful-classic - Profiling is setup and data is loaded - Admin user

Metrics [docs]

✅ unchanged

History

💛 Build #445155 was flaky 50ef19d
💔 Build #444814 failed 268a865

kibanamachine · 2026-05-22T10:04:39Z

Starting backport for target branches: 9.4

https://github.com/elastic/kibana/actions/runs/26281478381

kibanamachine · 2026-05-22T10:14:26Z

💔 All backports failed

Status	Branch	Result
❌	9.4	Backport failed because of merge conflicts

Manual backport

To create the backport manually run:

node scripts/backport --pr 269988

Questions ?

Please refer to the Backport tool documentation

…essing to a second OTEL metrics stream (#269988) ## Summary Fixes #269984 When a second classic stream (e.g. a second OTEL metrics data stream) sharing the same template-level ingest pipeline receives its first processing step, `updateExistingStreamsManagedPipeline` was only updating the shared pipeline's processors. It never pushed `update_default_ingest_pipeline` actions for the new stream's current write index, so `index.default_pipeline` was never set on it and documents indexed into that stream bypassed processing entirely. This did not affect logs data streams because their write indices already have `default_pipeline` set by built-in templates. OTEL metrics data streams do not, so they depend entirely on Streams to set the `index.default_pipeline` setting. ## Root cause In `translate_classic_stream_pipeline_actions.ts`, the `updateExistingStreamsManagedPipeline` path (taken when the template pipeline already exists as a streams-managed pipeline) was missing the `update_default_ingest_pipeline` pushes that `createStreamsManagedPipeline` correctly performs for the first stream. ## Fix For every `append_processor_to_ingest_pipeline` action processed by `updateExistingStreamsManagedPipeline`, emit a matching `update_default_ingest_pipeline` action — mirroring what `createStreamsManagedPipeline` already does. ## Testing - Updated the existing unit test (`translate_classic_stream_pipeline_actions.test.ts`) to assert `update_default_ingest_pipeline` is emitted when `updateExistingStreamsManagedPipeline` appends a processor. - Added a new FTR API integration test (`otel_metrics_processing.ts`) that: 1. Creates two `metrics-*.otel-default` data streams via the ES API 2. Adds a `set` processor to the first stream via the Streams API (creates the template pipeline) 3. Adds a `set` processor to the second stream via the Streams API (exercises the previously-broken path) 4. Indexes a document into the second stream and verifies that the processor was actually applied Made with [Cursor](https://cursor.com) --------- Co-authored-by: Cursor <cursoragent@cursor.com>

kibanamachine · 2026-05-26T10:49:03Z

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create automatically backports add a backport:* label or prevent reminders by adding the backport:skip label.
You can also create backports manually by running node scripts/backport --pr 269988 locally
cc: @flash1293

kibanamachine · 2026-05-27T10:49:23Z

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create automatically backports add a backport:* label or prevent reminders by adding the backport:skip label.
You can also create backports manually by running node scripts/backport --pr 269988 locally
cc: @flash1293

flash1293 force-pushed the flash1293/default-pipeline-fix branch 3 times, most recently from c4d1bd0 to 48721eb Compare May 19, 2026 15:56

flash1293 force-pushed the flash1293/default-pipeline-fix branch from 48721eb to 268a865 Compare May 19, 2026 15:57

flash1293 added release_note:fix Team:obs-onboarding Observability Onboarding Team backport:version Backport to applied version labels Feature:Streams This is the label for the Streams Project v9.4.2 labels May 19, 2026

flash1293 added the v9.5.0 label May 20, 2026

flash1293 marked this pull request as ready for review May 20, 2026 09:03

flash1293 requested review from a team as code owners May 20, 2026 09:03

nikita-lavrov approved these changes May 21, 2026

View reviewed changes

Merge branch 'main' into flash1293/default-pipeline-fix

4e5af69

flash1293 enabled auto-merge (squash) May 22, 2026 09:11

flash1293 merged commit 9429056 into elastic:main May 22, 2026
31 checks passed

This was referenced May 22, 2026

[Entity Store] Change defaults to conservative values #270617

Merged

[One Workflow] Fix the Workflows YAML editor freeze and ~2 GB memory spike #270103

Merged

kibanamachine added the backport missing Added to PRs automatically when the are determined to be missing a backport. label May 26, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(streams): update default_pipeline on write index when adding processing to a second OTEL metrics stream#269988

fix(streams): update default_pipeline on write index when adding processing to a second OTEL metrics stream#269988
flash1293 merged 3 commits into
elastic:mainfrom
flash1293:flash1293/default-pipeline-fix

flash1293 commented May 19, 2026 •

edited by kibanamachine

Loading

Uh oh!

infra-vault-gh-plugin-prod Bot commented May 20, 2026

Uh oh!

nikita-lavrov left a comment

Uh oh!

kibanamachine commented May 22, 2026

Uh oh!

Uh oh!

kibanamachine commented May 22, 2026

Uh oh!

kibanamachine commented May 22, 2026

Uh oh!

kibanamachine commented May 26, 2026

Uh oh!

kibanamachine commented May 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

flash1293 commented May 19, 2026 • edited by kibanamachine Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Root cause

Fix

Testing

Uh oh!

infra-vault-gh-plugin-prod Bot commented May 20, 2026

Uh oh!

nikita-lavrov left a comment

Choose a reason for hiding this comment

Uh oh!

kibanamachine commented May 22, 2026

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

Metrics [docs]

History

Uh oh!

Uh oh!

kibanamachine commented May 22, 2026

Uh oh!

kibanamachine commented May 22, 2026

💔 All backports failed

Manual backport

Questions ?

Uh oh!

kibanamachine commented May 26, 2026

Uh oh!

kibanamachine commented May 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

flash1293 commented May 19, 2026 •

edited by kibanamachine

Loading