[Streams] Improve condition filtering by flash1293 · Pull Request #251129 · elastic/kibana

flash1293 · 2026-01-30T18:51:50Z

Summary

Improves condition filtering metrics in Streams data enrichment by tagging simulation-only no-op processors per condition block so the UI can show accurate match-rate badges and filter simulation results by condition.

This branch is now up to date with upstream/main (conflicts resolved).

Changes

Add simulation-only no-op processors (set+remove) per condition to track match rates.
Filter out condition no-op processors from document status/diff calculations.
UI: show condition match-rate badges on WHERE blocks and filter derived document metrics based on the selected condition.
UX: auto-select parent condition when creating/editing processors under it.
UX: when updating/saving a processor, keep the current condition focus (don’t clear the condition filter).

Follow-ups from review / CI

Removed selectSamplesForSimulation (it only returned context.samples) and inlined context.samples at the call site.
Updated streams simulation tests to pass the now-required conditionProcessorTags argument.
Fixed Scout spec import to use expect from @kbn/scout/ui.
Re-ran the OAS snapshot tooling locally after merging upstream/main; it produced no diff under oas_docs/.

Test plan

yarn test:jest x-pack/platform/plugins/shared/streams/server/routes/internal/streams/processing/simulation_handler.test.ts x-pack/platform/plugins/shared/streams/server/routes/internal/streams/processing/simulation_condition_noops.test.ts
yarn test:jest x-pack/platform/plugins/shared/streams_app/public/components/data_management/stream_detail_enrichment/state_management/simulation_state_machine/utils.test.ts
yarn test:jest x-pack/platform/plugins/shared/streams_app/public/components/data_management/stream_detail_enrichment/state_management/interactive_mode_machine/interactive_mode_machine.test.ts
node scripts/scout.js run-tests --stateful --testFiles x-pack/platform/plugins/shared/streams_app/test/scout/ui/tests/data_management/data_processing/condition_filtering.spec.ts

- Fixed type error in pattern_suggestion_helpers.ts by adding missing selectedConditionId parameter to selectPreviewRecords call - Updated simulation_condition_noops.test.ts assertions to match actual processor counts (3 and 5 instead of 4 and 7) - Updated utils.test.ts to reflect new condition filtering behavior where condition IDs are directly included in processed_by array

…lters to zero documents When creating/editing processors under conditions with 0% match rate, field suggestions would be empty because they relied on condition-filtered preview records. This fix makes field suggestions fall back to all samples when the filtered records are empty.

…ring Adds playwright tests for the new condition filtering behavior: - Verify condition match rate badges appear on WHERE blocks - Test 0%, 50%, and 100% match rate scenarios - Validate "Selected X%" indicator when editing processors under conditions - Test badge persistence after saving and page reload - Verify nested conditions show match rate badges

LucaWintergerst · 2026-02-03T14:57:51Z

CleanShot.2026-02-03.at.15.54.03.mp4

this is really close overall
the only slightly odd thing I was able to notice is that the "partially parsed" column would always show 100%
but other than that it feels great to me

flash1293 · 2026-02-03T15:11:37Z

the only slightly odd thing I was able to notice is that the "partially parsed" column would always show 100%

@LucaWintergerst What would you expect to show in this scenario?

…t status The remove processor in condition noop pairs was missing a tag, causing: 1. Documents matching conditions without sub-processors to show as "Failed 100%" 2. Documents with sub-processors to show as "Partially parsed" instead of "Parsed" Added a distinct tag suffix (:noop-cleanup) to the remove processor so it gets filtered out without double-counting in processor metrics.

LucaWintergerst · 2026-02-03T16:30:24Z

pushed a fix for this. We always detected a failing processor in all docs due to the remove processor as part of the condition, causing issues in the calculations

before:

flash1293 · 2026-02-03T17:17:27Z

-    return docIndexes
-      .filter((docIndex) => samples.at(docIndex) !== undefined)
-      .map((index) => samples[index]);
+    // Always simulate on the full sample set.


selectSamplesForSimulation should be simplified since it just fetches the samples and nothing else. Is it even used anymore? If not, we should remove it.

flash1293 · 2026-02-10T16:59:36Z

/ralph

Update with upstream/main and resolve conflicts
Address the comment above

Co-authored-by: Cursor <cursoragent@cursor.com>

- Remove `selectSamplesForSimulation` and inline `context.samples` for simulation input - Update `computeSimulationDocDiff` tests to pass `conditionProcessorTags` - Fix Scout UI test to import `expect` from `@kbn/scout/ui` Co-authored-by: Cursor <cursoragent@cursor.com>

flash1293 · 2026-02-10T18:19:15Z

Updated the PR branch and addressed the remaining review/CI follow-ups.

Changes:

Merged upstream/main into flash1293/improved-condition-filtering (conflicts resolved)
Removed selectSamplesForSimulation and inlined context.samples (selector was just a passthrough)
Updated streams simulation tests to pass conditionProcessorTags
Fixed Scout spec import to use expect from @kbn/scout/ui
Ran the OAS snapshot tooling locally after the upstream merge; it produced no diff under oas_docs/

Written by cursor, wait for @flash1293 to chime in

flash1293 · 2026-02-11T20:01:08Z

@LucaWintergerst finally got to review this properly, sorry for the delay. I think it works really well, only nit I can see is this behavior:

After clicking "Update" on a processor, it unselects the current condition again. Is that on purpose or should we keep the focus?

Also, this should definitely be split out, but do we want something similar for the yaml mode? It's quite useful to scope the simulation like this, I can also imagine it for YAML, but we need to think the interaction through.

flash1293 · 2026-02-12T13:23:21Z

/ralph
Make sure the thing above is handled properly: After clicking "Update" on a processor, it unselects the current condition again. In this case we should keep the focus as it was before instead of removing it again.

Avoid clearing the auto-selected condition filter when saving a processor edit so the currently focused condition remains selected after clicking Update. Add a regression test for the save flow.

flash1293 · 2026-02-12T15:40:21Z

Addressed the review note about processor Update clearing the selected condition.\n\n- Saving/updating a processor no longer clears the auto-selected condition filter, so the condition focus stays where it was.\n- Cancel still clears the auto condition filter (unchanged).\n- Added a regression test: .\n\n(waiting for @flash1293 to chime in if anything else is needed)

flash1293 · 2026-02-12T15:40:32Z

Follow-up: the regression test added for this behavior is at:

x-pack/platform/plugins/shared/streams_app/public/components/data_management/stream_detail_enrichment/state_management/interactive_mode_machine/interactive_mode_machine.test.ts

flash1293 · 2026-02-12T17:13:07Z

The type error should be fixed on main already, let's see

flash1293 · 2026-02-12T19:39:33Z

/ralph there are failing jest tests:

Jest Tests
x-pack/platform/plugins/shared/streams_app/public/components/data_management/stream_detail_enrichment/state_management/interactive_mode_machine
interactiveModeMachine condition focus behavior does not clear auto-selected condition focus on processor save (Update)

Command Line:
node scripts/jest --config x-pack/platform/plugins/shared/streams_app/jest.config.js --runInBand --coverage=false --passWithNoTests
Owners:
Unable to determine code owners
Failures in tracked branches: 1
https://dryrun/
Buildkite Job
https://buildkite.com/elastic/kibana-pull-request/builds/394122#019c52df-9aaa-4da5-a780-50a74e049390

Error: expect(received).toBeDefined()

Received: undefined
at Object.toBeDefined (/opt/buildkite-agent

Fix them, validate locally it works, then push the fix

- Fix test to send step.save to child stepRefs instead of parent actor (matching actual UI behavior) - Add htmlIdGenerator mock to generate unique IDs (EUI test-env mock returns the same ID for all calls, breaking multi-step tests) - Fix grokCollection mock to include setCustomPatterns method - Add step.filterByCondition and step.clearConditionFilter event handlers to creating and editing states in interactiveModeMachine (previously only handled in idle state, but needed when editing) Co-authored-by: Cursor <cursoragent@cursor.com>

flash1293 · 2026-02-12T20:15:04Z

Fixed the failing Jest test for condition focus on processor save.

Root causes identified:

Test was sending events to wrong actor: The test was sending step.save events directly to the parent interactiveModeMachine actor, but in the actual UI, save events are sent to the child stepRef actors. The child then notifies the parent. This mismatch caused the state machine to not transition properly.
EUI test-env mock returns same ID: The default EUI test environment mocks htmlIdGenerator to always return 'generated-id'. This caused both the condition block and processor to have the same customIdentifier, breaking the test logic. Fixed by adding a local mock that generates unique IDs.
grokCollection mock missing setCustomPatterns: The stepMachine calls grokCollection.setCustomPatterns() on entry to draft state. The mock was missing this method, causing silent failures.
step.filterByCondition not handled in creating/editing states: The step.filterByCondition and step.clearConditionFilter events were only handled in the idle state of the interactiveModeMachine. When a processor is being edited (which puts the machine in editing state), these events were being ignored. Added handlers to both creating and editing states.

(written by cursor, wait for @flash1293 to chime in)

…condition-filtering

CoenWarmer · 2026-02-17T10:13:01Z

Perhaps we can consider passing an object instead with descriptive key names, so we don't have to rely as much on arguments being descriptively named when the function is used

flash1293 · 2026-02-18T08:41:32Z

/ralph Address @CoenWarmer s comment on the PR - I think it's a good idea.

@CoenWarmer

Addresses @CoenWarmer's review comment to use an object parameter with descriptive key names instead of positional arguments, improving readability at call sites. Co-authored-by: Cursor <cursoragent@cursor.com>

flash1293 · 2026-02-18T09:10:46Z

Addressed @CoenWarmer's review comment about converting computeSimulationDocDiff from positional arguments to an object parameter with descriptive keys.

Changes:

Converted the function signature from 5 positional args to a single object parameter (ComputeSimulationDocDiffParams)
Updated the internal call site in simulation_handler.ts
Updated all 13 test call sites in simulation_handler.test.ts
All tests pass, type checks pass, linting passes

(written by cursor, wait for @flash1293 to chime in)

elasticmachine · 2026-02-18T09:19:23Z

💔 Build Failed

Buildkite Build
Commit: 85435fe

Failed CI Steps

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`streamsApp`	1.7MB	1.7MB	+3.4KB

History

💔 Build #396636 failed c37e6cf
💚 Build #395177 succeeded 1562bfb
💔 Build #394220 failed c5580a9
💔 Build #394213 failed d272e01

flash1293 · 2026-02-18T09:42:56Z

/ralph xstate5 got renamed to xstate, please fix the linting and type failures.

The package was renamed from xstate5 to xstate. Update the import in the interactive mode machine test file to fix linting and type check failures. Co-authored-by: Cursor <cursoragent@cursor.com>

flash1293 · 2026-02-18T09:50:25Z

Fixed the xstate5 → xstate import issue.

Change:

Updated createActor import from 'xstate5' to 'xstate' in interactive_mode_machine.test.ts

Validation:

Linting passes
Type check passes (yarn test:type_check --project x-pack/platform/plugins/shared/streams_app/tsconfig.json)
Jest test passes (yarn test:jest interactive_mode_machine.test.ts)

(written by cursor, wait for @flash1293 to chime in)

flash1293 · 2026-02-18T13:23:59Z

@CoenWarmer should be good for another look.

CoenWarmer · 2026-02-18T21:53:34Z

A little hard to judge without more context or a related ticket but code looks good.

…condition-filtering

Update test files to use proper type annotations instead of `as any` casts to comply with the no-explicit-any ESLint rule that was added to the streams plugins. Co-authored-by: Cursor <cursoragent@cursor.com>

## Summary Improves condition filtering metrics in Streams data enrichment by tagging simulation-only no-op processors per condition block so the UI can show accurate match-rate badges and filter simulation results by condition. This branch is now up to date with `upstream/main` (conflicts resolved). ## Changes - Add simulation-only no-op processors (set+remove) per condition to track match rates. - Filter out condition no-op processors from document status/diff calculations. - UI: show condition match-rate badges on WHERE blocks and filter derived document metrics based on the selected condition. - UX: auto-select parent condition when creating/editing processors under it. - UX: when updating/saving a processor, keep the current condition focus (don’t clear the condition filter). ## Follow-ups from review / CI - Removed `selectSamplesForSimulation` (it only returned `context.samples`) and inlined `context.samples` at the call site. - Updated streams simulation tests to pass the now-required `conditionProcessorTags` argument. - Fixed Scout spec import to use `expect` from `@kbn/scout/ui`. - Re-ran the OAS snapshot tooling locally after merging `upstream/main`; it produced **no diff** under `oas_docs/`. ## Test plan - `yarn test:jest x-pack/platform/plugins/shared/streams/server/routes/internal/streams/processing/simulation_handler.test.ts x-pack/platform/plugins/shared/streams/server/routes/internal/streams/processing/simulation_condition_noops.test.ts` - `yarn test:jest x-pack/platform/plugins/shared/streams_app/public/components/data_management/stream_detail_enrichment/state_management/simulation_state_machine/utils.test.ts` - `yarn test:jest x-pack/platform/plugins/shared/streams_app/public/components/data_management/stream_detail_enrichment/state_management/interactive_mode_machine/interactive_mode_machine.test.ts` - `node scripts/scout.js run-tests --stateful --testFiles x-pack/platform/plugins/shared/streams_app/test/scout/ui/tests/data_management/data_processing/condition_filtering.spec.ts` --------- Co-authored-by: LucaWintergerst <luca.wintergerst@elastic.co> Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: flash1293 <flash1293@users.noreply.github.com>

LucaWintergerst and others added 7 commits January 30, 2026 19:22

improve filtering

cd582db

Changes from node scripts/eslint_all_files --no-cache --fix

3a5b3d9

Merge branch 'main' into flash1293/improved-condition-filtering

63cce17

Changes from yarn openapi:bundle

c92ffa9

Merge branch 'main' into flash1293/improved-condition-filtering

11f7dc7

flash1293 commented Feb 3, 2026

View reviewed changes

flash1293 and others added 2 commits February 10, 2026 18:29

Merge upstream/main into flash1293/improved-condition-filtering

d59edf7

Co-authored-by: Cursor <cursoragent@cursor.com>

Merge branch 'main' into flash1293/improved-condition-filtering

c03b2f5

flash1293 changed the title ~~[Streams] Improve condition filtering (rebased)~~ [Streams] Improve condition filtering Feb 11, 2026

flash1293 added release_note:enhancement backport:skip This PR does not require backporting Team:obs-onboarding Observability Onboarding Team Feature:Streams This is the label for the Streams Project v9.4.0 labels Feb 11, 2026

[Streams] Preserve condition focus on processor update

f1adc77

Avoid clearing the auto-selected condition filter when saving a processor edit so the currently focused condition remains selected after clicking Update. Add a regression test for the save flow.

do not reset filter on cancel either

a172447

Merge branch 'main' into flash1293/improved-condition-filtering

2b2de52

flash1293 added 4 commits February 12, 2026 21:31

Merge branch 'main' into flash1293/improved-condition-filtering

c5580a9

Merge branch 'main' into flash1293/improved-condition-filtering

df90db9

Merge upstream/main into flash1293/improved-condition-filtering

3d0e992

Merge remote-tracking branch 'upstream/main' into flash1293/improved-…

1562bfb

…condition-filtering

CoenWarmer reviewed Feb 17, 2026

View reviewed changes

Merge branch 'main' into flash1293/improved-condition-filtering

c37e6cf

refactor(streams): convert computeSimulationDocDiff to object params

85435fe

Addresses @CoenWarmer's review comment to use an object parameter with descriptive key names instead of positional arguments, improving readability at call sites. Co-authored-by: Cursor <cursoragent@cursor.com>

fix(streams): update xstate5 import to xstate

dd4bdf1

The package was renamed from xstate5 to xstate. Update the import in the interactive mode machine test file to fix linting and type check failures. Co-authored-by: Cursor <cursoragent@cursor.com>

CoenWarmer requested review from CoenWarmer and Copilot and removed request for a team and Copilot February 18, 2026 21:54

CoenWarmer approved these changes Feb 18, 2026

View reviewed changes

flash1293 and others added 2 commits February 19, 2026 10:32

Merge remote-tracking branch 'upstream/main' into flash1293/improved-…

6861a38

…condition-filtering

fix(streams): resolve no-explicit-any ESLint errors after merge

5848170

Update test files to use proper type annotations instead of `as any` casts to comply with the no-explicit-any ESLint rule that was added to the streams plugins. Co-authored-by: Cursor <cursoragent@cursor.com>

flash1293 merged commit bf58cee into elastic:main Feb 19, 2026
16 checks passed

flash1293 mentioned this pull request Mar 6, 2026

[Streams] Improve condition filtering #247095

Closed

10 tasks

Conversation

flash1293 commented Jan 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Follow-ups from review / CI

Test plan

Uh oh!

LucaWintergerst commented Feb 3, 2026

Uh oh!

flash1293 commented Feb 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

LucaWintergerst commented Feb 3, 2026

Uh oh!

flash1293 Feb 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

flash1293 commented Feb 10, 2026

Uh oh!

flash1293 commented Feb 10, 2026

Uh oh!

flash1293 commented Feb 11, 2026

Uh oh!

flash1293 commented Feb 12, 2026

Uh oh!

flash1293 commented Feb 12, 2026

Uh oh!

flash1293 commented Feb 12, 2026

Uh oh!

flash1293 commented Feb 12, 2026

Uh oh!

flash1293 commented Feb 12, 2026

Uh oh!

flash1293 commented Feb 12, 2026

Uh oh!

CoenWarmer Feb 17, 2026

Choose a reason for hiding this comment

Uh oh!

flash1293 commented Feb 18, 2026

Uh oh!

flash1293 commented Feb 18, 2026

Uh oh!

elasticmachine commented Feb 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💔 Build Failed

Failed CI Steps

Metrics [docs]

Async chunks

History

Uh oh!

flash1293 commented Feb 18, 2026

Uh oh!

flash1293 commented Feb 18, 2026

Uh oh!

flash1293 commented Feb 18, 2026

Uh oh!

CoenWarmer commented Feb 18, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

flash1293 commented Jan 30, 2026 •

edited

Loading

flash1293 commented Feb 3, 2026 •

edited

Loading

flash1293 Feb 3, 2026 •

edited

Loading

elasticmachine commented Feb 18, 2026 •

edited

Loading