[Streams][Streamlang] Align ES|QL condition transpiler with Painless on null propagation by diogolima-elastic · Pull Request #264751 · elastic/kibana

diogolima-elastic · 2026-04-21T15:23:06Z

Summary

The Streamlang Painless and ES|QL transpilers disagreed on how a condition should behave when the field it references is missing or NULL. Painless uses two-valued logic (null-guards every leaf, so missing == "active" → false), while ES|QL uses three-valued logic (so the same comparison → NULL, which then propagates through NOT, AND, OR, and CASE). As a result, an else branch of { if: foo == "active" } / { else: ... } silently disappeared in ES|QL when foo was missing — neither the if nor the else fired, and the document came out unchanged

This PR closes that gap in two complementary changes:

Wrap every ES|QL leaf in COALESCE(<predicate>, <default>) so NULL leaves collapse to a decisive boolean before NOT / AND / OR / CASE see them. Once every leaf is two-valued, every operator above it behaves classically
Pick the right default per leaf: FALSE for positive leaves (eq, gt, range, contains, …) — a missing field "doesn't match". TRUE for the negative leaf (neq) — a missing field "is not equal to" any concrete value. This makes neq and not(eq) semantically equivalent on missing fields (both TRUE) and aligns Streamlang with Query DSL (bool.must_not.term)

The Painless transpiler is updated in lockstep so the two backends produce identical results: every neq shorthand binary now emits a disjunctive null-guard (field === null || …) instead of the conjunctive one (field !== null && …). Every other leaf keeps the conjunctive guard.

exists is explicitly not wrapped — IS NULL is a total predicate in ES|QL and never returns NULL itself, so no wrapper is needed.

Behavior changes (compiled ES|QL)

Positive leaves — wrapped with FALSE default:

Before:

| EVAL `flag` = CASE(`attributes.status` == "active", "yes", `flag`)
| WHERE NOT (`http.status_code` == 200)

After:

| EVAL `flag` = CASE(COALESCE(`attributes.status` == "active", FALSE), "yes", `flag`)
| WHERE NOT COALESCE(`http.status_code` == 200, FALSE)

Negative leaves (neq) — wrapped with TRUE default:

Before:

| WHERE `attributes.status` != "deleted"

After:

| WHERE COALESCE(`attributes.status` != "deleted", TRUE)

The TRUE default makes a missing attributes.status satisfy the neq predicate, mirroring not(eq), Painless, and Query DSL must_not.term semantics.

Behavior changes (compiled Painless)

neq shorthand binaries swap their null-guard from conjunctive (!== null &&) to disjunctive (=== null ||):

// Before — neq on a missing field returns false
(val_status !== null && (val_status instanceof Number && … || val_status != "deleted"))
// After — neq on a missing field returns true
(val_status === null || (val_status instanceof Number && … || val_status != "deleted"))

Every other leaf is unchanged

Tradeoff: Discover divergence

This change makes Streamlang neq semantics diverge from raw ES|QL != semantics in Discover and other surfaces where users hand-write ES|QL:

Streamlang DSL where: { neq: "deleted" } → matches missing-field documents.
Raw ES|QL WHERE field != "deleted" (typed in Discover) → excludes missing-field documents (native ES|QL three-valued logic).

elasticmachine · 2026-04-21T15:23:26Z

Pinging @elastic/obs-onboarding-team (Team:obs-onboarding)

macroscopeapp · 2026-04-21T15:25:26Z

Scout Test Review

No issues found ✅

^{Share feedback in the #appex-qa channel.}

^{Posted via Macroscope — Scout Test Review}

macroscopeapp · 2026-04-21T15:25:36Z

Approvability

Verdict: Needs human review

This PR changes how ES|QL and Painless transpilers evaluate conditions when fields are null/missing, altering runtime behavior in data processing pipelines. The author does not own any of the changed files, which belong to several different teams who should review this semantic change to their transpiler and query generation code.

No code changes detected at 3ce26db. Prior analysis still applies.

^{You can customize Macroscope's approvability policy. Learn more.}

diogolima-elastic · 2026-04-22T16:28:44Z

/ci

…opagation

elasticmachine · 2026-04-23T16:14:29Z

💔 Build Failed

Buildkite Build
Commit: b004b07

Failed CI Steps

Test Failures

[job] [logs] Defend Workflows Cypress Tests #6 / Endpoint exceptions - under Security Management/Assets Navigation and access control ESS should be able to navigate to Endpoint Exceptions from Manage side panel should be able to navigate to Endpoint Exceptions from Manage side panel

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`entityStore`	131.0KB	131.1KB	+156.0B
`streams`	227.4KB	227.5KB	+156.0B
total			+312.0B

History

💛 Build #432351 was flaky 4d21c5c
💛 Build #431982 was flaky 4d21c5c
💔 Build #431553 failed d5e9971

…opagation

flash1293

Tested and works as expected, @elastic/security-entity-analytics @elastic/core-analysis please make sure that this doesn't break your usage somehow.

Also FYI @elastic/obs-sig-events-team , not sure whether you rely on this code path

uri-weisman · 2026-04-27T08:04:51Z

@romulets can you please have a look?

…opagation

macroscopeapp · 2026-04-27T15:33:09Z

Catch flakiness early (recommended)

Recommended before merge: run the flaky test runner against this PR to catch flakiness early.

Covers the new neq fires on documents where the condition field is missing Scout API test added in filter_conditions.spec.ts.

Trigger a run with the Flaky Test Runner UI or post this comment on the PR:

/flaky scoutConfig:x-pack/platform/packages/shared/kbn-streamlang-tests/test/scout/api/playwright.config.ts:30

^{Share feedback in the #appex-qa channel.}

^{Posted via Macroscope — Flaky Test Runner nudge}

flash1293

Still looks good to me after the changes as discussed via Slack, would feel better with another look from @Kerry350 here though.

Kerry350

LGTM, thanks Diogo!

…opagation

diogolima-elastic · 2026-04-29T14:46:13Z

/ci

…opagation

kibanamachine · 2026-05-04T16:22:00Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: 72b34de

Failed CI Steps

FTR Configs #3

Test Failures

[job] [logs] FTR Configs #3 / Endpoint plugin @ess @serverless @skipInServerlessMKI Endpoint Scripts Library RBAC Get one API "before each" hook for "should return script when user has READ privileges"

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`entityStore`	131.4KB	131.7KB	+245.0B

History

💔 Build #437345 failed 3365edd
💔 Build #437258 failed 3ce26db
💛 Build #435773 was flaky 5eea093
💛 Build #435684 was flaky 5eea093
💛 Build #434380 was flaky b0eacd1

…ilder_new_vis_attachment * commit '6fd683609eb6dee81f242f8ff6951edbe3bfd66c': (226 commits) Remove Model Author group-by option from external inference endpoints (elastic#264761) [Streams][Streamlang] Align ES|QL condition transpiler with Painless on null propagation (elastic#264751) chore(axios,workflows-eng): remove axios from workflows connector utils (elastic#267512) [failed-test-reporter] avoid opening issues for scout env failures (elastic#267649) [kbn-api-contracts] Detect request-body additionalProperties:false tightening (elastic#267546) [main] Sync bundled packages with Package Storage (elastic#267644) Centralize phase colors and descriptions (elastic#266680) [Unified Waterfall] Add "Scroll to origin" button (elastic#266594) [APM] Add alert and SLO badges to the service map embeddable (elastic#266360) [CI] Speed up telemetry_check by pre-filtering to collector files (elastic#265978) [Discover] Address flaky large CSV test (elastic#266642) avoid passing unrelated props within integration card icon component conditional render (elastic#266569) [Cases][Templates] Extend cases search by template field label (elastic#266414) [Background search] Migrate custom SplitButton to EuiSplitButton (elastic#267447) [i18n] Report translation coverage during integrate (elastic#264124) [api-docs] 2026-05-05 Daily api_docs build (elastic#267639) [Scout] Update test config manifests (elastic#267636) [content list] Add saved object provider services (elastic#266428) [Fleet] Otel UI add health and implement it in OTelComponentDetail (elastic#267292) Update dependency msw to v2.13.4 (main) (elastic#266770) ...

…n alignment in maintainer ES|QL snapshots Refresh 7 of 8 maintainer ES|QL golden snapshots to track the upstream alignment of the ES|QL condition transpiler with Painless null propagation (PR elastic#264751, "[Streams][Streamlang] Align ES|QL condition transpiler with Painless on null propagation"). That change rewrote the output of `getEuidDocumentsContainsIdFilter`, `getFieldEvaluationsEsql`, and `euid.esql.getEuidEvaluation` (all from `@kbn/entity-store`) so that comparisons sitting under a NOT/AND/OR chain are now wrapped in `COALESCE(expr, TRUE | FALSE)`, preserving the same truth value when one of the operands is NULL. The previous, un-wrapped form silently evaluated NULL-on-either-side comparisons to NULL, which Painless treats as FALSE rather than three-valued — the upstream change brings ES|QL into line. The diff is 100% the COALESCE wrap (`grep -v COALESCE` over the added lines is empty); no column names, identifiers, FROM clauses, or query shapes changed. The azure_auditlogs override snapshot is unchanged (it does not consume the affected helpers). All 200 maintainer unit tests pass with the refreshed snapshots. Co-authored-by: Cursor <cursoragent@cursor.com>

align not and null behavior

78bdbef

diogolima-elastic requested review from a team as code owners April 21, 2026 15:23

diogolima-elastic added release_note:skip Skip the PR/issue when compiling release notes backport:skip This PR does not require backporting Team:obs-onboarding Observability Onboarding Team Feature:Streams This is the label for the Streams Project labels Apr 21, 2026

kibanamachine and others added 2 commits April 21, 2026 15:50

Changes from node scripts/eslint_all_files --no-cache --fix

d5e9971

fix tests and snapshots

4d21c5c

diogolima-elastic requested review from a team as code owners April 22, 2026 10:29

diogolima-elastic requested a review from CAWilson94 April 22, 2026 10:29

Merge branch 'main' into streamlang-align-behavior-of-not-and-null-pr…

b004b07

…opagation

Merge branch 'main' into streamlang-align-behavior-of-not-and-null-pr…

2a4aaca

…opagation

flash1293 approved these changes Apr 24, 2026

View reviewed changes

diogolima-elastic added 2 commits April 27, 2026 12:21

Merge branch 'main' into streamlang-align-behavior-of-not-and-null-pr…

e48cdd0

…opagation

flip negative

c62c6f8

macroscopeapp Bot reviewed Apr 27, 2026

View reviewed changes

Comment thread x-pack/platform/packages/shared/kbn-streamlang/src/conditions/condition_to_painless.ts Outdated

safer check

b0eacd1

flash1293 approved these changes Apr 28, 2026

View reviewed changes

Kerry350 self-requested a review April 28, 2026 11:04

romulets approved these changes Apr 29, 2026

View reviewed changes

CAWilson94 approved these changes Apr 29, 2026

View reviewed changes

Kerry350 approved these changes Apr 29, 2026

View reviewed changes

Merge branch 'main' into streamlang-align-behavior-of-not-and-null-pr…

5eea093

…opagation

crespocarlos approved these changes May 1, 2026

View reviewed changes

diogolima-elastic added 3 commits May 4, 2026 09:48

Merge branch 'main' into streamlang-align-behavior-of-not-and-null-pr…

3ce26db

…opagation

Merge branch 'main' into streamlang-align-behavior-of-not-and-null-pr…

3365edd

…opagation

test and snapshot update

72b34de

diogolima-elastic merged commit ba3d6d9 into elastic:main May 5, 2026
24 checks passed

kibanamachine added the v9.5.0 label May 5, 2026

Conversation

diogolima-elastic commented Apr 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Behavior changes (compiled ES|QL)

Behavior changes (compiled Painless)

Tradeoff: Discover divergence

Uh oh!

elasticmachine commented Apr 21, 2026

Uh oh!

macroscopeapp Bot commented Apr 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Scout Test Review

Uh oh!

macroscopeapp Bot commented Apr 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Approvability

Uh oh!

diogolima-elastic commented Apr 22, 2026

Uh oh!

elasticmachine commented Apr 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💔 Build Failed

Failed CI Steps

Test Failures

Metrics [docs]

Async chunks

History

Uh oh!

flash1293 left a comment

Choose a reason for hiding this comment

Uh oh!

uri-weisman commented Apr 27, 2026

Uh oh!

macroscopeapp Bot commented Apr 27, 2026

Catch flakiness early (recommended)

Uh oh!

Uh oh!

flash1293 left a comment

Choose a reason for hiding this comment

Uh oh!

Kerry350 left a comment

Choose a reason for hiding this comment

Uh oh!

diogolima-elastic commented Apr 29, 2026

Uh oh!

kibanamachine commented May 4, 2026

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

Metrics [docs]

Async chunks

History

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

diogolima-elastic commented Apr 21, 2026 •

edited

Loading

macroscopeapp Bot commented Apr 21, 2026 •

edited

Loading

macroscopeapp Bot commented Apr 21, 2026 •

edited

Loading

elasticmachine commented Apr 23, 2026 •

edited

Loading