Fix metadata fields being nullified/loaded by unmapped_fields setting by quackaplop · Pull Request #143155 · elastic/elasticsearch

quackaplop · 2026-02-26T15:12:56Z

Summary

Fixes #141907

When SET unmapped_fields="nullify" or "load" is used, metadata fields like _score, _id, _index, _version, etc. are incorrectly treated as regular unmapped fields. Instead of producing an error that guides the user to add METADATA _score to the FROM clause, the fields are silently resolved as NULL (nullify mode) or treated as unmapped keyword fields (load mode).

This is problematic because:

It hides user mistakes — the user likely intended to use the metadata field but forgot METADATA
In the case of _score, it obscures the fact that no search scoring was performed
It violates the principle that metadata fields require explicit opt-in via the METADATA clause

Root cause

ResolveUnmapped.collectUnresolved() collects all UnresolvedAttributes in the plan and feeds them into the nullify/load logic. Since metadata fields that aren't declared in the METADATA clause remain unresolved after ResolveRefs, they get picked up and silently resolved by ResolveUnmapped.

Fix

Added a check in collectUnresolved() to exclude attributes whose names match known metadata fields (MetadataAttribute.isSupported()). This lets them stay unresolved so the verifier produces the proper Unknown column [_score] error, guiding the user to declare FROM ... METADATA _score.

When metadata fields are declared via the METADATA clause, they get resolved during ResolveRefs and never appear as UnresolvedAttributes — so the new filter has no effect on the happy path.

Test plan

All tests iterate over every entry in MetadataAttribute.ATTRIBUTES_MAP, so new metadata fields are automatically covered.

Failure tests (both nullify and load modes):

KEEP — FROM test | KEEP <field>
EVAL — FROM test | EVAL x = <field>
WHERE — FROM test | WHERE <field> IS NOT NULL
SORT — FROM test | SORT <field>
STATS — FROM test | STATS x = COUNT(<field>)
RENAME — FROM test | RENAME <field> AS renamed

Complex query shapes (both modes):

After pipeline breaker — FROM test | STATS c = COUNT(*) | KEEP _score
Inside FORK branch — FROM test | FORK (WHERE _score > 1) ...
Inside subquery — FROM (FROM test | WHERE _score > 1)

Happy path (metadata declared via METADATA clause, full plan structure validated):

Nullify mode — FROM test METADATA <field> | KEEP <field>
Load mode — same

Regression suites:

AnalyzerUnmappedTests — all pass
AnalyzerTests — all pass

elasticsearchmachine · 2026-02-26T15:31:25Z

Pinging @elastic/es-analytical-engine (Team:Analytics)

alex-spies

Looks good. We can consider adding a yaml test before merging that confirms a correct error message outside of just unit tests. It's probably also fine without, though.

alex-spies · 2026-02-26T16:12:00Z

...k/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/analysis/AnalyzerUnmappedTests.java

        assertThat(Expressions.name(row.fields().getFirst()), is("x"));
    }

+    public void testFailMetadataFieldInKeep() {


This is nice, I'd suggest a couple more complex query shapes though.

For nullify, we don't disallow FORK, LOOKUP JOIN, subqueries, etc.

I'd also suggest trying to refer to the missing metadata field after (not inside) pipeline breakers.

Good idea. Added a few

Metadata fields (_score, _id, _index, etc.) were incorrectly treated as unmapped fields when SET unmapped_fields="nullify" or "load" was used. This silently returned NULL instead of producing a proper error guiding the user to add METADATA to the FROM clause. Closes elastic#141907

alex-spies

LGTM. Thank you and feel free to at your own discretion @quackaplop !

quackaplop · 2026-02-26T20:29:15Z

buildkite test this

quackaplop · 2026-02-27T10:24:49Z

buildkite test this

…elastic#143155) Metadata fields (_score, _id, _index, etc.) were incorrectly treated as unmapped fields when SET unmapped_fields="nullify" or "load" was used. This silently returned NULL instead of producing a proper error guiding the user to add METADATA to the FROM clause. Closes elastic#141907

…cations * upstream/main: (35 commits) Create ARM bulk sqrI8 implementation (elastic#142461) Rework get-snapshots predicates (elastic#143161) Refactor downsampling fetchers and producers (elastic#140357) ESQL: Unmute test and add extra logging to generative test validation (elastic#143168) Fix metadata fields being nullified/loaded by unmapped_fields setting (elastic#143155) Determine remote cluster version (elastic#142494) Populate failure message for aborted clones (elastic#143206) Allow kibana_system role to read and manage logs streams (elastic#143053) Mute org.elasticsearch.xpack.esql.CsvIT test {csv-spec:eval.DocsLength} elastic#143224 Mute org.elasticsearch.xpack.esql.CsvIT test {csv-spec:eval.DocsByteLength} elastic#143223 Mute org.elasticsearch.xpack.esql.CsvIT test {csv-spec:docs.DocsBitLength} elastic#143222 Fix FloatVectorScorerSupplier bulkScore bug (elastic#143211) ESQL: Add data node execution for external sources (elastic#143209) [ESQL] Cleanup commands docs (elastic#143058) [ML]Fix latest transforms disregarding updates when sort and sync fields are non-monotonic (elastic#142856) Mute org.elasticsearch.index.mapper.IpFieldMapperTests testSyntheticSourceInObject elastic#143212 Tests: Fix StoreDirectoryMetricsIT (elastic#143084) ESQL: Add distribution strategy for external sources (elastic#143194) CSV IT spec (elastic#142585) Fix VectorScorerOSQBenchmark.score to read corrections properly (elastic#143137) ...

alex-spies · 2026-03-02T10:54:08Z

I noticed that this should probably be backported to 9.3. because it's a bugfix and because otherwise we'll make the backport of #141340 more complex.

alex-spies · 2026-03-02T11:04:40Z

💚 All backports created successfully

Status	Branch	Result
✅	9.3

Questions ?

Please refer to the Backport tool documentation

…elastic#143155) Metadata fields (_score, _id, _index, etc.) were incorrectly treated as unmapped fields when SET unmapped_fields="nullify" or "load" was used. This silently returned NULL instead of producing a proper error guiding the user to add METADATA to the FROM clause. Closes elastic#141907 (cherry picked from commit e76c5c1) # Conflicts: # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/analysis/rules/ResolveUnmapped.java

…etting (#143155) (#143373) * Fix metadata fields being nullified/loaded by unmapped_fields setting (#143155) Metadata fields (_score, _id, _index, etc.) were incorrectly treated as unmapped fields when SET unmapped_fields="nullify" or "load" was used. This silently returned NULL instead of producing a proper error guiding the user to add METADATA to the FROM clause. Closes #141907 (cherry picked from commit e76c5c1) # Conflicts: # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/analysis/rules/ResolveUnmapped.java * Make metadata ATTRIBUTES_MAP public --------- Co-authored-by: Oleg Lvovitch <oleg.lvovitch@elastic.co>

…elastic#143155) Metadata fields (_score, _id, _index, etc.) were incorrectly treated as unmapped fields when SET unmapped_fields="nullify" or "load" was used. This silently returned NULL instead of producing a proper error guiding the user to add METADATA to the FROM clause. Closes elastic#141907

elasticsearchmachine added needs:triage Requires assignment of a team area label v9.4.0 labels Feb 26, 2026

quackaplop added the >bug label Feb 26, 2026

quackaplop force-pushed the unmapped_fields_metadata branch 2 times, most recently from af82b7c to 52d3206 Compare February 26, 2026 15:28

quackaplop added :Analytics/ES|QL AKA ESQL :Analytics/Compute Engine Analytics in ES|QL labels Feb 26, 2026

elasticsearchmachine added Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) and removed needs:triage Requires assignment of a team area label labels Feb 26, 2026

quackaplop requested review from GalLalouche and alex-spies February 26, 2026 16:02

alex-spies approved these changes Feb 26, 2026

View reviewed changes

quackaplop force-pushed the unmapped_fields_metadata branch from 52d3206 to 1fb3b9a Compare February 26, 2026 16:22

quackaplop removed the request for review from GalLalouche February 26, 2026 16:28

quackaplop force-pushed the unmapped_fields_metadata branch from 1fb3b9a to f39378f Compare February 26, 2026 16:39

quackaplop force-pushed the unmapped_fields_metadata branch from f39378f to 85e6475 Compare February 26, 2026 17:25

alex-spies approved these changes Feb 26, 2026

View reviewed changes

quackaplop added 3 commits February 26, 2026 18:19

Merge remote-tracking branch 'origin/main' into unmapped_fields_metadata

bb4ca2c

Merge remote-tracking branch 'origin/main' into unmapped_fields_metadata

c590ba0

Merge remote-tracking branch 'origin/main' into unmapped_fields_metadata

f376b2b

quackaplop added 7 commits February 26, 2026 20:29

Merge remote-tracking branch 'origin/main' into unmapped_fields_metadata

fbc438e

Merge remote-tracking branch 'origin/main' into unmapped_fields_metadata

14b4175

Merge remote-tracking branch 'origin/main' into unmapped_fields_metadata

fbb6d0b

Merge remote-tracking branch 'origin/main' into unmapped_fields_metadata

ab0aaab

Merge remote-tracking branch 'origin/main' into unmapped_fields_metadata

9d4f2ff

Merge remote-tracking branch 'origin/main' into unmapped_fields_metadata

b15a481

Merge remote-tracking branch 'origin/main' into unmapped_fields_metadata

345d043

Merge branch 'main' into unmapped_fields_metadata

cc3eb40

quackaplop merged commit e76c5c1 into elastic:main Feb 27, 2026
7 of 10 checks passed

quackaplop deleted the unmapped_fields_metadata branch February 27, 2026 12:56

prwhelan mentioned this pull request Feb 27, 2026

[Transform] Clean up internal tests #143246

Merged

alex-spies added the v9.3.2 label Mar 2, 2026

alex-spies mentioned this pull request Mar 2, 2026

[9.3] Fix metadata fields being nullified/loaded by unmapped_fields setting (#143155) #143373

Merged

alex-spies removed the :Analytics/Compute Engine Analytics in ES|QL label Mar 2, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix metadata fields being nullified/loaded by unmapped_fields setting#143155

Fix metadata fields being nullified/loaded by unmapped_fields setting#143155
quackaplop merged 12 commits intoelastic:mainfrom
quackaplop:unmapped_fields_metadata

quackaplop commented Feb 26, 2026 •

edited

Loading

Uh oh!

elasticsearchmachine commented Feb 26, 2026

Uh oh!

alex-spies left a comment

Uh oh!

alex-spies Feb 26, 2026

Uh oh!

quackaplop Feb 26, 2026

Uh oh!

alex-spies left a comment

Uh oh!

quackaplop commented Feb 26, 2026

Uh oh!

quackaplop commented Feb 27, 2026

Uh oh!

Uh oh!

alex-spies commented Mar 2, 2026

Uh oh!

alex-spies commented Mar 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

quackaplop commented Feb 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Root cause

Fix

Test plan

Uh oh!

elasticsearchmachine commented Feb 26, 2026

Uh oh!

alex-spies left a comment

Choose a reason for hiding this comment

Uh oh!

alex-spies Feb 26, 2026

Choose a reason for hiding this comment

Uh oh!

quackaplop Feb 26, 2026

Choose a reason for hiding this comment

Uh oh!

alex-spies left a comment

Choose a reason for hiding this comment

Uh oh!

quackaplop commented Feb 26, 2026

Uh oh!

quackaplop commented Feb 27, 2026

Uh oh!

Uh oh!

alex-spies commented Mar 2, 2026

Uh oh!

alex-spies commented Mar 2, 2026

💚 All backports created successfully

Questions ?

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

quackaplop commented Feb 26, 2026 •

edited

Loading