Skip to content

Fix metadata fields being nullified/loaded by unmapped_fields setting #143155

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
quackaplop merged 12 commits into from
Feb 27, 2026
Merged

Fix metadata fields being nullified/loaded by unmapped_fields setting #143155

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
85e6475
Fix metadata fields being nullified/loaded by unmapped_fields setting
quackaplop Feb 26, 2026
bb4ca2c
Merge remote-tracking branch 'origin/main' into unmapped_fields_metadata
quackaplop Feb 26, 2026
c590ba0
Merge remote-tracking branch 'origin/main' into unmapped_fields_metadata
quackaplop Feb 26, 2026
f376b2b
Merge remote-tracking branch 'origin/main' into unmapped_fields_metadata
quackaplop Feb 26, 2026
fbc438e
Merge remote-tracking branch 'origin/main' into unmapped_fields_metadata
quackaplop Feb 26, 2026
14b4175
Merge remote-tracking branch 'origin/main' into unmapped_fields_metadata
quackaplop Feb 26, 2026
fbb6d0b
Merge remote-tracking branch 'origin/main' into unmapped_fields_metadata
quackaplop Feb 26, 2026
ab0aaab
Merge remote-tracking branch 'origin/main' into unmapped_fields_metadata
quackaplop Feb 26, 2026
9d4f2ff
Merge remote-tracking branch 'origin/main' into unmapped_fields_metadata
quackaplop Feb 27, 2026
b15a481
Merge remote-tracking branch 'origin/main' into unmapped_fields_metadata
quackaplop Feb 27, 2026
345d043
Merge remote-tracking branch 'origin/main' into unmapped_fields_metadata
quackaplop Feb 27, 2026
cc3eb40
Merge branch 'main' into unmapped_fields_metadata
quackaplop Feb 27, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion ...lugin/esql/src/main/java/org/elasticsearch/xpack/esql/analysis/rules/ResolveUnmapped.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
import org.elasticsearch.xpack.esql.core.expression.Expressions;
import org.elasticsearch.xpack.esql.core.expression.FieldAttribute;
import org.elasticsearch.xpack.esql.core.expression.Literal;
import org.elasticsearch.xpack.esql.core.expression.MetadataAttribute;
import org.elasticsearch.xpack.esql.core.expression.NameId;
import org.elasticsearch.xpack.esql.core.expression.NamedExpression;
import org.elasticsearch.xpack.esql.core.expression.UnresolvedAttribute;
Expand Down Expand Up @@ -282,7 +283,9 @@ private static LinkedHashSet<UnresolvedAttribute> unresolvedLinkedSet(List<Unres
public static List<UnresolvedAttribute> collectUnresolved(LogicalPlan plan) {
List<UnresolvedAttribute> unresolved = new ArrayList<>();
Consumer<UnresolvedAttribute> collectUnresolved = ua -> {
if ((ua instanceof UnresolvedPattern || ua instanceof UnresolvedTimestamp) == false) {
// Exclude metadata fields so they fail with a proper verification error instead of being silently nullified/loaded.
if ((ua instanceof UnresolvedPattern || ua instanceof UnresolvedTimestamp) == false
&& MetadataAttribute.isSupported(ua.name()) == false) {
unresolved.add(ua);
}
};
Expand Down
137 changes: 137 additions & 0 deletions ...lugin/esql/src/test/java/org/elasticsearch/xpack/esql/analysis/AnalyzerUnmappedTests.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@
import org.elasticsearch.xpack.esql.core.expression.FieldAttribute;
import org.elasticsearch.xpack.esql.core.expression.FoldContext;
import org.elasticsearch.xpack.esql.core.expression.Literal;
import org.elasticsearch.xpack.esql.core.expression.MetadataAttribute;
import org.elasticsearch.xpack.esql.core.expression.ReferenceAttribute;
import org.elasticsearch.xpack.esql.core.expression.UnresolvedTimestamp;
import org.elasticsearch.xpack.esql.core.type.DataType;
Expand Down Expand Up @@ -62,6 +63,7 @@
import static org.hamcrest.Matchers.containsString;
import static org.hamcrest.Matchers.hasItems;
import static org.hamcrest.Matchers.hasSize;
import static org.hamcrest.Matchers.instanceOf;
import static org.hamcrest.Matchers.is;
import static org.hamcrest.Matchers.not;
import static org.hamcrest.Matchers.nullValue;
Expand Down Expand Up @@ -3303,6 +3305,141 @@ public void testRow() {
assertThat(Expressions.name(row.fields().getFirst()), is("x"));
}

public void testFailMetadataFieldInKeep() {
Copy link
Copy Markdown
Contributor

@alex-spies alex-spies Feb 26, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is nice, I'd suggest a couple more complex query shapes though.

For nullify, we don't disallow FORK, LOOKUP JOIN, subqueries, etc.

I'd also suggest trying to refer to the missing metadata field after (not inside) pipeline breakers.

Copy link
Copy Markdown
Contributor Author

@quackaplop quackaplop Feb 26, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good idea. Added a few

for (String field : MetadataAttribute.ATTRIBUTES_MAP.keySet()) {
var query = "FROM test | KEEP " + field;
var failure = "Unknown column [" + field + "]";
verificationFailure(setUnmappedNullify(query), failure);
verificationFailure(setUnmappedLoad(query), failure);
}
}

public void testFailMetadataFieldInEval() {
for (String field : MetadataAttribute.ATTRIBUTES_MAP.keySet()) {
var query = "FROM test | EVAL x = " + field;
var failure = "Unknown column [" + field + "]";
verificationFailure(setUnmappedNullify(query), failure);
verificationFailure(setUnmappedLoad(query), failure);
}
}

public void testFailMetadataFieldInWhere() {
for (String field : MetadataAttribute.ATTRIBUTES_MAP.keySet()) {
var query = "FROM test | WHERE " + field + " IS NOT NULL";
var failure = "Unknown column [" + field + "]";
verificationFailure(setUnmappedNullify(query), failure);
verificationFailure(setUnmappedLoad(query), failure);
}
}

public void testFailMetadataFieldInSort() {
for (String field : MetadataAttribute.ATTRIBUTES_MAP.keySet()) {
var query = "FROM test | SORT " + field;
var failure = "Unknown column [" + field + "]";
verificationFailure(setUnmappedNullify(query), failure);
verificationFailure(setUnmappedLoad(query), failure);
}
}

public void testFailMetadataFieldInStats() {
for (String field : MetadataAttribute.ATTRIBUTES_MAP.keySet()) {
var query = "FROM test | STATS x = COUNT(" + field + ")";
var failure = "Unknown column [" + field + "]";
verificationFailure(setUnmappedNullify(query), failure);
verificationFailure(setUnmappedLoad(query), failure);
}
}

public void testFailMetadataFieldInRename() {
for (String field : MetadataAttribute.ATTRIBUTES_MAP.keySet()) {
var query = "FROM test | RENAME " + field + " AS renamed";
var failure = "Unknown column [" + field + "]";
verificationFailure(setUnmappedNullify(query), failure);
verificationFailure(setUnmappedLoad(query), failure);
}
}

public void testFailMetadataFieldAfterStats() {
var query = """
FROM test
| STATS c = COUNT(*)
| KEEP _score
""";
var failure = "Unknown column [_score]";
verificationFailure(setUnmappedNullify(query), failure);
verificationFailure(setUnmappedLoad(query), failure);
}

public void testFailMetadataFieldInFork() {
var query = """
FROM test
| FORK (WHERE _score > 1)
(WHERE salary > 50000)
""";
var failure = "Unknown column [_score]";
verificationFailure(setUnmappedNullify(query), failure);
verificationFailure(setUnmappedLoad(query), failure);
}

public void testFailMetadataFieldInSubquery() {
assumeTrue("Requires subquery in FROM command support", EsqlCapabilities.Cap.SUBQUERY_IN_FROM_COMMAND.isEnabled());

var query = """
FROM
(FROM test
| WHERE _score > 1)
""";
var failure = "Unknown column [_score]";
verificationFailure(setUnmappedNullify(query), failure);
verificationFailure(setUnmappedLoad(query), failure);
}

/*
* Limit[1000[INTEGER],false,false]
* \_Project[[_score{m}#5]]
* \_EsRelation[test][_meta_field{f}#11, emp_no{f}#5, ...]
*/
public void testMetadataFieldDeclaredNullify() {
for (String field : MetadataAttribute.ATTRIBUTES_MAP.keySet()) {
var plan = analyzeStatement(setUnmappedNullify("FROM test METADATA " + field + " | KEEP " + field));

var limit = as(plan, Limit.class);
assertThat(limit.limit().fold(FoldContext.small()), is(1000));

var project = as(limit.child(), Project.class);
assertThat(project.projections(), hasSize(1));
assertThat(Expressions.name(project.projections().getFirst()), is(field));
assertThat(project.projections().getFirst(), instanceOf(MetadataAttribute.class));

// No Eval(NULL) — the field was resolved via METADATA, not nullified
var relation = as(project.child(), EsRelation.class);
assertThat(relation.indexPattern(), is("test"));
}
}

/*
* Limit[1000[INTEGER],false,false]
* \_Project[[_score{m}#5]]
* \_EsRelation[test][_meta_field{f}#11, emp_no{f}#5, ...]
*/
public void testMetadataFieldDeclaredLoad() {
for (String field : MetadataAttribute.ATTRIBUTES_MAP.keySet()) {
var plan = analyzeStatement(setUnmappedLoad("FROM test METADATA " + field + " | KEEP " + field));

var limit = as(plan, Limit.class);
assertThat(limit.limit().fold(FoldContext.small()), is(1000));

var project = as(limit.child(), Project.class);
assertThat(project.projections(), hasSize(1));
assertThat(Expressions.name(project.projections().getFirst()), is(field));
assertThat(project.projections().getFirst(), instanceOf(MetadataAttribute.class));

// The field was resolved via METADATA, not loaded as an unmapped field into EsRelation
var relation = as(project.child(), EsRelation.class);
assertThat(relation.indexPattern(), is("test"));
}
}

public void testChangedTimestmapFieldWithRate() {
verificationFailure(setUnmappedNullify("""
TS k8s
Expand Down