[ESQL] Per-file filter pushdown awareness by costin · Pull Request #145755 · elastic/elasticsearch

costin · 2026-04-06T17:54:48Z

Make filter pushdown aware of per-file column availability and
types in UNION_BY_NAME scenarios. Files whose filter columns are
entirely absent are skipped at split discovery time. For files
that do contain the columns, pushed ESQL expressions are adapted
to the file's column set and re-translated to format-native
filters. Type-widened columns (e.g. INTEGER file vs LONG unified)
have their filter literals downcast with overflow detection.

Developed with AI-assisted tooling

elasticsearchmachine · 2026-04-06T17:55:14Z

Pinging @elastic/es-analytical-engine (Team:Analytics)

elasticsearchmachine · 2026-04-06T17:55:15Z

Hi @costin, I've created a changelog YAML for you.

github-actions · 2026-04-06T17:56:03Z

🔍 Preview links for changed docs

⏳ Building and deploying preview... View progress

This comment will be updated with preview links when the build is complete.

github-actions · 2026-04-06T17:58:15Z

ℹ️ Important: Docs version tagging

👋 Thanks for updating the docs! Just a friendly reminder that our docs are now cumulative. This means all 9.x versions are documented on the same page and published off of the main branch, instead of creating separate pages for each minor version.

We use applies_to tags to mark version-specific features and changes.

Expand for a quick overview

When to use applies_to tags:

✅ At the page level to indicate which products/deployments the content applies to (mandatory)
✅ When features change state (e.g. preview, ga) in a specific version
✅ When availability differs across deployments and environments

What NOT to do:

❌ Don't remove or replace information that applies to an older version
❌ Don't add new information that applies to a specific version without an applies_to tag
❌ Don't forget that applies_to tags can be used at the page, section, and inline level

🤔 Need help?

Check out the cumulative docs guidelines
Reach out in the #docs Slack channel

Make filter pushdown aware of per-file column availability and types in UNION_BY_NAME scenarios. Files whose filter columns are entirely absent are skipped at split discovery time. For files that do contain the columns, pushed ESQL expressions are adapted to the file's column set and re-translated to format-native filters. Type-widened columns (e.g. INTEGER file vs LONG unified) have their filter literals downcast with overflow detection.

elasticsearchmachine · 2026-04-07T18:39:22Z

Hi @costin, I've created a changelog YAML for you.

bpintea · 2026-04-08T10:05:27Z

...c/main/java/org/elasticsearch/xpack/esql/datasources/AsyncExternalSourceOperatorFactory.java

+    /**
+     * Infers the file's native type from the unified attribute type and the cast target.
+     * The cast target is the unified (wider) type; the file has the narrower type.
+     */
+    /**
+     * Infers the file's native type from the cast target. Only returns a narrower type when
+     * the adaptation is safe for integral comparisons (LONG→INTEGER). DOUBLE→INTEGER narrowing
+     * is not supported because literal truncation can cause incorrect predicate semantics.
+     */


Nit: redundancy

Fixed — merged the two javadoc blocks into one.

bpintea · 2026-04-08T10:07:25Z

...c/main/java/org/elasticsearch/xpack/esql/datasources/AsyncExternalSourceOperatorFactory.java

+        // DOUBLE→INTEGER narrowing is intentionally not supported: Number.longValue() truncates
+        // fractional values, which can change comparison semantics (e.g., col < 2.7 vs col < 2).


Nit: if the methods stays, this can be a javadoc comment and the method simplified.

Done — moved inline comment into the javadoc and removed the redundant one.

bpintea · 2026-04-08T10:07:36Z

...c/main/java/org/elasticsearch/xpack/esql/datasources/AsyncExternalSourceOperatorFactory.java

+     * the adaptation is safe for integral comparisons (LONG→INTEGER). DOUBLE→INTEGER narrowing
+     * is not supported because literal truncation can cause incorrect predicate semantics.
+     */
+    private static DataType inferFileType(DataType unifiedType, DataType castTarget) {


unifiedType isn't used.

Removed — the parameter was left over from when DOUBLE→INTEGER was considered.

bpintea · 2026-04-08T14:55:51Z

...c/main/java/org/elasticsearch/xpack/esql/datasources/AsyncExternalSourceOperatorFactory.java

+        if (adapted.isEmpty()) {
+            return formatReader.withPushedFilter(null);
+        }
+        FilterPushdownSupport.PushdownResult result = pushdownSupport.pushFilters(adapted);


Would probably be useful if we could cash the resolution at a level higher than per file (at some point).

Agreed — we could cache the adapted PushdownResult keyed on the set of missing/widened columns so files with identical schemas share one translation.

* upstream/main: Mute org.elasticsearch.xpack.esql.expression.function.aggregate.FirstDocIdGroupingAggregatorFunctionTests testSimple elastic#145923 Reindex relocation: store source TaskResult at destination node (elastic#145488) Bump versions after 9.2.8 release [CI] DLMFrozenTransitionServiceTests testCheckForFrozenIndicesReturnsEarlyWhenCapacityExhausted failing [elastic#145778] (elastic#145906) Update branches.json for 9.2.8 release ESQL: Clarify inheriting from Attributes (elastic#145898) Bump versions after 9.3.3 release Update branches.json for 9.3.3 release Prune changelogs after 8.19.14 release Bump versions after 8.19.14 release Update branches.json for 8.19.14 release [ML] Call old inference API (elastic#145690) ESQL: Unmute CsvIT sumWithOverflowRow (elastic#145893) Index a document when testing runtime fields shadowing dimensions & metrics (elastic#145882) [TEST] Fix version check in testSequenceNumbersDisabled (elastic#145879) [ESQL] Per-file filter pushdown awareness (elastic#145755) Unmute testGetReindexFollowsRelocation (elastic#145841) Correctly ignore system indices when validating dot-prefixed indices (elastic#128868) [Transform] Remove tests for deleted code (elastic#145685) ESQL: Add generative tests for LIMIT BY (elastic#144238)

costin added >enhancement :Analytics/ES|QL AKA ESQL v9.4.0 ES|QL|DS ES|QL datasources labels Apr 6, 2026

costin requested a review from bpintea April 6, 2026 17:54

costin enabled auto-merge (squash) April 6, 2026 17:55

elasticsearchmachine added the Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) label Apr 6, 2026

costin closed this Apr 6, 2026

auto-merge was automatically disabled April 6, 2026 17:59
Pull request was closed

costin reopened this Apr 6, 2026

costin force-pushed the esql/per-file-filter-pushdown branch from bdbbba9 to 151f2a2 Compare April 7, 2026 18:38

costin enabled auto-merge (squash) April 7, 2026 18:39

Update docs/changelog/145755.yaml

dcf93fc

bpintea approved these changes Apr 8, 2026

View reviewed changes

costin merged commit 82359d3 into elastic:main Apr 8, 2026
34 of 35 checks passed

costin deleted the esql/per-file-filter-pushdown branch April 8, 2026 15:05

costin mentioned this pull request Apr 8, 2026

[ESQL] Fix filter pushdown review nits #145924

Open

		// DOUBLE→INTEGER narrowing is intentionally not supported: Number.longValue() truncates
		// fractional values, which can change comparison semantics (e.g., col < 2.7 vs col < 2).

Conversation

costin commented Apr 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticsearchmachine commented Apr 6, 2026

Uh oh!

elasticsearchmachine commented Apr 6, 2026

Uh oh!

github-actions bot commented Apr 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔍 Preview links for changed docs

Uh oh!

github-actions bot commented Apr 6, 2026

ℹ️ Important: Docs version tagging

When to use applies_to tags:

What NOT to do:

🤔 Need help?

Uh oh!

elasticsearchmachine commented Apr 7, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

costin commented Apr 6, 2026 •

edited

Loading

github-actions bot commented Apr 6, 2026 •

edited

Loading