[core][spark] Support push down transform predicate #6506
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Zouxxyy
force-pushed
the
dev/spark-transform
branch
4 times, most recently
from
8b37699 to
fe5b4e9
Compare
Zouxxyy
force-pushed
the
dev/spark-transform
branch
from
fe5b4e9 to
7b13471
Compare
Zouxxyy
commented
Nov 1, 2025
| /** Represents a transform function. */ | ||
| public interface Transform extends Serializable { | ||
|
|
||
| List<Object> inputs(); |
There was a problem hiding this comment.
I believe we can introduce a Literal object in the future, so that our inputs will only be of either Literal or FieldRef type now.
|
|
||
| /** Leaf node of a {@link Predicate} tree. Compares a field in the row with literals. */ | ||
| public class LeafPredicate implements Predicate { | ||
| public class LeafPredicate extends TransformPredicate { |
There was a problem hiding this comment.
Since LeafPredicate extends TransformPredicate, and we have
T visit(LeafPredicate predicate);
T visit(TransformPredicate predicate);
Maybe we can combine them in the future
There was a problem hiding this comment.
Pull Request Overview
This PR adds support for pushing down CONCAT expressions to Paimon for Spark 3.4+. The implementation introduces a transform-based predicate system that can handle not only simple field references but also general expressions like CONCAT.
Key Changes:
- Introduced a new
Transforminterface hierarchy to represent field transformations and expressions - Refactored predicate handling to work with
Transformobjects instead of just field indices - Added support for CONCAT expression pushdown in Spark filters
Reviewed Changes
Copilot reviewed 16 out of 16 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
SparkExpressionConverter.scala |
New utility to convert Spark expressions to Paimon transforms and literals |
SparkV2FilterConverter.scala |
Refactored to use Transform-based predicates instead of field indices |
PredicateBuilder.java |
Added Transform-based predicate builder methods alongside index-based ones |
TransformPredicate.java |
Enhanced with factory method and field name extraction |
Transform.java |
Added withNewInputs method for transform reconstruction |
FieldTransform.java |
New class representing field extraction as a Transform |
ConcatTransform.java |
Added withNewInputs implementation |
ConcatWsTransform.java |
Added withNewInputs implementation |
LeafPredicate.java |
Refactored to extend TransformPredicate |
PaimonPushDownTestBase.scala |
Added test for CONCAT pushdown |
PaimonCatalogUtils.scala |
Removed unused imports |
ExpressionHelper.scala |
Removed unused import |
PaimonBasePushDown.scala |
Removed unused import and improved code style |
PaimonScan.scala |
Updated isSupportedRuntimeFilter call to use instance method |
PaimonBaseScan.scala |
Removed unused imports |
TransformPredicateTest.java |
Updated to use TransformPredicate.of factory method |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
JingsongLi
reviewed
Nov 3, 2025
| } | ||
|
|
||
| @Override | ||
| public Transform withNewInputs(List<Object> inputs) { |
|
|
||
| /** Transform that extracts a field from a row. */ | ||
| public class FieldTransform implements Transform { | ||
|
|
| public class LeafPredicate implements Predicate { | ||
| public class LeafPredicate extends TransformPredicate { | ||
|
|
||
| private static final long serialVersionUID = 1L; |
|
+1 |
gmdfalk
added a commit
to gmdfalk/paimon
that referenced
this pull request
Nov 5, 2025
* master: (162 commits) [Python] Rename to BATCH_COMMIT_IDENTIFIER in snapshot.py [Python] Suppport multi prepare commit in the same TableWrite (apache#6526) [spark] Fix drop temporary view (apache#6529) [core] skip validate main branch before orphan files cleaning (apache#6524) [core][spark] Introduce upper transform (apache#6521) [Python] Keep the variable names of Identifier consistent with Java (apache#6520) [core] Remove hash lookup to simplify interface (apache#6519) [core][format] Format Table plan partitions should ignore hidden & illegal dirs (apache#6522) [hotfix] Print partition spec and type when error in InternalRowPartitionComputer [hotfix] Add more informat to check partition spec in InternalRowPartitionComputer [hotfix] Use deleteDirectoryQuietly in TempFileCommitter.clean [core] format table: support write file in _temporary at first (apache#6510) [core] Support non null column with write type (apache#6513) [core][fix] Blob with rolling file failed (apache#6518) [core][rest] Support schema validation and infer for external paimon table (apache#6501) [hotfix] Correct visitors for TransformPredicate [hotfix] Rename to copy from withNewInputs in TransformPredicate [core][spark] Support push down transform predicate (apache#6506) [spark] Implement SupportsReportStatistics for PaimonFormatTableBaseScan (apache#6515) [docs] add docs for auto-clustering of historical partitions (apache#6516) ...
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Purpose
Tests
API and Format
Documentation