[SPARK-41112][SQL] RuntimeFilter should apply ColumnPruning eagerly with in-subquery filter #38619
Conversation
| case Join(_, agg: Aggregate, LeftSemi, _, _) => agg | ||
| } | ||
| assert(agg.size == 1) | ||
| assert(agg.head.fastEquals(ColumnPruning(agg.head))) |
There was a problem hiding this comment.
this test can pass without this pr because
I think it is just a coincidence since we converted the subquery to left semi join ..
|
cc @wangyum @cloud-fan @sigmod thank you |
| val actualFilterKeyExpr = mayWrapWithHash(filterCreationSideExp) | ||
| val alias = Alias(actualFilterKeyExpr, actualFilterKeyExpr.toString)() | ||
| val aggregate = Aggregate(Seq(alias), Seq(alias), filterCreationSidePlan) | ||
| val aggregate = ColumnPruning(Aggregate(Seq(alias), Seq(alias), filterCreationSidePlan)) |
There was a problem hiding this comment.
This looks a bit hacky. Can we use the catalyst framework to optimize it?
There was a problem hiding this comment.
I'm afraid that will be overkill to apply the whole optimizer rules. The filterCreationSidePlan is simple enough, it can only contain project and filter. Apply ColumnPruning here seems more safer ?
There was a problem hiding this comment.
how about the filter push rule?
There was a problem hiding this comment.
The filterCreationSidePlan has been optimized , so it should be done if there is a filter can be pushed.
I think one more useful rule is CollapseProject, but it should be fine not to apply here since PhysicalOperation support collect adjacent projects.
|
thanks, merging to master! |
|
thank you @cloud-fan @dongjoon-hyun |
…ith in-subquery filter ### What changes were proposed in this pull request? Apply ColumnPruning for in subquery filter. Note that, the bloom filter side has already fixed by apache#36047 ### Why are the changes needed? The inferred in-subquery filter should apply ColumnPruning before get plan statistics and check if can be broadcasted. Otherwise, the final physical plan will be different from expected. ### Does this PR introduce _any_ user-facing change? no ### How was this patch tested? add test Closes apache#38619 from ulysses-you/SPARK-41112. Authored-by: ulysses-you <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>
3 participants
What changes were proposed in this pull request?
Apply ColumnPruning for in subquery filter.
Note that, the bloom filter side has already fixed by #36047
Why are the changes needed?
The inferred in-subquery filter should apply ColumnPruning before get plan statistics and check if can be broadcasted. Otherwise, the final physical plan will be different from expected.
Does this PR introduce any user-facing change?
no
How was this patch tested?
add test