[SPARK-34014][SQL] Ignore Distinct if it is the right child of a left semi or anti join#31045
[SPARK-34014][SQL] Ignore Distinct if it is the right child of a left semi or anti join#31045tanelk wants to merge 3 commits into
Conversation
|
Test build #133688 has finished for PR 31045 at commit
|
|
retest this please |
|
Kubernetes integration test starting |
|
Kubernetes integration test status failure |
| * SELECT DISTINCT f1, f2 FROM t ==> SELECT f1, f2 FROM t GROUP BY f1, f2 | ||
| * }}} | ||
| * | ||
| * The [[Distinct]] can be ignored if it is the right child of a left semi or anti join. |
There was a problem hiding this comment.
nit: could you move this comment between L1619/L1620?
// The [[Distinct]] can be ignored if it is the right child of a left semi or anti join.
case j @ Join(_, Distinct(right), LeftSemiOrAnti(_), _, _) => j.copy(right = right)
| object ReplaceDistinctWithAggregate extends Rule[LogicalPlan] { | ||
| def apply(plan: LogicalPlan): LogicalPlan = plan transform { | ||
| def apply(plan: LogicalPlan): LogicalPlan = plan transformDown { | ||
| case j @ Join(_, Distinct(right), LeftSemiOrAnti(_), _, _) => j.copy(right = right) |
There was a problem hiding this comment.
Just out of curiosity, this rule can always make performance better? For example, how about the case where a right table has too many duplicates?
There was a problem hiding this comment.
I benchmarked a bit and you are right - this can cause a performance regression. For example q87 took a 10% hit on sf100. I think it's best to close this PR and issue.
|
Test build #133690 has finished for PR 31045 at commit
|
3 participants
What changes were proposed in this pull request?
Remove
Distinctfrom the right child of left semi and anti joins.Why are the changes needed?
Left semi and anti joins ignore duplicates in the right child. Finding distinct values there will only add overhead.
This pattern is present in some TPCDS queries.
Does this PR introduce any user-facing change?
No
How was this patch tested?
UT