Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce TreeNode::exists() API, avoid copying expressions #10008

Merged
merged 5 commits into from
Apr 9, 2024

Conversation

peter-toth
Copy link
Contributor

Which issue does this PR close?

Part of #8913.

Rationale for this change

This PR adds TreeNode::exists() API as a common pattern to check the presence of a matching node for a predicate in a tree.
This PR also solves the issue found by @alamb here: #9913 (comment). Since #9913 there is no need to call LogicalPlan::expressions() (that copies the plan's expressions) at certain places, but iterating over the expressions with LogicalPlan::apply_expressions() is enough.

What changes are included in this PR?

  • Adds the TreeNode::exists() API.
  • Replaces contains_outer_reference helper with a better solution.

Are these changes tested?

Yes, with existing UTs.

Are there any user-facing changes?

No.

@github-actions github-actions bot added logical-expr Logical plan and expressions optimizer Optimizer rules labels Apr 9, 2024
@peter-toth
Copy link
Contributor Author

@alamb, I think there are other places where LogicalPlan::expressions() can be replaced with the faster LogicalPlan::apply_expressions() but the change is non-trivial, so let's start with this smaller PR first.

@alamb alamb changed the title Introduce TreeNode::exists() API Introduce TreeNode::exists() API, avoid copying expressions Apr 9, 2024
Copy link
Contributor

@alamb alamb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @peter-toth -- this looks great! (and thank you for keeping the PRs small).

I have some comment suggestions, but I also think we could add them as a follow on PR too

datafusion/common/src/tree_node.rs Show resolved Hide resolved
@@ -233,13 +233,6 @@ fn check_inner_plan(
}
}

fn contains_outer_reference(inner_plan: &LogicalPlan) -> bool {
inner_plan
.expressions()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❤️ for removing this copy 🚀

datafusion/expr/src/logical_plan/plan.rs Show resolved Hide resolved
@@ -91,7 +91,7 @@ impl TreeNodeRewriter for PullUpCorrelatedExpr {
_ => Ok(Transformed::no(plan)),
}
}
_ if plan.expressions().iter().any(|expr| expr.contains_outer()) => {
_ if plan.contains_outer_reference() => {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And another copy gone!

@alamb alamb changed the title Introduce TreeNode::exists() API, avoid copying expressions Introduce TreeNode::exists() API, avoid copying expressions Apr 9, 2024
Copy link
Contributor

@alamb alamb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I took the liberty of adding the doc comments directly to this PR. Thanks again @peter-toth

@alamb alamb merged commit 57b2656 into apache:main Apr 9, 2024
24 checks passed
@peter-toth
Copy link
Contributor Author

I took the liberty of adding the doc comments directly to this PR. Thanks again @peter-toth

Thanks @alamb for the doc comments and the review! Actually, I wanted to ask you to do that as I'm travelling this week and have limited access to github.

@alamb
Copy link
Contributor

alamb commented Apr 9, 2024

Thank you @peter-toth for all your help

This pull request was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
logical-expr Logical plan and expressions optimizer Optimizer rules
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants