Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Push down filter for Unnest plan #10974

Merged
merged 4 commits into from
Jun 19, 2024
Merged

Conversation

jayzhan211
Copy link
Contributor

Which issue does this PR close?

Closes #10935.

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Signed-off-by: jayzhan211 <[email protected]>
@github-actions github-actions bot added optimizer Optimizer rules sqllogictest SQL Logic Tests (.slt) labels Jun 18, 2024
@jayzhan211 jayzhan211 marked this pull request as ready for review June 18, 2024 12:29
Copy link
Contributor

@alamb alamb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @jayzhan211 -- very nice. I had a suggestion for improving the docs, and I think we should add a few more test cases, but otherwise I think this is ready to go

statement ok
CREATE TABLE IF NOT EXISTS v AS VALUES(1,[1,2,3]),(2,[3,4,5]);

# filter plan is pushed down into projection plan
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please also add a negative test with a predicate on uc2? Something like

explain select uc2 from (select unnest(column2) as uc2, column1 from v) where uc2  > 3;

it might also be good to add tests to actually run those queries

Copy link
Contributor Author

@jayzhan211 jayzhan211 Jun 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We hadn't fix the comparison for unnest yet

query error DataFusion error: External error: Arrow error: Invalid argument error: Invalid comparison operation: List\(Field \{ name: "item", data_type: Int64, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: \{\} \}\) > Int64
select uc2 from (select unnest(column2) as uc2, column1 from v) where uc2 > 3;

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will leave the query in slt, and fix it in the follow up PR

@@ -951,6 +939,54 @@ impl OptimizerRule for PushDownFilter {
}
}

/// return transformed projection plan and optinal filter's predicate
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a nice refactor. While reading it I figured it might also be nice to improve the documenatation

Suggested change
/// return transformed projection plan and optinal filter's predicate
/// Attempts to push `predicate` into a `FilterExec` below `projection
///
/// # Returns
/// (plan, remaining_predicate)
///
/// `plan` is a LogicalPlan for `projection` with possibly a new FilterExec below it.
/// `remaining_predicate` is any part of the predicate that could not be pushed down
///
/// # Example
///
/// Pushing a predicate like `foo=5 AND bar=6` with an input plan like this:
///
/// ```text
/// Projection(foo, c+d as bar)
/// ```
///
/// Might result in returning `remaining_predicate` of `bar=6` and a plan like
///
/// ```text
/// Projection(foo, c+d as bar)
/// Filter(foo=5)
/// ...
/// ```

@jayzhan211 jayzhan211 merged commit 4109f58 into apache:main Jun 19, 2024
23 checks passed
@jayzhan211 jayzhan211 deleted the unnest-filter branch June 19, 2024 03:36
findepi pushed a commit to findepi/datafusion that referenced this pull request Jul 16, 2024
* add unnset

Signed-off-by: jayzhan211 <[email protected]>

* improve doc and tset

Signed-off-by: jayzhan211 <[email protected]>

* fix test

Signed-off-by: jayzhan211 <[email protected]>

---------

Signed-off-by: jayzhan211 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
optimizer Optimizer rules sqllogictest SQL Logic Tests (.slt)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Push down filters below Unnest in sub queries
2 participants