Is there a join type that can pushdown join filter to one side's scan? #7158
alexwilcoxson-rel
started this conversation in
Ideas
Replies: 1 comment 1 reply
-
To the best of my knowledge this is not supported today by DataFusion -- I think this type of filter pushdown is called "sideways information passing" in database literature and is a typical technique to improve the performance for column stores. I think it has come up before in the context of improving join performance (for example, on the TPCH benchmarks -- cc @Dandandan and @mingmwang ) @alexwilcoxson-rel would you be willing to help file a feature request for this item? I can provide some additional commentary / pointers as well |
Beta Was this translation helpful? Give feedback.
1 reply
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
I was looking through the join options' code and it didn't look like this was possible. e.g.
LHS table produces small number of rows. Collect these and provide them as a filter pushed down to RHS scan to reduce files read.
Beta Was this translation helpful? Give feedback.
All reactions