You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The SQL Query Planner and Logical Plan need to implement these features. Note that DataFusion does not necessarily need to implement a physical plan for these features, so that reduces the scope of this work.
We do not need these for the benchmark suite but these are features and bugs that we are likely to eventually run into so it makes sense to be proactive and work on these.
We currently bring in the full datafusion crate as a dependency, including the physical plans and execution engine. We really should just depend on the features necessary for SQL query planning and logical plan optimization. These are the issues that need to be implemented to achieve that.
Thanks @alamb. Strictly speaking, we're not blocked on this because we are only using the SQL query planner and the logical optimization rules. However, implementing the physical plan for OFFSET and adding some integration tests would give me greater confidence that the planning and optimization rules for LIMIT and OFFSET are correct, so I will add it to the list.
The purpose of this issue is to track the work that we need to do in the DataFusion project to support moving the dask-sql planner to DataFusion.
High Priority Tech Debt
We need to fix some issues before we can really get started on the main features.
physical-expr
crate toexpr
crate apache/datafusion#2251LogicalPlan::TableScan
should not depend on the physical plan apache/datafusion#2247High Priority Features
The SQL Query Planner and Logical Plan need to implement these features. Note that DataFusion does not necessarily need to implement a physical plan for these features, so that reduces the scope of this work.
LogicalPlan::TableScan
should not depend on the physical plan apache/datafusion#2247Query
for datafusion apache/datafusion#2181Expr::InSubquery
andExpr::ScalarSubquery
apache/datafusion#2342ROLLUP
andCUBE
grouping sets in SQL query planner and logical plan apache/datafusion#2378grouping
aggregate function in the SQL planner apache/datafusion#2477Decimal
toFloat64
apache/datafusion#2380High Priority Bugs
These are the bugs that we are seeing when attempting to parse all the queries from our benchmark suite.
round
function with two arguments apache/datafusion#2420Ongoing improvements
We do not need these for the benchmark suite but these are features and bugs that we are likely to eventually run into so it makes sense to be proactive and work on these.
OFFSET
in SQL query planner + logical plan apache/datafusion#2377order by
expression that references complexgroup by
expression apache/datafusion#2360UNION
vsUNION ALL
(introduce a LogicalPlan::Distinct) apache/datafusion#2573Refactoring of DataFusion crates
We currently bring in the full datafusion crate as a dependency, including the physical plans and execution engine. We really should just depend on the features necessary for SQL query planning and logical plan optimization. These are the issues that need to be implemented to achieve that.
ExecutionProps
fromOptimizerRule
trait apache/datafusion#2614Lower Priority Tech Debt / Misc Other Items
The text was updated successfully, but these errors were encountered: