Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DF] DataFusion Missing Features & Bugs #474

Open
40 of 42 tasks
andygrove opened this issue Apr 14, 2022 · 2 comments
Open
40 of 42 tasks

[DF] DataFusion Missing Features & Bugs #474

andygrove opened this issue Apr 14, 2022 · 2 comments
Labels
datafusion Related to work in DataFusion

Comments

@andygrove
Copy link
Contributor

andygrove commented Apr 14, 2022

The purpose of this issue is to track the work that we need to do in the DataFusion project to support moving the dask-sql planner to DataFusion.

High Priority Tech Debt

We need to fix some issues before we can really get started on the main features.

High Priority Features

The SQL Query Planner and Logical Plan need to implement these features. Note that DataFusion does not necessarily need to implement a physical plan for these features, so that reduces the scope of this work.

High Priority Bugs

These are the bugs that we are seeing when attempting to parse all the queries from our benchmark suite.

Ongoing improvements

We do not need these for the benchmark suite but these are features and bugs that we are likely to eventually run into so it makes sense to be proactive and work on these.

Refactoring of DataFusion crates

We currently bring in the full datafusion crate as a dependency, including the physical plans and execution engine. We really should just depend on the features necessary for SQL query planning and logical plan optimization. These are the issues that need to be implemented to achieve that.

Lower Priority Tech Debt / Misc Other Items

@andygrove andygrove added enhancement New feature or request needs triage Awaiting triage by a dask-sql maintainer labels Apr 14, 2022
@jdye64 jdye64 added datafusion Related to work in DataFusion and removed enhancement New feature or request needs triage Awaiting triage by a dask-sql maintainer labels Apr 14, 2022
@alamb
Copy link

alamb commented May 24, 2022

apache/datafusion#2551 might also belong listed in the "other" category

@andygrove
Copy link
Contributor Author

apache/arrow-datafusion#2551 might also belong listed in the "other" category

Thanks @alamb. Strictly speaking, we're not blocked on this because we are only using the SQL query planner and the logical optimization rules. However, implementing the physical plan for OFFSET and adding some integration tests would give me greater confidence that the planning and optimization rules for LIMIT and OFFSET are correct, so I will add it to the list.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
datafusion Related to work in DataFusion
Projects
None yet
Development

No branches or pull requests

3 participants