Docs: Update roadmap to point at EPIC's, clarify project goals #6639
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -19,100 +19,27 @@ under the License. | |
|
|
||
| # Roadmap | ||
|
|
||
|
There was a problem hiding this comment. These goals are largely redundant with the introduction, so I figured it would be better to leave a link and direct people back there rather than partially replicate the content |
||
| The [project introduction](../user-guide/introduction) explains the | ||
| overview and goals of DataFusion, and our development efforts largely | ||
| align to that vision. | ||
|
|
||
| ## Planning `EPIC`s | ||
|
|
||
| DataFusion uses [GitHub | ||
|
There was a problem hiding this comment. I began this PR by trying to summarize the outstanding work and to do so I looked at the While a more free form version of the roadmap in text (rather than a github issue list) is probably easier to consume, unless we have a volunteer to commit to doing, keeping our efforts focused on keeping github updated seemed better. |
||
| issues](https://github.com/apache/arrow-datafusion/issues) to track | ||
| planned work. We collect related tickets using tracking issues labeled | ||
| with `[EPIC]` which contain discussion and links to more detailed items. | ||
|
|
||
| Epics offer a high level roadmap of what the DataFusion | ||
| community is thinking about. The epics are not meant to restrict | ||
| possibilities, but rather help the community see where development is | ||
| headed, align our work, and inspire additional contributions. | ||
|
|
||
| As this project is entirely driven by volunteers, we welcome | ||
| contributions for items not currently covered by epics. However, | ||
| before submitting a large PR, we strongly suggest and request you | ||
| start a conversation using a github issue or the | ||
| [[email protected]](mailto:[email protected]) mailing list to | ||
| make review efficient and avoid surprises. | ||
|
|
||
| [The current list of `EPIC`s can be found here](https://github.com/apache/arrow-datafusion/issues?q=is%3Aissue+is%3Aopen+epic). | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
|
|
@@ -22,8 +22,20 @@ | |||||
| DataFusion is a very fast, extensible query engine for building | ||||||
| high-quality data-centric systems in [Rust](http://rustlang.org), | ||||||
| using the [Apache Arrow](https://arrow.apache.org) in-memory format. | ||||||
| DataFusion is part of the [Apache Arrow](https://arrow.apache.org/) | ||||||
| project. | ||||||
|
|
||||||
| DataFusion offers SQL and Dataframe APIs, excellent [performance](https://benchmark.clickhouse.com/), built-in support for CSV, Parquet, JSON, and Avro, [python bindings], extensive customization, a great community, and more. | ||||||
|
|
||||||
| [python bindings]: https://github.com/apache/arrow-datafusion-python | ||||||
|
|
||||||
| ## Project Goals | ||||||
|
|
||||||
| DataFusion aims to be the query engine of choice for new, fast | ||||||
| data centric systems such as databases, dataframe libraries, machine | ||||||
| learning and streaming applications by leveraging the unique features | ||||||
| of [Rust](https://www.rust-lang.org/) and [Apache | ||||||
| Arrow](https://arrow.apache.org/). | ||||||
|
|
||||||
| ## Features | ||||||
|
|
||||||
|
|
@@ -34,37 +46,47 @@ DataFusion offers SQL and Dataframe APIs, excellent [performance](https://benchm | |||||
| - Many extension points: user defined scalar/aggregate/window functions, DataSources, SQL, | ||||||
| other query languages, custom plan and execution nodes, optimizer passes, and more. | ||||||
| - Streaming, asynchronous IO directly from popular object stores, including AWS S3, | ||||||
| Azure Blob Storage, and Google Cloud Storage (Other storage systems are supported via the | ||||||
| `ObjectStore` trait). | ||||||
| - [Excellent Documentation](https://docs.rs/datafusion/latest) and a | ||||||
| [welcoming community](https://arrow.apache.org/datafusion/contributor-guide/communication.html). | ||||||
| - A state of the art query optimizer with expression coercion and | ||||||
| simplification, projection and filter pushdown, sort and distribution | ||||||
| aware optimizations, automatic join reordering, and more. | ||||||
| - Permissive Apache 2.0 License, predictable and well understood | ||||||
| [Apache Software Foundation](https://www.apache.org/) governance. | ||||||
| - Implementation in [Rust](https://www.rust-lang.org/), a modern | ||||||
| system language with development productivity similar to Java or | ||||||
| Golang, the performance of C++, and [loved by programmers | ||||||
| everywhere](https://insights.stackoverflow.com/survey/2021#technology-most-loved-dreaded-and-wanted). | ||||||
| - Support for [Substrait](https://substrait.io/) query plans, to | ||||||
| easily pass plans across language and system boundaries. | ||||||
|
|
||||||
| ## Use Cases | ||||||
|
|
||||||
| DataFusion can be used without modification as an embedded SQL | ||||||
| engine or can be customized and used as a foundation for | ||||||
| building new systems. | ||||||
|
|
||||||
| While most current usecases are "analytic" or (throughput) some | ||||||
|
There was a problem hiding this comment. This is trying to channel @avantgardnerio 's suggestion on #6441 (comment) though I am not sure how faithfully I have done so There was a problem hiding this comment. I'm not sure I could say it any better. There was a problem hiding this comment. I don't think you need the There was a problem hiding this comment.
Suggested change
Nice catch -- 🦅 👁️ |
||||||
| components of DataFusion such as the plan representations, are | ||||||
| suitable for "streaming" and "transaction" style systems (low | ||||||
| latency). | ||||||
|
|
||||||
| Here are some example systems built using DataFusion: | ||||||
|
|
||||||
| - Specialized Analytical Database systems such as [CeresDB] and more general Apache Spark like system such a [Ballista]. | ||||||
| - New query language engines such as [prql-query] and accelerators such as [VegaFusion] | ||||||
| - Research platform for new Database Systems, such as [Flock] | ||||||
| - SQL support to another library, such as [dask sql] | ||||||
| - Streaming data platforms such as [Synnada] | ||||||
| - Tools for reading / sorting / transcoding Parquet, CSV, AVRO, and JSON files such as [qv] | ||||||
| - Native Spark runtime replacement such as [Blaze] | ||||||
|
|
||||||
| By using DataFusion, projects are freed to focus on their specific | ||||||
| features, and avoid reimplementing general (but still necessary) | ||||||
| features such as an expression representation, standard optimizations, | ||||||
| parellelized streaming execution plans, file format support, etc. | ||||||
|
|
||||||
| ## Known Users | ||||||
|
|
||||||
|
|
@@ -119,7 +141,7 @@ Here are some of the projects known to use DataFusion: | |||||
| ## Integrations and Extensions | ||||||
|
|
||||||
| There are a number of community projects that extend DataFusion or | ||||||
| provide integrations with other systems, some of which are described below: | ||||||
|
|
||||||
| ### Language Bindings | ||||||
|
|
||||||
|
|
@@ -137,5 +159,5 @@ provide integrations with other systems. | |||||
|
|
||||||
| - _High Performance_: Leveraging Rust and Arrow's memory model, DataFusion is very fast. | ||||||
| - _Easy to Connect_: Being part of the Apache Arrow ecosystem (Arrow, Parquet and Flight), DataFusion works well with the rest of the big data ecosystem | ||||||
| - _Easy to Embed_: Allowing extension at almost any point in its design, and published regularly as a crate on [crates.io](http://crates.io), DataFusion can be integrated and tailored for your specific usecase. | ||||||
| - _High Quality_: Extensively tested, both by itself and with the rest of the Arrow ecosystem, DataFusion can and is used as the foundation for production systems. | ||||||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is trying to address @boazberman 's comments in #6441 (comment)