Discussion: "Goals of DataFusion" / "Vision" for DataFusion

[alamb]

For a variety of reasons, I have been thinking about how to articulate the "Vision" for DataFusion.

While the current website does a reasonable job explaining what DataFusion is https://arrow.apache.org/datafusion/, I want to frame the longer term discussion and wanted to hear what the community thinks.

[alamb]

Here are some thoughts:

Vision

In 5 years time: DataFusion underpins several of the most commonly used analytics systems in the world and is widely recognized as an exemplar of a high performance, vectorized execution engine created by a powerful and broad open source community. In addition to being widely used by itself, it has spawned several similar projects in other languages, affirming the trend of a deconstructed database across the industry.

What is DataFusion good for?

Building new, fast data centric systems (databases, dataframe libraries, streaming systems, etc) in Rust

** Examples:
Databases:
Dataframe libraries
Streaming Systems:

Who is Datafusion targeted at?

Developers (e.g. creators who who make analytic systems)

[thinkharderdev]

Agree with everything @alamb said but want to add I'd love to see is a high-performance runtime tuned to database workloads or the ability to use datafusion with a custom runtime. The work that @tustvold has done on the Scheduler is a great start but it would be great to be able to use custom runtimes that can leverage io_uring/DPDK/kernel-bypass networking for really bleeding edge performance. I hope to work on this (just as soon as I get the time :)).

[avantgardnerio]

I'd love it if it was also the query engine for a mutable OLTP/OLAP database in Rust. Of course this is an external concern to the core project, but having things like index-awareness and mutability in mind will allow for that to happen.

[liurenjie1024]

For datafusion, I think index-awareness and mutability would be more about optimizer?

[alamb]

🤔 it sounds like it might be helpful to organize a "current roadmap" type issue / discussion. Maybe we should try a new discussion for that topic 🤔

[liurenjie1024]

I would say datafusion is more like velox, e.g. single host runtime for olap engine. But it also dataframe api/optimizer for easy use.

[alamb]

I agree DataFusion has the single host fast runtime like Velox -- the primary differences in my mind are:

1. DataFusion is written in Rust
2. DataFusion also has a query optimizer and SQL engine, so it can be used to build systems without bringing in some other large dependency (like Spark or Trino) if you don't already have one

Aka you can embed DataFusion easily into new Rust based analytic systems

[MrPowers]

Cool question! Some thoughts:

• Fastest single node DataFrame library
• De facto solution for querying CSVs / Parquet files on the command line
• The query parser for other SQL libraries like dask-sql and ray-sql

[alamb]

Thanks @MrPowers -- I like them

Fastest single node DataFrame library

This is basically the goal of pola.rs. Given pola.rs is tightly integrated and focused on the DataFrame usecase I think it would take a significant investment by the DataFusion community to make this happen. It would be great if they did.

It is true that there is significant overlap in pola.rs's functionality and DataFusion's ExecutionPlan implementations, e.g. sorting, grouping, joining, etc. In my ideal universe we would figure out how to combine efforts and share the development / maintenance of these components, but I think pola.rs may be too tightly integrated at this point for this unified dream to become a reality (we haven't even been able to combine forces on an Arrow implementation yet, though I think we are agreed in principle it would be a good idea)

cc @ritchie46 in case you are interested

[boazberman]

I think a large value Datafusion can bring is being the disaggregated query processing engine. If it can be more around enabling developers to mix and extend any part of it they need, that could bring the most value. The parts as I see them: Sql parser, logical plan, physical plan, logical optimizer, physical optimizer, storage layer.

If, as discussed here, Datafusion could be used to build a transactional database, that would be awesome.

[alamb]

I agree and I really like your breakdown of the parser/logical plan/ etc -- I think that helps a lot.

As for transactional database, that would also be neat - I don't personally have a lot of experience building such a thing -- maybe some of those people who are building something similar on top of DataFusion could offer some thoughts

[avantgardnerio]

build a transactional database

@alamb and I happened to be talking today, and I think we arrived at several concrete examples of a theme:

• For OLTP, the query optimizer performance matters a lot more, and databases may require options to tweak it for performance
• In databases with inline MVCC data, the selection vectors discussed elsewhere might help performance significantly
• for databases with multiple indexes, the ability for TableProviders to provide potential sort orders to the optimizer could be important

I'm sure there are many more - I don't think I could compile a comprehensive list. I think my earlier request was more along the lines of "don't forget about mutable / OLTP!" in general.

[raulcd]

@alamb I was wondering whether the in Rust comment is required. I suppose that arrow-datafusion-python would also be part of the new Apache DataFusion, so shouldn't the scope be broader and maybe we could drop the in Rust from Building new, fast data centric systems (databases, dataframe libraries, streaming systems, etc) in Rust

edit: added some formatting

[westonpace]

If "in Rust" means "Datafusion is a Rust project" then I think it is ok. If "in Rust" means "datafusion is a project to be used by Rust developers" then I agree with @raulcd that this is unnecessarily limiting, espeically giving the ability for fast in-process exchange of Arrow data over the C data interface. For example, it would be interesting to explore datafusion as a "substrait planner" to be used by any substrait project, regardless of whether it is in Rust or not.

[alamb]

The reason I was thinking about highlighting "Rust" is that it is one of the things that I think differentiates DataFusion from other similar technologies such as Velox, Acero, etc in the C/C++ world, and Calcite, Avatica in the java world

The fact that DataFusion is "just" a Rust crate and can be used by adding 1 line to a Cargo.toml file I think is valuable

However, I realize that calling out the Rust implementation language may be artificially limiting on its scope 🤔

[westonpace]

I agree that the fact that Datafusion is implemented in Rust is a huge selling point 👍 and deserves to go on the byline. I just think there is a risk that non-rust people with either feel that "they can't use datafusion because its a rust project" or "they won't understand the powerful safety guarantees that 'built in rust' is communicating".

That being said, I don't know how much we need to nitpick the wording :)

[andygrove]

I think it is reasonable to say that DataFusion is implemented in Rust, and also has Python bindings available.

[alamb]

Here is an attempt to update the documentation to reflect the comments on this discussion: #6639