[WEBSITE] DataFusion 16.0.0 blog post #294

alamb · 2023-01-07T16:05:32Z

This blog post highlights some improvements and features in DataFusion the last 3 releases 😅

Rendered: https://arrow.apache.org/blog/2023/01/19/datafusion-16.0.0/

github-actions · 2023-01-07T16:05:46Z

Thanks for opening a pull request!

Could you open an issue for this pull request on JIRA?
https://issues.apache.org/jira/browse/ARROW

Then could you also rename pull request title in the following format?

ARROW-${JIRA_ID}: [${COMPONENT}] ${SUMMARY}

See also:

alamb · 2023-01-07T16:06:01Z

It is a work in progress, but I think it is no coherent enough to gather some community input

alamb

It would also be great to add a section to this document about planned feature work

alamb · 2023-01-07T16:07:58Z

_posts/2023-01-07-datafusion-16.0.0.md

+
+## Community Growth
+
+The three months since [our last update](https://arrow.apache.org/blog/2022/10/25/datafusion-13.0.0/) again saw significant growth in the DataFusion.


It would be great if someone could help clean this section up and clearly explain the growth of the community; There is a wonderful story there to tell

alamb · 2023-01-07T16:08:26Z

_posts/2023-01-07-datafusion-16.0.0.md

+closing the gap quickly. Performance highlights from the last three
+months:
+
+* XX% Faster Sorting and Merging using the new [Row Format](https://arrow.apache.org/blog/2022/11/07/multi-column-sorts-in-arrow-rust-part-1/)


@tustvold do you have any suggstions about what numbers to use here?

alamb · 2023-01-07T16:08:44Z

_posts/2023-01-07-datafusion-16.0.0.md

+* Basic filter selectivity analysis (#3868)
+
+
+In the coming few months, we plan work on:


Feedback from the rest of the community would be great

_posts/2023-01-07-datafusion-16.0.0.md

alamb · 2023-01-07T16:09:51Z

_posts/2023-01-07-datafusion-16.0.0.md

+- Implement current_date scalar function (#4022)
+- Compressed CSV/JSON support (#3642)
+
+The community has also been investing in sqllogic based tests to help keep DataFusion's quality high with less work (TODO add some more detail / lnks)


@xudong963 I wonder if you have any thoughts on how to word this better

alamb · 2023-01-07T16:10:04Z

_posts/2023-01-07-datafusion-16.0.0.md

+
+# Substrait
+
+TODO motivating introduction of substrait and why this is interesting


@andygrove perhaps you can help with content for the substrait area

Substrait isn't going to make it into 16, and maybe this should be a separate post? I started a google doc https://docs.google.com/document/d/1vK0AyDBhIibmKZ2scGN3jBypBvqMPuztbBzZ1eh0dKM/edit?usp=sharing

Well, maybe there is a chance it makes it in. Assuming the vote passes tomorrow, we could get the first PR merged 🤔

alamb · 2023-01-07T16:11:23Z

_posts/2023-01-07-datafusion-16.0.0.md

+DataFusion has basic python bindings which has the potential to expand datafusion  to more end users a major missing piece are the python bindings
+
+
+# python bindings and growing the community and ecosystem


I tried to work in a mention of the python bindings and encouraging a champion for them to step forward, however it felt more like it should be a separate post 🤔 -- @andygrove what do you think about a post describing the python bindings, why they are cool, and trying to find people to help drive that project?

I can't figure out how to work this in so I think we should write another post -- maybe on a different site

I took the content / notes I had and put them in a google doc: https://docs.google.com/document/d/1zNfK8pIOqgHURX2lHK0JhSKTaCH3t23tYDKpGLWdFRY/edit

Thanks. I agree. I can work on the Python post

_posts/2023-01-07-datafusion-16.0.0.md

Co-authored-by: Andy Grove <[email protected]>

ozankabak · 2023-01-08T20:05:34Z

_posts/2023-01-07-datafusion-16.0.0.md

+
+Growth of new systems based on as the engine in [many open source and commercial projects](https://github.com/apache/arrow-datafusion#known-uses) and was one of the early open source projects to provide this capability.
+
+Several new databases built on datafusion (synnada.ai, greptimedb, probably others)


Here is what I am aware of:

Databases: greptimedb (new), IOx (GA)
Data platform: Synnada (new)
Use case: Backend for PRQL (relatively new?)

Thanks -- added in ffe2e0a. Still needs polish

andygrove · 2023-01-10T16:41:18Z

I will start contributing to this tomorrow

alamb · 2023-01-10T21:42:50Z

Ok I think this one is now ready for some more review -- it is plausibly ready to publish

_posts/2023-01-07-datafusion-16.0.0.md

ozankabak

Read from start to finish. Apart from the small preposition comment I left inline, it looks great. Looking forward to seeing this published.

…date_16

…ite into alamb/datafusion_update_16

alamb · 2023-01-17T21:20:23Z

I plan to merge this tomorrow unless there are any other comments

[WEBSITE] Datafsuion 16.0.0 blog post

84073d3

alamb changed the title ~~[WEBSITE] DataFusion 16.0.0 blog post~~ [WEBSITE] DataFusion 16.0.0 blog post (WIP) Jan 7, 2023

alamb mentioned this pull request Jan 7, 2023

Blog post about datafusion 16 release apache/datafusion#4804

Closed

alamb commented Jan 7, 2023

View reviewed changes

andygrove reviewed Jan 7, 2023

View reviewed changes

_posts/2023-01-07-datafusion-16.0.0.md Outdated Show resolved Hide resolved

Update _posts/2023-01-07-datafusion-16.0.0.md

00b6ad6

Co-authored-by: Andy Grove <[email protected]>

ozankabak reviewed Jan 8, 2023

View reviewed changes

alamb added 2 commits January 10, 2023 14:49

Add note about streaming

4641b93

work on comunity growth page

ffe2e0a

alamb added 2 commits January 10, 2023 22:30

Updates

ee51391

update more

e4c330f

alamb changed the title ~~[WEBSITE] DataFusion 16.0.0 blog post (WIP)~~ [WEBSITE] DataFusion 16.0.0 blog post Jan 10, 2023

ozankabak reviewed Jan 10, 2023

View reviewed changes

_posts/2023-01-07-datafusion-16.0.0.md Outdated Show resolved Hide resolved

ozankabak reviewed Jan 10, 2023

View reviewed changes

alamb added 6 commits January 10, 2023 17:53

Update _posts/2023-01-07-datafusion-16.0.0.md

b5ca1c7

Merge remote-tracking branch 'origin/master' into alamb/datafusion_up…

76ff403

…date_16

Update publishing date

539b735

Update statistics

3223c81

copy edit

1b55bc5

Merge branch 'alamb/datafusion_update_16' of github.com:alamb/arrow-s…

4e1750f

…ite into alamb/datafusion_update_16

Update date

ad5427a

alamb merged commit 768be07 into apache:master Jan 19, 2023

alamb deleted the alamb/datafusion_update_16 branch January 19, 2023 12:38

alamb mentioned this pull request Mar 31, 2023

Blog post with DataFusion Jan - April 2023 apache/datafusion#5812

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WEBSITE] DataFusion 16.0.0 blog post #294

[WEBSITE] DataFusion 16.0.0 blog post #294

alamb commented Jan 7, 2023 •

edited

Loading

github-actions bot commented Jan 7, 2023

alamb commented Jan 7, 2023 •

edited

Loading

alamb left a comment

alamb Jan 7, 2023

alamb Jan 7, 2023

alamb Jan 7, 2023

alamb Jan 7, 2023

alamb Jan 7, 2023

andygrove Jan 11, 2023

andygrove Jan 11, 2023

alamb Jan 7, 2023

alamb Jan 10, 2023

andygrove Jan 11, 2023

ozankabak Jan 8, 2023

alamb Jan 10, 2023

andygrove commented Jan 10, 2023

alamb commented Jan 10, 2023

ozankabak left a comment •

edited

Loading

alamb commented Jan 17, 2023


		## Community Growth

		The three months since [our last update](https://arrow.apache.org/blog/2022/10/25/datafusion-13.0.0/) again saw significant growth in the DataFusion.

		* Basic filter selectivity analysis (#3868)


		In the coming few months, we plan work on:


		# Substrait

		TODO motivating introduction of substrait and why this is interesting

		DataFusion has basic python bindings which has the potential to expand datafusion to more end users a major missing piece are the python bindings


		# python bindings and growing the community and ecosystem


		Growth of new systems based on as the engine in [many open source and commercial projects](https://github.com/apache/arrow-datafusion#known-uses) and was one of the early open source projects to provide this capability.

		Several new databases built on datafusion (synnada.ai, greptimedb, probably others)

[WEBSITE] DataFusion 16.0.0 blog post #294

[WEBSITE] DataFusion 16.0.0 blog post #294

Conversation

alamb commented Jan 7, 2023 • edited Loading

github-actions bot commented Jan 7, 2023

alamb commented Jan 7, 2023 • edited Loading

alamb left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andygrove commented Jan 10, 2023

alamb commented Jan 10, 2023

ozankabak left a comment • edited Loading

Choose a reason for hiding this comment

alamb commented Jan 17, 2023

alamb commented Jan 7, 2023 •

edited

Loading

alamb commented Jan 7, 2023 •

edited

Loading

ozankabak left a comment •

edited

Loading