DataFusion weekly project plan (Andrew Lamb) - Jan 8, 2024

[alamb]

Follow on to Jan 1, 2024

Boilerplate Overview

The idea of this ticket is make my plans for DataFusion visible, largely for my own personal organizational needs, but also to:

1. Try some different ways to communicate / coordinate in the community
2. Help provide an interesting summary of what is happening in DataFusion this week

It would be great if anyone else who has plans like this for DataFusion could try to make them visible somehow as well 🙏 (feel free to copy / modify the format)

My (personal) plans for this week

• Work on more advanced pruning predicates Don't error on unknown column when pruning if predicate can still be proven false #7869 (comment)
• Work on a blog post about DataFusion: [Website]: DataFusion 26-34 blog post arrow-site#457
• Prepare next steps for making function packages more modular [EPIC] Unify Function Interface (remove BuiltInScalarFunction) #8045

Project Queue (list of future projects)

• Statistics: Epic: Statistics improvements #8227 (specifically, Introduce a way to represent constrained statistics / bounds on values in Statistics #8078)

Projects I plan to prioritize reviewing / helping

• Memory explosion in HashJoin outputs: HashJoinStream memory tracking insufficient #7848 from @korowa
• Improve planning speed [Epic] A collection of issues to improve planning performance / speed / efficiency #5637 with @matthewmturner
• Making function packages more modular: [EPIC] Unify Function Interface (remove BuiltInScalarFunction) #8045 (looking for help)

Algorithm for (my) prioritizing PR reviews

Note there are many committers who can and do review and merge PRs, so this is not the priorities of the project as a whole, just the approximate algorithm I am using

Priority:

1. Bug fixes (where something is just incorrect), especially regressions (where it used to work and now does not)
2. Improvements directly related to features needed for InfluxDB (my employer)
3. Documentation and test improvements (I view these as very strategically important)
4. PRs that I think are strategically important
5. Other new features / additions to functionality

The current strategically important projects in my head are:

• Making function packages modular and easier to mix/match: [EPIC] Unify Function Interface (remove BuiltInScalarFunction) #8045
• Getting started guide for new users (who want to use DataFusion in their project)
• Improving performance of aggregation - especially high cardinality groupings, where we still lag state of the art by 2x

Thus, if you are interested in contributing to DataFusion and are interested in a fast turn around time I would recommend looking into bug fixes / test improvements / documentation / etc.

If you propose adding new functionality, the review cycle will likely be longer. You can make it a shorter cycle by looking at the comments on other recent PRs and following the same model (e.g. ensure there are tests in sqllogictest for example, the CI passes, includes documentation, etc)

[alamb]

DataFusion

• fix: struct field don't push down to TableScan #8774
• fix: failed to create ValuesExec with non-nullable schema #8776
• Upgrade to object_store 0.9.0 and arrow 50.0.0 #8758
• Minor: Fix error messages in array expressions #8781
• feat: Add bloom filter metric to ParquetExec #8772 (review)
• Move tests from expr.rs to sqllogictests. Part1 #8773
• Refactor size estimation of Hashset into a function #8779
• Implement trait based API for define AggregateUDF #8733

[matthewmturner]

@alamb FYI I am away for a few days and then will get back into planning performance work.

Let me know if there's any area in particular you would like to tackle next.

[matthewmturner]

I had in mind checking for more places to replace results with boolean and then seeing if I could extract some of the optimization ideas from @zeodotr on #7698. Or are you keen to make the DFSchema cheaper to copy?

[alamb]

I had in mind checking for more places to replace results with boolean and then seeing if I could extract so

I think that is the best idea. I would personally suggest focusing on the planning benchmarks and trying to improve whatever they show as the tall pole

[matthewgapp]

@alamb just bumping the recursive CTE PR :) #7581

I fixed an issue that was raised and brought it up to date with main.

[alamb]

DataFusion

• Serialize function signature simplifications #8802
• feat: Add bloom filter metric to ParquetExec #8772
• Cleanup the usage of round-robin repartitioning #8794
• Implement monotonicity for ScalarUDF #8799
• Standardize CompressionTypeVariant encoding in protobuf #8785
• Enhance simplifier by adding Canonicalize #8780
• feat: support array_resize #8744
• Add support for recursive CTEs #7581

[alamb]

@alamb just bumping the recursive CTE PR :) #7581

I fixed an issue that was raised and brought it up to date with main.

Thank you @matthewgapp -- I keep hoping someone else can help with that review too that has an interest in recursive CTEs. I'll try and take a look later today

[alamb]

DataFusion:

• Cleanup the usage of round-robin repartitioning #8794
• [Minor] extract const and add doc and more tests for in_list pruning #8815
• Minor: chores: Update clippy in pre-commit.sh #8810 (review)
• refactor: standardize exec_from funcs arg order #8809 (review)
• [MINOR]: Add size check for aggregate #8813 (review)
• Serialize function signature simplifications #8802 (comment)
• Port tests in timestamp.rs to sqllogictest. Part 1 #8818

[maruschin]

@alamb hi, сould you suggest an area where to direct my attention. I spend a lot of time looking for issue to start working on.

[alamb]

DataFusion

• Update ClickBench benchmarks with DataFusion 34 #8789 (comment)
• docs: document SessionConfig #8771
• [CI] Refactor CI builders  #8826 (comment)
• Fix ApproxPercentileCont signature #8825
• Simplify physical expression creation API (not require schema) #8823
• feat: support array_resize #8744

[alamb]

@alamb hi, сould you suggest an area where to direct my attention. I spend a lot of time looking for issue to start working on.

Thanks @maruschin !

If you have time, looking into the bug report here #8819 would be super helpful

Debugging this report #8702 would also be helpful (as would in general making the datafusion-cli output generation code simpler)

There is also supporting partitioned writing that seems like it would be great to implement: #8493

[alamb]

Review Queue:

DataFusion

• Port tests in group_by.rs to sqllogictest #8834
• fix: don't extract common sub expr in CASE WHEN clause #8833
• feat: add more components to the wasm-pack compatible list #8843
• Determine causal window frames to produce early results. #8842
• Support for order sensitive NTH_VALUE aggregation, make reverse ARRAY_AGG more efficient #8841
• Add hash_join_single_partition_threshold_rows config #8720
• Remove Transformed enum #8835
• Enhance simplifier by adding Canonicalize #8780
• Change ScalarValue::Struct to ArrayRef #7893
• Add ForeignKey constraint type #8566
• feat: support array_resize #8744

[alamb]

Done in #8864