DataFusion weekly project plan (Andrew Lamb) - Dec 11, 2023

[alamb]

Follow on to Dec 4 2023:

Boilerplate Overview

The idea of this ticket is make my plans for DataFusion visible, largely for my own personal organizational needs, but also to:

1. Try some different ways to communicate / coordinate in the community
2. Help provide an interesting summary of what is happening in DataFusion this week

It would be great if anyone else who has plans like this for DataFusion could try to make them visible somehow as well

My (personal) plans for this week

• Shepherd the PRs through Support general pruning based on <col> = 'const' in PruningPredicate #8376
• Fix Incorrect results due to repartitioning a sorted ParquetExec #8451)
• Draft proposal to move DataFusion to a new top level apache project #8491
• Reproduce other bugs / limitations we have hit while upgrading DataFusion in InfluxDB IOx

Project Queue (list of future projects)

• FIFO JoinHashMap for HashJoin #8130, in support of memory explosion in HashJoin outputs: HashJoinStream memory tracking insufficient #7848
• Epic: Statistics improvements #8227 (specifically, Consolidate statistics aggregation #8229 as a step towards Introduce a way to represent constrained statistics / bounds on values in Statistics #8078)

Projects I hope to help review and coordinate

• Next steps towards function packages more modular (make all BuiltInScalarFunctions ScalarUDFs): [EPIC] Unify Function Interface (remove BuiltInScalarFunction) #8045 with @2010YOUY01 and @edmondop . This week I hope to help get Support Expr creation for ScalarUDF: Resolve function calls by name during planning #8157 complete, and move on to RFC: Demonstrate what a function package might look like -- encoding expressions #8046
• Review any new ticket / bug reports

Algorithm for (my) prioritizing PR reviews

Note there are many committers who can review and merge PRs, so this is not the priorities of the project as a whole, just the approximate algorithm I am using

Priority:

1. Bug fixes (where something is just incorrect), especially regressions (where it used to work and now does not)
2. Improvements directly related to features needed for InfluxDB (my employer)
3. Documentation and test improvements (I view these as very strategically important)
4. PRs that I think are strategically important
5. Other new features / additions to functionality

The current strategically important projects in my head are:

• Making function packages modular and easier to mix/match: [EPIC] Unify Function Interface (remove BuiltInScalarFunction) #8045
• Improving performance of high cardinality aggregation (I have dreams of working on this over christmas break)

Thus, if you are interested in contributing to DataFusion and are interested in a fast turn around time I would recommend looking into bug fixes / test improvements / documentation / etc.

If you propose adding new functionality, the review cycle will likely be longer. You can make it a shorter cycle by looking at the comments on other recent PRs and following the same model (e.g. ensure there are tests in sqllogictest for example, the CI passes, includes documentation, etc)

[alamb]

Review Queue
DataFusion

• fix: make sure CASE WHEN pick first true branch when WHEN clause is true #8477
• Move Coercion for MakeArray to coerce_arguments_for_signature and introduce another one for ArrayAppend #8317
• Sort filenames when reading parquet to ensure consistent schema #6629
• fix: incorrect set preserve_partitioning in SortExec #8485
• Add metrics for UnnestExec #8482
• fix: support uppercase when parsing Interval #8478
• Fix compute_record_batch_statistics wrong with projection #8489

Arrow

• feat: Support quote and escape in Csv WriterBuilder arrow-rs#5196
• Introduce Schema::field_names method arrow-rs#5186

sqlparser-rs:

• add support for CALL statements datafusion-sqlparser-rs#1063
• add support for JSON_TABLE datafusion-sqlparser-rs#1062

[alamb]

DataFusion

• fix: volatile expressions should not be target of common subexpt elimination #8520
• Minor/Doc: Clarify DataFrame::write_table Documentation #8519
• Prepare version 34.0.0 #8508
• Fallback on null empty value in ExprBoundaries::try_from_column #8501
• fix: Pull stats in IdentVisitor/GraphvizVisitor only when requested #8514
• fix: make sure CASE WHEN pick first true branch when WHEN clause is true #8477
• Add metrics for UnnestExec #8482
• Fix DataFrame::cache errors with Plan("Mismatch between schema and batches") #8510
• Fix count(null) and count(distinct null) #8511
• refactor: use ExprBuilder to consume substrait expr and use macro to generate error #8515

Arrow

• Introduce Schema::field_names method arrow-rs#5186

sqlparser-rs:

• add support for CALL statements datafusion-sqlparser-rs#1063
• add support for JSON_TABLE datafusion-sqlparser-rs#1062

[alamb]

Datafusion

• fix: volatile expressions should not be target of common subexpt elimination #8520

[alamb]

Datafusion

• feat: improve string statistics display in datafusion-cli parquet_metadata function #8535
• Remove order_bys from AggregateExec state #8537
• Fix compute_record_batch_statistics wrong with projection #8489
• feat: support scientific notation in parse_float_as_decimal  #8494

[alamb]

Datafusion

• docs: update udf docs for udtf #8546 (review)
• refactor: HashJoinStream state machine #8538
• feat: implement Repartition plan in substrait #8526
• Add support for conflicting order sensitive aggregates in ARRAY_AGG aggregate function #8558
• Move Coercion for MakeArray to coerce_arguments_for_signature and introduce another one for ArrayAppend #8317

[Xuanwo]

The weekly project plan looks great! Are you updating this manually or using a tool?

[alamb]

The weekly project plan looks great! Are you updating this manually or using a tool?

I am creating these manually @Xuanwo -- though now what I do is copy the last weeks's description to the next week, and update the text rather than writing it from scratch each time.

[alamb]

Datafusion

• Convert list array and non-list array to scalars #7862 (review)
• Add support for conflicting order sensitive aggregates in ARRAY_AGG aggregate function #8558
• Move Coercion for MakeArray to coerce_arguments_for_signature and introduce another one for ArrayAppend #8317
• Refactor TreeNode recursions #7942
• Change ScalarValue::{List, LargeList, FixedSizedList} to take specific types rather than ArrayRef #8562
• Add new configuration item listing_table_ignore_subdirectory #8565
• Port tests in parquet.rs to sqllogictest #8560
• Substrait insubquery #8363

Arrow

• Introduce Schema::field_names method arrow-rs#5186
• feat(ord): Support equality of StructArray arrow-rs#5217

sqlparser-rs:

• add support for CALL statements datafusion-sqlparser-rs#1063
• add support for JSON_TABLE datafusion-sqlparser-rs#1062

[alamb]

#8577 for next week