DataFusion weekly project plan (Andrew Lamb) - Jan 15, 2024

[alamb]

Follow on to Jan 8, 2024

Boilerplate Introduction

The idea of this ticket is make my plans for DataFusion visible, largely for my own personal organizational needs, but also to:

1. Communicate / coordinate in the community
2. Help provide an interesting summary of what is happening in DataFusion this week

It would be great for other contributors to DataFusion who plan non trivial work could try to make them visible somehow as well 🙏 (feel free to copy / modify the format)

My (personal) plans for this week

• Complete blog post about DataFusion: [Website]: DataFusion 26-34 blog post arrow-site#457
• Complete some performance work (Improve aggregate performance with specialized groups accumulator for single string group by #7064) + benchmarking (Add 'clickbench_extended' benchmark #8860 and Add test coverage for grouping on dictionary encoded columns #8791
• (Stretch) Prepare next steps for making function packages more modular [EPIC] Unify Function Interface (remove BuiltInScalarFunction) #8045

Project Queue (list of future projects)

• More advanced pruning predicates Don't error on unknown column when pruning if predicate can still be proven false #7869
• Statistics: Epic: Statistics improvements #8227 (specifically, Introduce a way to represent constrained statistics / bounds on values in Statistics #8078)

Projects I plan to prioritize reviewing / helping

• Memory explosion in HashJoin outputs: HashJoinStream memory tracking insufficient #7848 from @korowa
• Improve planning speed [Epic] A collection of issues to improve planning performance / speed / efficiency #5637 with @matthewmturner
• Making function packages more modular: [EPIC] Unify Function Interface (remove BuiltInScalarFunction) #8045 (thank you @guojidan)
• Supporting recursive CTEs Add support for recursive CTEs #462 with @matthewgapp and @jonahgao

Algorithm for (my) prioritizing PR reviews

Note there are many committers who can and do review and merge PRs, so this is not the priorities of the project as a whole, just the approximate algorithm I use to prioritize my own time

Priority:

1. Bug fixes (where something is just incorrect), especially regressions (where it used to work and now does not)
2. Improvements directly related to features needed for InfluxDB (my employer)
3. Documentation and test improvements (I view these as very strategically important)
4. PRs that I think are strategically important
5. Other new features / additions to functionality (note this is the lowest)

The current strategically important projects in my head are:

• Making function packages modular and easier to mix/match: [EPIC] Unify Function Interface (remove BuiltInScalarFunction) #8045
• Getting started guide for new users (who want to use DataFusion in their project)
• Improving performance of aggregation - especially high cardinality groupings, where we still lag state of the art by 2x

Thus, if you are interested in contributing to DataFusion and are interested in a fast turn around time I would recommend looking into bug fixes / test improvements / documentation or the projects named above

If you propose adding new functionality, the review cycle will likely be longer. You can make it a shorter cycle by looking at the comments on other recent PRs and following the same model (e.g. ensure there are tests in sqllogictest for example, the CI passes, includes documentation, etc)

[alamb]

Review Queue:

DataFusion

• Minor: Update with_column_rename method doc #8858
• test: Port order.rs tests to sqllogictest #8857 (review)
• fix bug with to_timestamp and InitCap logical serialization, add roundtrip test between expression and proto,  #8868
• Disable Parallel Parquet Writer by Default, Improve Writing Test Coverage #8854
• Add options of parquet bloom filter and page index in Session config #8869 (review)
• Find the correct fields when using page filter on struct fields in parquet #8848
• Change cli to propagate error to exit code #8856
• docs: fix wrong pushdown name & a typo #8875
• feat: emitting partial join results in HashJoinStream #8020
• Determine causal window frames to produce early results. #8842

Sqlparser

• Add support for PostgreSQL Insert table aliases (#1069) datafusion-sqlparser-rs#1084 (review)
• Add "timezone_region" and "timezone_abbr" support for EXTRACT (and DATE_PART) datafusion-sqlparser-rs#1090
• Add support for PostgreSQL ^@ starts-with operator datafusion-sqlparser-rs#1091
• Add support for JSONB datatype datafusion-sqlparser-rs#1089

[alamb]

DataFusion:

• Deprecate make_scalar_function #8878
• Change cli to propagate error to exit code #8856
• Minor: add tests for casts between nested List and LargeList #8882
• refactor: rename FileStream.file_reader to file_opener & update doc #8883
• Disable Parallel Parquet Writer by Default, Improve Writing Test Coverage #8854

[alamb]

DataFusion:

• Fix "Projection references non-aggregate values" by updating rebase_expr to use transform_down #8890
• Deprecate make_scalar_function #8878
• Support array literal with scalar function #8884
• docs: fix wrong name in sub-crates' README #8889

[alamb]

DataFusion

• docs: add an example for RecordBatchReceiverStreamBuilder #8888
• Support GroupsAccumulator accumulator for udaf #8892
• support to_timestamp with optional chrono formats #8886
• Add serde support for Arrow FileTypeWriterOptions #8850
• Recursive CTEs: Stage 2 - add support for sql -> logical plan generation #8839
• feat:implement sql style 'ends_with' and 'instr' string function #8862
• Consolidate TreeNode transform and rewrite APIs #8891
• Add support for FixedSizeList type in arrow_cast, hashing #8344

Sqlparser

• Allow string values in pragma commands datafusion-sqlparser-rs#1101
• Replace AtomicUsize with Cell<usize> in the recursion counter datafusion-sqlparser-rs#1098

[alamb]

DataFusion

• feat: support stride in array_slice, change indexes to be1 based #8829
• aggregate_statistics should only optimize MIN/MAX when relation is not empty #8914
• Feat/make dfschema wrap schemaref #8905
• Add ForeignKey constraint type #8566
• Add hash_join_single_partition_threshold_rows config #8720
• Fix expr partial ord test #8908
• Change ScalarValue::Struct to ArrayRef #7893
• Remove Transformed enum #8835

Sqlparser

• Add APIs to reuse token buffers in Tokenizer datafusion-sqlparser-rs#1094
• ISSUE-1088: Fix array_agg wildcard behavior datafusion-sqlparser-rs#1093
• Add Postgres operators for the LIKE expression variants datafusion-sqlparser-rs#1096
• Add support for constraint characteristics clause datafusion-sqlparser-rs#1099
• Add support for BigQuery table and view options datafusion-sqlparser-rs#1061

Watching

• Enhance simplifier by adding Canonicalize #8780

[alamb]

Next week: #8933