Skip to content

chore: update datafusion to 52.0 #1394

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
killzoner wants to merge 2 commits into
base: main
Choose a base branch
from
Open

chore: update datafusion to 52.0 #1394

killzoner wants to merge 2 commits into from
+651 −631

Conversation

@killzoner
Copy link
Contributor

@killzoner killzoner commented Jan 19, 2026 •
edited
Loading

Which issue does this PR close?

Closes #1357

Rationale for this change

Datafusion upgrade changes: https://datafusion.apache.org/library-user-guide/upgrading.html#datafusion-52-0-0

What changes are included in this PR?

Are there any user-facing changes?

No

@killzoner killzoner force-pushed the update-df-52 branch 2 times, most recently from 2b1423b to 8a7460c Compare January 19, 2026 19:16
killzoner
killzoner commented Jan 19, 2026
View reviewed changes
milenkovicm
milenkovicm reviewed Jan 19, 2026
View reviewed changes
Copy link
Contributor

@milenkovicm milenkovicm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like test failures should be fixable.
Please update ballista versions to 52 as well

Thanks @killzoner

@killzoner
Copy link
Contributor Author

killzoner commented Jan 20, 2026

Looks like test failures should be fixable. Please update ballista versions to 52 as well

Thanks @killzoner

Hey, i will have a look soon at the failing tests. I need to take a look at FetchPartition inner working, seems like the CSV/parquet files written in the tests are valid ones, but there is something missing.

@milenkovicm
Copy link
Contributor

milenkovicm commented Jan 20, 2026

add file name to written csv&jsons at write time. not sure what changed in datafusion

killzoner
killzoner commented Jan 20, 2026
View reviewed changes
killzoner
killzoner commented Jan 20, 2026
View reviewed changes
@killzoner killzoner force-pushed the update-df-52 branch 2 times, most recently from d27d6e7 to ccc4f30 Compare January 20, 2026 12:05
@killzoner killzoner marked this pull request as ready for review January 20, 2026 12:40
@andygrove
Copy link
Member

andygrove commented Jan 20, 2026

@sqlbenchmark run tpch

@sqlbenchmark
Copy link

sqlbenchmark commented Jan 20, 2026

Ballista TPC-H Benchmark Results

PR: #1394 - chore: update ballista crates to 52
PR Commit: 8a63449
Base Commit: 38fca8b (main)
Scale Factor: SF1
Iterations: 3

Query Comparison

Query Main (ms) PR (ms) Change
Q1 743.40 785.50 🔴 +5.7%
Q2 1360.30 1388.80 ⚪ +2.1%
Q3 1080.00 1001.70 🟢 -7.2%
Q4 849.70 845.20 ⚪ -0.5%
Q5 1791.10 1712.80 ⚪ -4.4%
Q6 422.00 388.50 🟢 -7.9%
Q7 1848.70 1803.70 ⚪ -2.4%
Q8 2289.80 2310.80 ⚪ +0.9%
Q9 1860.10 1913.80 ⚪ +2.9%
Q10 1265.50 1400.20 🔴 +10.6%
Q11 1069.10 1083.10 ⚪ +1.3%
Q12 856.70 828.90 ⚪ -3.2%
Q13 957.30 930.50 ⚪ -2.8%
Q14 605.10 627.50 ⚪ +3.7%
Q15 967.50 1000.20 ⚪ +3.4%
Q16 1307.60 1286.80 ⚪ -1.6%
Q17 729.60 722.00 ⚪ -1.0%
Q18 1393.20 1230.40 🟢 -11.7%
Q19 779.00 743.60 ⚪ -4.5%
Q20 988.10 1070.00 🔴 +8.3%
Q21 1746.50 1726.10 ⚪ -1.2%
Q22 898.50 912.90 ⚪ +1.6%

Total: Main=25808.80ms, PR=25713.00ms (-0.4%)

Automated benchmark run by dfbench

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@milenkovicm milenkovicm milenkovicm left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Update to DataFusion v.52

4 participants

@killzoner @milenkovicm @andygrove @sqlbenchmark