Release DataFusion 42.0.0

[alamb]

Is your feature request related to a problem or challenge?

Tracking ticket for next release, also a place to track desired inclusions

Last release was https://crates.io/crates/datafusion/41.0.0 Aug 11, 2024 so next major release would be around September 10, 2024

List if items that would be good to get into this release:

• Update to parquet / arrow 53.0.0: Release arrow-rs / parquet major version 53.0.0 (September 2024) arrow-rs#6016

Additional context

• Release DataFusion 41.0.0 #11476
• Release DataFusion 40.0.0 #11077

Items to fix before release

• Incorrectly losing OFFSETs for LIMIT queries #12423
• Calling with_column twice generates an error when the second call uses a window function #12425

[alamb]

FYI this release is being planned for this week: https://lists.apache.org/thread/jgqktymcw8xtshth0lvr3317s77lft16

[alamb]

We (InfluxData) found two issues while upgrading to a pre-release version of DataFusion that are not yet filed. We are working to open tickets for them (after confirming they are indeed issues upstream)

[alamb]

In terms of using StringView by default, #11682, I hope to turn that on very soon after 42 is released (so it gets maximum bake time on main before we release version 43)

[alamb]

Here is one issue (a regression) that I think we should fix before release:

• Incorrectly losing OFFSETs for LIMIT queries #12423

[timsaucer]

Another regression to fix before the release if possible: #12425

[alamb]

Another regression to fix before the release if possible: #12425

I added a section to the ticket's description for "items to fix"

[andygrove]

@alamb I plan on starting the release prep today unless there are other issues?

[alamb]

@alamb I plan on starting the release prep today unless there are other issues?

The only other thing I think we might want to consider is #12414 that @wiedld and @berkaysynnada are working on.

I have just filed #12446 to track it. However, since the regression has existed since 40.0.0 I don't think it should block the release of 42.0.0

[alamb]

@andygrove there are a few PRs I was holding off merging until you do the release cut

• Upgrade sqlparser-rs to 0.51.0, support new interval logic from sqlparse-rs #12222
• use FileFormat::get_ext as the default file extension filter #12417

Do you have any estimate on when you might make the release candidate?

[andygrove]

I was waiting to see if #12446 would be resolved, but I can go ahead and cut the release candidate today

[alamb]

Release candidate thread: https://lists.apache.org/thread/bdnfy1skj5vs3s4nx44kycdd6lb917b2

I also filed #12470 to track releasing the next version 43.0.0

[andygrove]

Updates:

• The source release has been published to https://dist.apache.org/repos/dist/release/datafusion/datafusion-42.0.0/
• Crates have been published

I have some post-release tasks to do:

• Update release instructions with new crates and changes in crate publishing order
• Add README files to new crates
• Update download page to point to latest release