docs(rfc): DataFusion / Arrow upgrade, phased behind upstream (RFC 0021, drafted) by jensholdgaard · Pull Request #337 · jensholdgaard/ourios

jensholdgaard · 2026-07-02T23:09:56Z

What

RFC 0021 (drafted) — the #314 epic's RFC: the coordinated DataFusion /
Arrow upgrade, phased behind what upstream has actually shipped.

Phase 1 (now): DataFusion 53.1 → 54.0. DF 54 pins arrow/parquet ^58.3 —
the same arrow the querier already pulls — so ourios-parquet moves off
arrow 55 and the workspace unifies on one arrow, deleting the RFC 0017
dual decoder + schema_force_view_types workaround (RFC 0017 row path: revisit arrow-58 decoder duplication + view-types workaround (feature-flag the alternative?) #276). MSRV 1.85 → 1.88
(DF 54's floor).
Phase 2 (upstream-gated, expected ~DF 55): object_store ≥ 0.14 (chore(deps): update cargo (minor/patch) #310;
renovate hold chore(renovate): hold object_store at 0.13.x until DataFusion upgrades #313) + parquet 59, which drops thrift entirely (deps: remediate thrift advisory (GHSA-2f9f-gq7v-9h6m) via arrow/parquet/DataFusion upgrade #295).
quick-xml deny-ignores removal iff the object_store release pins
quick-xml ≥ 0.41.

Why phased (§2.4 lock table): no released DataFusion accepts
object_store 0.14 or parquet 59 — DF 54.0.0 pins object_store ^0.13.2 and
parquet ^58.3.0 (thrift ^0.17 still present). The epic's single coordinated
bump is not constructible today; what is constructible (arrow unification)
is the riskiest slice — the on-disk-format pillar — and doing it in isolation
means the property/corpus/reconstruction suites validate exactly one change.

Invariants pinned (§3.1)

§3.5: no schema bytes change; any arrow-58 behaviour that would alter
written bytes is pinned back or explicitly RFC'd — never absorbed silently.
A committed pre-upgrade Parquet fixture makes "old files still read" a
permanent regression test (RFC0021.2).
§3.3: reconstruction property/corpus suites run unchanged.
CAS untouched in phase 1 (object_store doesn't move); re-proven by the
localstack suites in phase 2 (RFC0021.7).
B1/B2 pruning gates re-measured (indicative ci-runner; authoritative
baseline on opt-in) — RFC0021.5.

§5

Nine scenarios: .1–.6 phase 1, .7–.9 phase 2 (stubs land red only
when upstream opens the phase).

Process

Opened at drafted per the RFC ladder; on your sign-off I flip to
specified and merge (RFC sign-off is the merge gate), then take phase 1 to
red with the §5 stubs.

🤖 Generated with Claude Code

Summary by CodeRabbit

Documentation
- Added a new RFC entry to the project’s documentation index.
- Introduced a detailed RFC covering a phased dependency upgrade plan, including expected milestones, acceptance criteria, and testing guidance.

coderabbitai · 2026-07-02T23:10:05Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 480707f8-560f-4cd7-8306-f0b0e43f0eb5

📥 Commits

Reviewing files that changed from the base of the PR and between 552d350 and 68b4c9a.

📒 Files selected for processing (2)

docs/SUMMARY.md
docs/rfcs/0021-datafusion-arrow-upgrade.md

📝 Walkthrough

Walkthrough

This PR adds a new RFC document (docs/rfcs/0021-datafusion-arrow-upgrade.md) describing a two-phase DataFusion/Arrow/Parquet/object_store upgrade plan, and adds a corresponding link entry in docs/SUMMARY.md.

Changes

RFC 0021 documentation

Layer / File(s)	Summary
RFC index entry `docs/SUMMARY.md`	Adds a link to RFC 0021 in the RFC list.
RFC 0021 content `docs/rfcs/0021-datafusion-arrow-upgrade.md`	Adds front-matter, motivation, phased design (Phase 1 DataFusion 54/Arrow 58, Phase 2 object_store/Parquet/thrift removal), alternatives, acceptance criteria (RFC0021.1–9), testing strategy, open questions, and references.

Estimated code review effort: 1 (Trivial) | ~5 minutes

Possibly related issues

epic: upgrade DataFusion + Arrow (unblocks object_store 0.14, thrift advisory, arrow-58 decoder dup) #314: The RFC directly documents the DataFusion/Arrow upgrade epic tracked by this issue.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly names the RFC and the phased DataFusion/Arrow upgrade, matching the main change.
Description check	✅ Passed	The description is detailed and covers the RFC, related epic, rationale, and rollout plan, though it doesn't use the template headings exactly.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch rfc0021-datafusion-arrow-upgrade

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

Copilot

Pull request overview

Adds RFC 0021 (drafted) documenting a phased plan to upgrade the workspace’s DataFusion/Arrow stack behind what upstream has shipped, with Phase 1 focusing on unifying Arrow versions (via DF 54 / arrow+parquet 58.3) and Phase 2 gated on upstream releases (object_store ≥0.14 + parquet 59).

Changes:

Add drafted RFC 0021 describing the phased DataFusion/Arrow upgrade plan and acceptance scenarios.
Register RFC 0021 in the docs table of contents (mdBook SUMMARY).

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File	Description
docs/SUMMARY.md	Adds RFC 0021 to the RFC index.
docs/rfcs/0021-datafusion-arrow-upgrade.md	Introduces the drafted RFC describing the phased DataFusion/Arrow upgrade plan, invariants, and acceptance criteria.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

…1, drafted) The #314 epic's RFC. Two phases, each following what upstream has shipped: - Phase 1 (now): DataFusion 53.1 → 54.0 unifies the workspace on one arrow (58.3; ourios-parquet moves off 55.2), removing the RFC 0017 dual decoder + view-types workaround (#276). MSRV 1.85 → 1.88 (DataFusion 54's floor). - Phase 2 (upstream-gated, expected ~DataFusion 55): object_store ≥ 0.14 (#310, renovate hold #313) + parquet 59 (drops thrift → #295); quick-xml deny-ignores removal iff object_store pins quick-xml ≥ 0.41. The boundary is upstream reality, not preference: DataFusion 54 pins object_store ^0.13.2 and parquet ^58.3 (thrift ^0.17 still present), so the epic's single coordinated bump is not constructible from released crates. Phase 1 deliberately isolates the riskiest part — arrow 55 → 58 on the on-disk-format pillar — with §3.3/§3.5 pinned as invariants (no schema bytes change; a committed pre-upgrade Parquet fixture makes "old files still read" a permanent regression test) and B1/B2 re-measured. §5 has 9 scenarios (.1–.6 phase 1; .7–.9 upstream-gated phase 2). Alternatives (single DF-55 bump, object_store fork, parquet-59-first) rejected in §4. Verified: mdbook build renders; SUMMARY.md entry added. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

jensholdgaard · 2026-07-02T23:23:02Z

@coderabbitai review

coderabbitai · 2026-07-02T23:23:09Z

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

Maintainer ruling on Copilot's review finding: the RFC ladder places §5 acceptance-criteria authorship at the `specified` stage, so a document carrying a full §5/§6 carries `specified`, not `drafted` (matching the RFC 0020 precedent, which merged at `specified`). Status flipped; no content change. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

jensholdgaard requested a review from Copilot July 2, 2026 23:09

Copilot started reviewing on behalf of jensholdgaard July 2, 2026 23:10 View session

jensholdgaard mentioned this pull request Jul 2, 2026

epic: upgrade DataFusion + Arrow (unblocks object_store 0.14, thrift advisory, arrow-58 decoder dup) #314

Open

Copilot AI reviewed Jul 2, 2026

View reviewed changes

jensholdgaard requested a review from Copilot July 2, 2026 23:12

jensholdgaard force-pushed the rfc0021-datafusion-arrow-upgrade branch from 19c216e to fa479f8 Compare July 2, 2026 23:12

Copilot started reviewing on behalf of jensholdgaard July 2, 2026 23:12 View session

Copilot AI reviewed Jul 2, 2026

View reviewed changes

Comment thread docs/rfcs/0021-datafusion-arrow-upgrade.md

jensholdgaard requested a review from Copilot July 2, 2026 23:28

Copilot started reviewing on behalf of jensholdgaard July 2, 2026 23:28 View session

Copilot AI reviewed Jul 2, 2026

View reviewed changes

jensholdgaard merged commit f769d5d into main Jul 2, 2026
23 checks passed

jensholdgaard deleted the rfc0021-datafusion-arrow-upgrade branch July 2, 2026 23:37

This was referenced Jul 3, 2026

feat(deps)!: upgrade to datafusion 54 / arrow 58 — one arrow across the workspace #339

Merged

test(parquet): rfc 0021 phase 1 green — live lockfile gate + discharged markers #341

Merged

Navigation Menu

docs(rfc): DataFusion / Arrow upgrade, phased behind upstream (RFC 0021, drafted) - #337

docs(rfc): DataFusion / Arrow upgrade, phased behind upstream (RFC 0021, drafted)#337
jensholdgaard merged 2 commits into
mainfrom
rfc0021-datafusion-arrow-upgrade

jensholdgaard commented Jul 2, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

coderabbitai Bot commented Jul 2, 2026 •

edited

Loading

Walkthrough

Changes

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

jensholdgaard commented Jul 2, 2026

Uh oh!

coderabbitai Bot commented Jul 2, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

jensholdgaard commented Jul 2, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Invariants pinned (§3.1)

§5

Process

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

jensholdgaard commented Jul 2, 2026

Uh oh!

coderabbitai Bot commented Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jensholdgaard commented Jul 2, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Jul 2, 2026 •

edited

Loading

coderabbitai Bot commented Jul 2, 2026 •

edited

Loading