Pin pyarrow<24 in the cudf and pylibcudf conda recipes by galipremsagar · Pull Request #23319 · NVIDIA/cudf

galipremsagar · 2026-07-17T16:33:23Z

Description

The latest-deps conda CI jobs started failing on every PR (e.g. this run on #23272) with pyarrow 25.0.0 in the environment — feather tests/doctests (pyarrow.feather deprecated as of 24), pylibcudf quantiles (SortOptions(null_placement=) deprecated in 25), ORC tests (out-of-ns-range timestamps now raise ArrowInvalid instead of silently overflowing), and the narwhals suite.

cudf pins pyarrow>=19.0.0,<24 (#22229) in dependencies.yaml for the conda, requirements, and pyproject outputs — but the hand-maintained conda recipes only declare pyarrow>=19.0.0 with no upper bound (conda/recipes/cudf/recipe.yaml run dependency and conda/recipes/pylibcudf/recipe.yaml run constraint). The conda test environments don't list pyarrow directly (only the oldest-deps matrix pins pyarrow==19.*), so the env solve takes the bound from the built packages, and with the recipes unbounded the solver picked pyarrow 25.0.0. This is also how the narwhals job got pyarrow 25: its env installs the built cudf conda package in the same solve.

This mirrors the dependencies.yaml bound into both recipes, which constrains every conda test environment that installs the built packages.

Note: the wheel jobs were unaffected because the pip/pyproject metadata carries the <24 bound. The remaining failure in the linked run (conda-python-other-tests) was a runner infra flake (nvidia-smi: "No devices were found") — retry only.

For whenever the pin is actually lifted (#22229): the test-suite adaptations needed for pyarrow 24/25 (feather→pyarrow.ipc migration, per-sort-key null_placement, ORC timestamp-range handling, narwhals deselects) were worked out and verified in 9a3a288019 (previous head of this branch).

Checklist

I am familiar with the Contributing Guidelines.
New or existing tests cover these changes.
The documentation is up to date with these changes.

copy-pr-bot · 2026-07-17T16:33:27Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

galipremsagar · 2026-07-17T16:37:11Z

/okay to test 9a3a288

coderabbitai · 2026-07-17T16:38:00Z

📝 Walkthrough

Walkthrough

Changes

PyArrow runtime bounds

Layer / File(s)	Summary
Conda recipe dependency constraints `conda/recipes/cudf/recipe.yaml`, `conda/recipes/pylibcudf/recipe.yaml`	PyArrow runtime requirements now specify `>=19.0.0,<24` in both recipes; the pylibcudf entry also includes a reference comment.

Estimated code review effort: 1 (Trivial) | ~2 minutes

Suggested reviewers: msarahan, wence-, matt711

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title accurately summarizes the recipe change: pinning pyarrow below 24 in cudf and pylibcudf conda recipes.
Description check	✅ Passed	The description is related to the recipe update and explains why the pyarrow upper bound was added.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands.}

coderabbitai

Actionable comments posted: 1

🧹 Nitpick comments (1)

python/cudf/cudf/io/feather.py (1)
22-26: 🚀 Performance & Scalability | 🔵 Trivial | 💤 Low value

Consider early column projection via IpcReadOptions instead of post-read selection.

read_feather reads all columns then filters with pa_table.select(columns). Using pa.ipc.IpcReadOptions(included_columns=columns) passed to pa.ipc.open_file would avoid reading unnecessary columns, improving I/O efficiency for files with many columns.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@python/cudf/cudf/io/feather.py` around lines 22 - 26, The read_feather flow
should project requested columns during IPC reading instead of loading the full
table and calling pa_table.select(columns) afterward. Create IpcReadOptions with
included_columns=columns when columns is provided, pass it to pa.ipc.open_file,
and preserve full-column reads when columns is None; remove the post-read
selection.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@python/cudf/cudf/io/feather.py`:
- Around line 29-44: Restore compatibility between DataFrame.to_feather and the
feather.to_feather wrapper by updating the to_feather signature to accept and
forward *args/**kwargs, or update the DataFrame.to_feather wrapper consistently
after verifying its signature. Preserve support for existing extra arguments
such as compression, and add the required deprecation warning if narrowing the
public API is intentional.

---

Nitpick comments:
In `@python/cudf/cudf/io/feather.py`:
- Around line 22-26: The read_feather flow should project requested columns
during IPC reading instead of loading the full table and calling
pa_table.select(columns) afterward. Create IpcReadOptions with
included_columns=columns when columns is provided, pass it to pa.ipc.open_file,
and preserve full-column reads when columns is None; remove the post-read
selection.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 4b7c89a3-363d-4a23-9956-bacc0b814cb5

📥 Commits

Reviewing files that changed from the base of the PR and between 7b13484 and 9a3a288.

📒 Files selected for processing (5)

ci/test_narwhals.sh
python/cudf/cudf/io/feather.py
python/cudf/cudf/tests/input_output/test_feather.py
python/cudf/cudf/tests/input_output/test_orc.py
python/pylibcudf/tests/test_quantiles.py

coderabbitai · 2026-07-17T16:38:03Z

 @ioutils.doc_to_feather()
-def to_feather(df, path, *args, **kwargs):
+def to_feather(df, path):
    """{docstring}"""
    warnings.warn(
        "Using CPU via PyArrow to write Feather dataset, this may "
        "be GPU accelerated in the future"
    )
    # Feather doesn't support using an index
    pa_table = df.to_arrow(preserve_index=False)
-    feather.write_feather(pa_table, path, *args, **kwargs)
+    # Feather v2 is the Arrow IPC file format; pyarrow.feather is
+    # deprecated as of pyarrow 24. Keep write_feather's default lz4
+    # compression.
+    options = pa.ipc.IpcWriteOptions(compression="lz4")
+    with pa.OSFile(str(path), "wb") as sink:
+        with pa.ipc.new_file(sink, pa_table.schema, options=options) as writer:
+            writer.write_table(pa_table)


📐 Maintainability & Code Quality | 🟠 Major | 🏗️ Heavy lift

to_feather signature narrowing breaks DataFrame.to_feather wrapper.

The new to_feather(df, path) no longer accepts *args/**kwargs, but DataFrame.to_feather (in python/cudf/cudf/core/dataframe.py) still forwards them: feather.to_feather(self, path, *args, **kwargs). Any user passing extra arguments (e.g., compression) will now get a TypeError instead of the previous behavior. As per coding guidelines, API breaking changes to public methods should include deprecation warnings.

Please verify whether DataFrame.to_feather needs updating:

#!/bin/bash # Check the DataFrame.to_feather wrapper signature rg -n -A5 'def to_feather' python/cudf/cudf/core/dataframe.py

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@python/cudf/cudf/io/feather.py` around lines 29 - 44, Restore compatibility between DataFrame.to_feather and the feather.to_feather wrapper by updating the to_feather signature to accept and forward *args/**kwargs, or update the DataFrame.to_feather wrapper consistently after verifying its signature. Preserve support for existing extra arguments such as compression, and add the required deprecation warning if narrowing the public API is intentional.

Source: Coding guidelines

dependencies.yaml pins pyarrow>=19.0.0,<24 for the conda, requirements, and pyproject outputs, but the hand-maintained conda recipes only declare pyarrow>=19.0.0 with no upper bound: cudf's run dependency and pylibcudf's run constraint. The conda test environments don't list pyarrow directly (only the oldest-deps matrix pins pyarrow==19.*), so the solver takes the bound from the built packages and the latest-deps CI jobs picked up pyarrow 25.0.0, which fails the feather, ORC, and pylibcudf quantiles tests as well as the narwhals suite. Mirror the dependencies.yaml bound into both recipes.

galipremsagar · 2026-07-17T17:01:58Z

/okay to test 4c05dcc

galipremsagar · 2026-07-17T17:15:44Z

/merge

The latest-deps conda CI jobs started failing on every PR (e.g. [this run on NVIDIA#23272](https://github.com/rapidsai/cudf/actions/runs/29585048860)) with pyarrow 25.0.0 in the environment — feather tests/doctests (`pyarrow.feather` deprecated as of 24), pylibcudf quantiles (`SortOptions(null_placement=)` deprecated in 25), ORC tests (out-of-ns-range timestamps now raise `ArrowInvalid` instead of silently overflowing), and the narwhals suite. cudf pins `pyarrow>=19.0.0,<24` (NVIDIA#22229) in `dependencies.yaml` for the conda, requirements, and pyproject outputs — but the hand-maintained conda recipes only declare `pyarrow>=19.0.0` with **no upper bound** (`conda/recipes/cudf/recipe.yaml` run dependency and `conda/recipes/pylibcudf/recipe.yaml` run constraint). The conda test environments don't list pyarrow directly (only the oldest-deps matrix pins `pyarrow==19.*`), so the env solve takes the bound from the built packages, and with the recipes unbounded the solver picked pyarrow 25.0.0. This is also how the narwhals job got pyarrow 25: its env installs the built cudf conda package in the same solve. This mirrors the `dependencies.yaml` bound into both recipes, which constrains every conda test environment that installs the built packages. Note: the wheel jobs were unaffected because the pip/pyproject metadata carries the `<24` bound. The remaining failure in the linked run (`conda-python-other-tests`) was a runner infra flake (`nvidia-smi`: "No devices were found") — retry only. For whenever the pin is actually lifted (NVIDIA#22229): the test-suite adaptations needed for pyarrow 24/25 (feather→`pyarrow.ipc` migration, per-sort-key `null_placement`, ORC timestamp-range handling, narwhals deselects) were worked out and verified in [9a3a288](galipremsagar@9a3a288019) (previous head of this branch). Authors: - GALI PREM SAGAR (https://github.com/galipremsagar) Approvers: - Vyas Ramasubramani (https://github.com/vyasr) URL: NVIDIA#23319

galipremsagar requested review from a team as code owners July 17, 2026 16:33

galipremsagar requested review from Matt711, msarahan and wence- July 17, 2026 16:33

github-actions Bot assigned galipremsagar Jul 17, 2026

galipremsagar added bug Something isn't working non-breaking Non-breaking change Python Affects Python cuDF API. ci labels Jul 17, 2026

github-project-automation Bot added this to cuDF Python Jul 17, 2026

github-actions Bot added the pylibcudf Issues specific to the pylibcudf package label Jul 17, 2026

GPUtester moved this to In Progress in cuDF Python Jul 17, 2026

coderabbitai Bot reviewed Jul 17, 2026

View reviewed changes

galipremsagar force-pushed the fix-pyarrow-25-compat branch from 9a3a288 to 4c05dcc Compare July 17, 2026 16:55

galipremsagar requested a review from a team as a code owner July 17, 2026 16:55

galipremsagar added the conda label Jul 17, 2026

galipremsagar changed the title ~~Fix test failures with pyarrow 25~~ Pin pyarrow<24 in the cudf and pylibcudf conda recipes Jul 17, 2026

vyasr approved these changes Jul 17, 2026

View reviewed changes

galipremsagar added the 5 - Ready to Merge Testing and reviews complete, ready to merge label Jul 17, 2026

rapids-bot Bot merged commit d7a32b1 into NVIDIA:release/26.08 Jul 17, 2026
216 of 227 checks passed

github-project-automation Bot moved this from In Progress to Done in cuDF Python Jul 17, 2026

josephine-wolf-oberholtzer added this to cuDF Python Aug 12, 2026

josephine-wolf-oberholtzer moved this to Done in cuDF Python Aug 12, 2026

Navigation Menu

Uh oh!

Pin pyarrow<24 in the cudf and pylibcudf conda recipes - #23319

Pin pyarrow<24 in the cudf and pylibcudf conda recipes#23319
rapids-bot[bot] merged 1 commit into
NVIDIA:release/26.08from
galipremsagar:fix-pyarrow-25-compat

galipremsagar commented Jul 17, 2026 •

edited

Loading

Uh oh!

copy-pr-bot Bot commented Jul 17, 2026

Uh oh!

galipremsagar commented Jul 17, 2026

Uh oh!

coderabbitai Bot commented Jul 17, 2026 •

edited

Loading

Walkthrough

Changes

Uh oh!

coderabbitai Bot left a comment

Uh oh!

coderabbitai Bot Jul 17, 2026

Uh oh!

galipremsagar commented Jul 17, 2026

Uh oh!

galipremsagar commented Jul 17, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

galipremsagar commented Jul 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Checklist

Uh oh!

copy-pr-bot Bot commented Jul 17, 2026

Uh oh!

galipremsagar commented Jul 17, 2026

Uh oh!

coderabbitai Bot commented Jul 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

coderabbitai Bot Jul 17, 2026

Choose a reason for hiding this comment

Uh oh!

galipremsagar commented Jul 17, 2026

Uh oh!

galipremsagar commented Jul 17, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

galipremsagar commented Jul 17, 2026 •

edited

Loading

coderabbitai Bot commented Jul 17, 2026 •

edited

Loading