Add support for the Arrow device capsule interfaces by vyasr · Pull Request #15370 · NVIDIA/cudf

vyasr · 2024-03-21T22:30:18Z

Description

This PR adds support for using Arrow's device capsule interface to pylibcudf Column and Table objects. This will allow two-way zero-copy interchange of Arrow data with other libraries that know how to consume data conforming to the Arrow specification.

I use the Python nanoarrow package for testing since it is the only package I know of that supports two-way conversions (consumption and production) of Arrow device capsules at the moment.

Closes #14959

Checklist

I am familiar with the Contributing Guidelines.
New or existing tests cover these changes.
The documentation is up to date with these changes.

copy-pr-bot · 2024-03-21T22:30:22Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

vyasr · 2024-03-21T22:34:33Z

Everything outside the python directory is a change from #15047 and can be ignored.

review-notebook-app · 2025-04-10T23:23:09Z

Check out this pull request on

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

vyasr · 2025-04-10T23:25:57Z

/ok to test

vyasr · 2025-04-16T18:30:42Z

@zeroshade you may want to take a look at this if you have use cases to try it out.

zeroshade · 2025-04-18T15:54:18Z

@vyasr I'm currently travelling for conferences, but I'll definitely take a look here when I get a chance.

…device_capsule

Matt711

Just non blocking suggestions/questions

Matt711 · 2025-04-21T17:04:12Z

 - pydata-sphinx-theme>=0.15.4
 - pynvml>=12.0.0,<13.0.0a0
- pytest-benchmark<5.1.0
+- pytest-benchmark


Just curious why?

See the changes in dependencies.yaml (I deleted a bunch of relevant comments).

Matt711 · 2025-04-21T17:04:21Z


        return self._to_schema(), self._to_host_array()

+    def __arrow_c_device_array__(self, requested_schema=None, **kwargs):


Nearly identical to the method for table. Maybe factor out? Same for _to_device_array

Yeah, we could consider consolidating more. I'm not sure what the best way to do that is right now. Maybe we need some more common helper functions for this. It gets a bit tricky with the Cython typing though.

vyasr · 2025-04-21T18:19:30Z

/merge

This reverts commit 167762b.

bdice

Packaging approval.

vyasr · 2025-04-22T01:28:48Z

/merge

vyasr · 2025-04-24T19:13:47Z

@zeroshade this PR is merged, but of course feel free to comment here or open issues/PRs if you see any problems or have any suggestions for follow-up!

vyasr added feature request New feature or request non-breaking Non-breaking change labels Mar 21, 2024

vyasr self-assigned this Mar 21, 2024

github-actions Bot added libcudf Affects libcudf (C++/CUDA) code. Python Affects Python cuDF API. CMake CMake build issue labels Mar 21, 2024

vyasr force-pushed the feat/arrow_device_capsule branch from 995e88f to e6affc9 Compare April 8, 2024 23:21

vyasr mentioned this pull request Apr 9, 2024

Add to_arrow_device() functions that accept views #15465

Merged

3 tasks

trivialfis mentioned this pull request Dec 5, 2024

[FEA] A Python wrapper for to_arrow_device. #17528

Closed

vyasr changed the base branch from branch-24.06 to branch-25.02 December 11, 2024 00:13

vyasr mentioned this pull request Jan 29, 2025

Avoid converting Decimal32/Decimal64 in to_arrow and from_arrow APIs #17422

Merged

3 tasks

trivialfis mentioned this pull request Mar 5, 2025

Store categories from cuDF. dmlc/xgboost#11311

Merged

vyasr force-pushed the feat/arrow_device_capsule branch from e6affc9 to daf817d Compare April 10, 2025 23:23

vyasr changed the base branch from branch-25.02 to branch-25.06 April 10, 2025 23:23

github-actions Bot added the pylibcudf Issues specific to the pylibcudf package label Apr 10, 2025

vyasr changed the title ~~Add Python interface for to_arrow_device~~ Add support for the Arrow device capsule interfaces Apr 10, 2025

vyasr mentioned this pull request Apr 10, 2025

[FEA] Introduce a new owning type for Arrow interop data #16104

Closed

vyasr added 5 commits April 16, 2025 00:59

First version of device capsule support

c4b80f1

Properly handle lifetime management in the array rather than the capsule

911d3b7

Add initial test of just schema using nanoarrow

7fd9863

Enable ingestion of device array data in tables

a8bf16c

Add capsule support to columns

ff21354

vyasr requested review from KyleFromNVIDIA, bdice and galipremsagar April 16, 2025 01:02

vyasr added 2 commits April 16, 2025 01:09

Remove invalid TODOs

f05f075

pylibcudf test dependencies must be installed for testing

cf50a87

vyasr added 2 commits April 16, 2025 19:15

No conda-forge arm packages for nanoarrow

167762b

Remove pytest-benchmark bound now that the conda-forge fixes are merged

1cd438d

mroeschke reviewed Apr 18, 2025

View reviewed changes

Comment thread python/pylibcudf/pylibcudf/table.pyx

mroeschke reviewed Apr 18, 2025

View reviewed changes

Comment thread python/pylibcudf/pylibcudf/table.pyx

galipremsagar approved these changes Apr 18, 2025

View reviewed changes

Matt711 self-requested a review April 18, 2025 18:21

vyasr added 2 commits April 18, 2025 21:28

Merge remote-tracking branch 'upstream/branch-25.06' into feat/arrow_…

736da98

…device_capsule

PR review

f015e2a

vyasr requested a review from mroeschke April 18, 2025 21:31

mroeschke approved these changes Apr 18, 2025

View reviewed changes

Matt711 approved these changes Apr 21, 2025

View reviewed changes

Matt711 reviewed Apr 21, 2025

View reviewed changes

Comment thread dependencies.yaml

bdice requested changes Apr 21, 2025

View reviewed changes

Comment thread dependencies.yaml Outdated

vyasr added 2 commits April 21, 2025 18:27

Revert "No conda-forge arm packages for nanoarrow"

22c8a4c

This reverts commit 167762b.

Merge branch 'branch-25.06' into feat/arrow_device_capsule

ab46669

bdice approved these changes Apr 21, 2025

View reviewed changes

Add missed group

b57124f

rapids-bot Bot merged commit a978c51 into NVIDIA:branch-25.06 Apr 22, 2025

vyasr deleted the feat/arrow_device_capsule branch April 24, 2025 19:13

vyasr mentioned this pull request Apr 25, 2025

Implement __dlpack__ dunder for pylibcudf columns #18566

Closed


		return self._to_schema(), self._to_host_array()

		def __arrow_c_device_array__(self, requested_schema=None, **kwargs):

Uh oh!

Conversation

vyasr commented Mar 21, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Checklist

Uh oh!

copy-pr-bot Bot commented Mar 21, 2024

Uh oh!

vyasr commented Mar 21, 2024

Uh oh!

review-notebook-app Bot commented Apr 10, 2025

Uh oh!

vyasr commented Apr 10, 2025

Uh oh!

vyasr commented Apr 16, 2025

Uh oh!

zeroshade commented Apr 18, 2025

Uh oh!

Uh oh!

Uh oh!

Matt711 left a comment

Choose a reason for hiding this comment

Uh oh!

Matt711 Apr 21, 2025

Choose a reason for hiding this comment

Uh oh!

vyasr Apr 21, 2025

Choose a reason for hiding this comment

Uh oh!

Matt711 Apr 21, 2025

Choose a reason for hiding this comment

Uh oh!

vyasr Apr 21, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vyasr commented Apr 21, 2025

Uh oh!

Uh oh!

bdice left a comment

Choose a reason for hiding this comment

Uh oh!

vyasr commented Apr 22, 2025

Uh oh!

vyasr commented Apr 24, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

vyasr commented Mar 21, 2024 •

edited

Loading