Add to_arrow method to pylibcudf core types

[Matt711]

Description

Addsto_arrow methods to pylibcudf core types. Not a priority but a quality of life improvement for pylibcudf developers. Eg. Call .to_arrow instead of plc.interop.to_arrow

Checklist

• I am familiar with the Contributing Guidelines.
• New or existing tests cover these changes.
• The documentation is up to date with these changes.

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[mroeschke]

Curious, is there an immediate use case for this API?

[Matt711]

Curious, is there an immediate use case for this API?

This is really just a quality of life improvement for pylibcudf devs. Easier to type to_arrow vs plc.interop.to_arrow

[mroeschke]

OK I would be open to having to_arrow methods on our objects (to mirror to_py for example) if we eventually also depreciate and remove plc.interop.to_arrow as well. Could use a second opinion cc @vyasr

[vyasr]

I would prefer not to have two ways to do the arrow conversion. The argument for keeping the interop module approach is that I would also like to isolate our pyarrow dependency as much as possible so that we could completely avoid the pyarrow dependency except when a user strictly requests it. I'm not very convinced of the quality-of-life improvement of this PR TBH. Are you finding the current approach cumbersome? My hope is that we move away from converting to pyarrow representations as much as possible actually. Outside of our test suite, my view is that we view the arrow data spec as the mode of interop so that any library that directly supports it will "just work", but we shouldn't be converting device->host outside of debugging too often. Would it help to lazily alias to_arrow into the top-level pylibcudf namespace (e.g. with a module-level __getattr__ implementation)? I say lazily so that we still don't import pyarrow if we don't have to.

[Matt711]

I would prefer not to have two ways to do the arrow conversion.

Sure, but we have two ways of ingesting arrow objects Table.from_arrow and plc.interop.from_arrow. If we prefer to have one way, maybe it should hold for both directions?

I'm not very convinced of the quality-of-life improvement of this PR TBH. Are you finding the current approach cumbersome?

Heh, maybe I'm in the minority here 😅, but I do find the current approach cumbersome.

Outside of our test suite, my view is that we view the arrow data spec as the mode of interop so that any library that directly supports it will "just work", but we shouldn't be converting device->host outside of debugging too often.

OK yeah that makes sense.

Would it help to lazily alias to_arrow into the top-level pylibcudf namespace (e.g. with a module-level __getattr__ implementation)? I say lazily so that we still don't import pyarrow if we don't have to.

I don't think so. I like this style

(
    plc.Column.from_arrow(...)
    .do_stuff()
    .to_arrow()
)

[vyasr]

Matt and I discussed earlier today and we decided to move forward with this PR and make the class methods the source of truth and eventually deprecate the interop functions instead. We can still minimize and isolate our pyarrow interactions, it'll just have to be spread over a few more files to do it.

[vyasr]

Looks fine to me aside from a couple of tiny notes. Could you open a follow-up issue to migrate all calls in cudf-polars and cudf classic to use the new APIs, and then to deprecate the old interop module's functions?

[vyasr]

What is this comment referring to? I don't understand I'm afraid.

[Matt711]

There's a typo that should be pa.table

[Matt711]

I mean is that once arrow can consume more kinds of device arrays, we can call pa.table(self) directly. The following example fails today

In [1]: import pylibcudf as plc

In [2]: import pyarrow as pa

In [3]: pa.table(plc.Table([plc.Column.from_iterable_of_py([1,2])]))
---------------------------------------------------------------------------
ArrowKeyError                             Traceback (most recent call last)
Cell In[3], line 1
----> 1 pa.table(plc.Table([plc.Column.from_iterable_of_py([1,2])]))

File ~/.conda/envs/rapids/lib/python3.13/site-packages/pyarrow/table.pxi:6235, in pyarrow.lib.table()

File ~/.conda/envs/rapids/lib/python3.13/site-packages/pyarrow/table.pxi:6054, in pyarrow.lib.record_batch()

File ~/.conda/envs/rapids/lib/python3.13/site-packages/pyarrow/table.pxi:4044, in pyarrow.lib.RecordBatch._import_from_c_device_capsule()

File ~/.conda/envs/rapids/lib/python3.13/site-packages/pyarrow/error.pxi:155, in pyarrow.lib.pyarrow_internal_check_status()

File ~/.conda/envs/rapids/lib/python3.13/site-packages/pyarrow/error.pxi:92, in pyarrow.lib.check_status()

ArrowKeyError: Device type 2is not registered

[vyasr]

Oh I see. Yeah... I'm not sure that'll ever be supported, but if it is then great. The best bet might be backing pyarrow with pylibcudf for GPU bits!

[Matt711]

/merge