Skip to content

Support out-of-band buffers in Python pickling#375

Merged
kkraus14 merged 7 commits intorapidsai:branch-0.15from
jakirkham:use_reduce_DevBuf
May 21, 2020
Merged

Support out-of-band buffers in Python pickling#375
kkraus14 merged 7 commits intorapidsai:branch-0.15from
jakirkham:use_reduce_DevBuf

Conversation

@jakirkham
Copy link
Member

@jakirkham jakirkham commented May 21, 2020

When Python pickle's protocol 5 or greater is used, this change will support more efficient serialization of out-of-band buffers. This is analogous to Dask's custom serialization except for pickling. As such this is helpful in any Python serialization case where pickling is used. If an older pickling protocol is used, we simply proceed as before.

Also as we have switched from __getstate__/__setstate__ to __reduce_ex__, we no longer need to build a DeviceBuffer and then steal its unique_ptr for our own DeviceBuffer (which pickle also constructs). Instead we can just build a DeviceBuffer once and return it. So this is a bit more efficient.

Note this is analogous to the change made to Buffer ( rapidsai/cudf#5132 ) except for DeviceBuffer.

@jakirkham jakirkham requested a review from a team as a code owner May 21, 2020 05:18
@jakirkham
Copy link
Member Author

cc @kkraus14 @shwina

@kkraus14 kkraus14 added 5 - Ready to Merge Testing and reviews complete, ready to merge Python Related to RMM Python API labels May 21, 2020
@kkraus14 kkraus14 merged commit aeac458 into rapidsai:branch-0.15 May 21, 2020
@jakirkham jakirkham deleted the use_reduce_DevBuf branch May 21, 2020 08:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

5 - Ready to Merge Testing and reviews complete, ready to merge Python Related to RMM Python API

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants