Add OrderScheme.get_boundaries API - #1039
Merged
Merged
Conversation
rjzamora
added
improvement
This was referenced May 15, 2026
wence-
reviewed
May 15, 2026
3 tasks
wence-
reviewed
May 21, 2026
Comment on lines
+189
to
+197
| cdef const cpp_TableChunk* chunk = self._handle.boundaries.get() | ||
| cdef Stream stream = Stream._from_cudaStream_t(chunk.stream().value()) | ||
| tbl = Table.from_table_view_of_arbitrary( | ||
| chunk.table_view(), owner=self, stream=stream | ||
| ) | ||
| return TableChunk.from_pylibcudf_table( | ||
| tbl, stream, exclusive_view=False, br=br | ||
| ) | ||
|
|
Contributor
There was a problem hiding this comment.
As I am looking at this, did the OrderScheme need to keep the BufferResource corresponding to the TableChunk we created it from alive, I think yes?
Member
Author
There was a problem hiding this comment.
Isn't the buffer resource attached to the context? The context should outlive this metadata I think. Am I misunderstanding the question?
Member
There was a problem hiding this comment.
You are right, I think @wence- was asking what happens if users do not live up to that contract.
I am okay with this PR as-is, since the current contract is that the BufferResource outlives this metadata.
That said, I think this highlights a broader design issue we should address separately. We still do not have a clean ownership/lifetime story around BufferResource in Python.
I think it is time for me to start working on #641 :)
3 tasks
wence-
approved these changes
May 22, 2026
| return self._handle.boundaries.get().shape().first | ||
|
|
||
| def get_boundaries(self, BufferResource br not None) -> TableChunk: | ||
| """Return the boundary rows as a zero-copy TableChunk view.""" |
Contributor
There was a problem hiding this comment.
Please to docstring correctly.
Member
Author
|
/merge |
rapids-bot Bot
pushed a commit
to NVIDIA/cudf
that referenced
this pull request
Jul 15, 2026
- Part of #22128 - ~Depends on rapidsai/rapidsmpf#1039 - Uses `SparseAlltoAll` to enforce a new `Ordering` for a channel that is already partitioned with a compatible `Ordering` - The utility will typically be used to adjust boundaries or convert from non-strict to strict boundaries. This utility is intended for operators that can exploit existing ordered/range-partitioned input, but require a different concrete boundary layout before they can do so safely. For example, a downstream operator may need to adjust from one strict `Ordering` to another with different boundaries, or convert metadata that is ordered but non-strict into strict output partitions before using chunkwise execution. The immediate motivation is ordered join/groupby/sort planning in cudf-polars: once a stream is known to be ordered, we want to repartition only the boundary-overlap regions needed to align with the target operator, rather than falling back to a full shuffle or sort. **NOTE**: "Strict" partitioning means that a unique value may only exist in one chunk. It is possible for the data to be ordered without "strict" partitioning, but we need to enforce strictness before doing a sort-based join or groupby. ### Simple example: aligning ordered join inputs Suppose two input streams are both ordered on the join keys, but their partition boundaries do not line up. A chunkwise join can only be used safely when corresponding output partitions cover the same key ranges. `adjust_ordering` provides the data-movement primitive for reshaping one ordered stream to match the other stream's strict boundaries, moving only the boundary-overlap pieces that need to change ranks. This lets a downstream join operate partition-by-partition without requiring a full hash shuffle or global sort. The same primitive can also be used by future groupby/sort optimizations that need to turn ordered-but-misaligned or non-strict partitioning into "strict" operator-ready partitions. Authors: - Richard (Rick) Zamora (https://github.com/rjzamora) Approvers: - Tom Augspurger (https://github.com/TomAugspurger) URL: #22628
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
While implementing a prototype to use
OrderSchemeto sort a table in cudf_polars, I realized we need a Python method to extract the boundaries table.Note: I decided it was better to returntuple[Table, Slice]thanTableChunk, because this data is not uniquely owned, and we don't really have a python API forshared_ptr<TableChunk>.