StreamingEngine._reset() - #22364
Merged
Merged
Conversation
madsbk
added
improvement
github-actions
Bot
added
Python
madsbk
force-pushed
the
engine_reset
branch
4 times, most recently
from
May 4, 2026 19:45
94ae3b1 to
0156d3f
Compare
pentschev
reviewed
May 5, 2026
madsbk
added a commit
to madsbk/cudf
that referenced
this pull request
May 5, 2026
Singleton commit Depend on NVIDIA#22364
Contributor
Author
|
/merge |
galipremsagar
pushed a commit
to galipremsagar/cudf
that referenced
this pull request
May 6, 2026
Generalizes `RayEngine._reset()` to `SPMDEngine` and `DaskEngine`. All three engines now swap the rapidsmpf `Context` and the Polars `GPUEngine` configuration in place, while keeping the communicator and RMM resource alive. The test suite is refactored to take advantage of this. A session-scoped `streaming_engines` fixture bootstraps the SPMD communicator and constructs a shared `SPMDEngine`. Per-test `spmd_engine` and `streaming_engine_factory` fixtures call `_reset` on the cached engine instead of constructing a new one. This pattern will be extended to `RayEngine` and `DaskEngine` tests in the future as they are incorporated into the test matrix. Authors: - Mads R. B. Kristensen (https://github.com/madsbk) Approvers: - Peter Andreas Entschev (https://github.com/pentschev) URL: NVIDIA#22364
rapids-bot Bot
pushed a commit
that referenced
this pull request
May 7, 2026
…22381) Builds on the cached `streaming_engines` fixture from #22364, which amortizes SPMD bootstrap via `_reset()`, and extends the same pattern to Dask and Ray. With this change, the test matrix runs against: `["in-memory", "spmd", "spmd-small", "dask", "ray"]` subject to package availability and `rrun` gating. We might change the different setups later, but for now CI runs: | Engine | Block Size(s) | GPU Configuration | |----------------|-----------------------|-------------------| | `SPMDEngine` | `"medium"`, `"small"` | Single GPU | | `DaskEngine` | `"medium"` | Single GPU | | `RayEngine` | `"medium"` | Two GPUs | Authors: - Mads R. B. Kristensen (https://github.com/madsbk) - Peter Andreas Entschev (https://github.com/pentschev) Approvers: - Matthew Murray (https://github.com/Matt711) - Bradley Dice (https://github.com/bdice) - Peter Andreas Entschev (https://github.com/pentschev) - Matthew Roeschke (https://github.com/mroeschke) URL: #22381
galipremsagar
pushed a commit
to galipremsagar/cudf
that referenced
this pull request
May 8, 2026
…VIDIA#22381) Builds on the cached `streaming_engines` fixture from NVIDIA#22364, which amortizes SPMD bootstrap via `_reset()`, and extends the same pattern to Dask and Ray. With this change, the test matrix runs against: `["in-memory", "spmd", "spmd-small", "dask", "ray"]` subject to package availability and `rrun` gating. We might change the different setups later, but for now CI runs: | Engine | Block Size(s) | GPU Configuration | |----------------|-----------------------|-------------------| | `SPMDEngine` | `"medium"`, `"small"` | Single GPU | | `DaskEngine` | `"medium"` | Single GPU | | `RayEngine` | `"medium"` | Two GPUs | Authors: - Mads R. B. Kristensen (https://github.com/madsbk) - Peter Andreas Entschev (https://github.com/pentschev) Approvers: - Matthew Murray (https://github.com/Matt711) - Bradley Dice (https://github.com/bdice) - Peter Andreas Entschev (https://github.com/pentschev) - Matthew Roeschke (https://github.com/mroeschke) URL: NVIDIA#22381
shrshi
pushed a commit
to shrshi/cudf
that referenced
this pull request
May 12, 2026
Generalizes `RayEngine._reset()` to `SPMDEngine` and `DaskEngine`. All three engines now swap the rapidsmpf `Context` and the Polars `GPUEngine` configuration in place, while keeping the communicator and RMM resource alive. The test suite is refactored to take advantage of this. A session-scoped `streaming_engines` fixture bootstraps the SPMD communicator and constructs a shared `SPMDEngine`. Per-test `spmd_engine` and `streaming_engine_factory` fixtures call `_reset` on the cached engine instead of constructing a new one. This pattern will be extended to `RayEngine` and `DaskEngine` tests in the future as they are incorporated into the test matrix. Authors: - Mads R. B. Kristensen (https://github.com/madsbk) Approvers: - Peter Andreas Entschev (https://github.com/pentschev) URL: NVIDIA#22364
shrshi
pushed a commit
to shrshi/cudf
that referenced
this pull request
May 12, 2026
…VIDIA#22381) Builds on the cached `streaming_engines` fixture from NVIDIA#22364, which amortizes SPMD bootstrap via `_reset()`, and extends the same pattern to Dask and Ray. With this change, the test matrix runs against: `["in-memory", "spmd", "spmd-small", "dask", "ray"]` subject to package availability and `rrun` gating. We might change the different setups later, but for now CI runs: | Engine | Block Size(s) | GPU Configuration | |----------------|-----------------------|-------------------| | `SPMDEngine` | `"medium"`, `"small"` | Single GPU | | `DaskEngine` | `"medium"` | Single GPU | | `RayEngine` | `"medium"` | Two GPUs | Authors: - Mads R. B. Kristensen (https://github.com/madsbk) - Peter Andreas Entschev (https://github.com/pentschev) Approvers: - Matthew Murray (https://github.com/Matt711) - Bradley Dice (https://github.com/bdice) - Peter Andreas Entschev (https://github.com/pentschev) - Matthew Roeschke (https://github.com/mroeschke) URL: NVIDIA#22381
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Generalizes
RayEngine._reset()toSPMDEngineandDaskEngine. All three engines now swap the rapidsmpfContextand the PolarsGPUEngineconfiguration in place, while keeping the communicator and RMM resource alive.The test suite is refactored to take advantage of this. A session-scoped
streaming_enginesfixture bootstraps the SPMD communicator and constructs a sharedSPMDEngine. Per-testspmd_engineandstreaming_engine_factoryfixtures call_reseton the cached engine instead of constructing a new one.This pattern will be extended to
RayEngineandDaskEnginetests in the future as they are incorporated into the test matrix.