Repository navigation
ci(rccl): point PyTorch c10d tests at the current ROCm nightly index - #13789
Merged
prasanna-amd merged 5 commits intoOct 29, 2026
Merged
Conversation
github-actions
Bot
added
the
github actions
✅ All Policy Checks Passed
📖 Need help? See the Policy FAQ for details on every check and how to fix failures. |
Contributor
There was a problem hiding this comment.
🟢 Approval recommended
No blocking issues were identified in the focused index update; GPU CI validation remains pending.
No open issues 🎉
What changed in this PR
Restores RCCL’s PyTorch c10d CI coverage against current ROCm nightlies instead of the stale wheel index.
Changes:
- Switches PyTorch installation to the current nightly index.
- Adds the core nightly index for device dependencies and documents why it is required.
| File | Description |
|---|---|
.github/workflows/therock-rccl-test-pytorch-distributed.yml |
Updates package indexes while preserving floating nightly installs. |
Review effort: Balanced
5 tasks
![amd-argus[bot]](https://avatars.githubusercontent.com/in/4740020?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
ARGUS
Findings
5 more finding(s), not posted inline
projects/rccl/ci/scripts/known_failures.json:36(Low) test_nccl_backend_4gpu_module (an active hang) is suppressed by name, so any other failure of it will also pass silentlyprojects/rccl/ci/scripts/known_failures.json:47(Low) split_group_parent_guard says 'not pure upstream drift' without naming the ROCm-side part, which leaves the cause unclearprojects/rccl/ci/scripts/known_failures.json:48(Low) nccl_backend_4gpu_hang is an 'ACTIVE DEFECT', but the file's _comment (line 2) says listed tests are 'not regressions'projects/rccl/ci/scripts/known_failures.json:48(Nit) AICOMRCCL-2468 is cited twice in the nccl_backend_4gpu_hang description.github/workflows/therock-rccl-test-pytorch-distributed.yml:109(Low) Follow-up: therock-rccl-test-jax-collective.yml and rocjitsu-corpus-tests.yml still install from the stale whl-multi-arch index
Bottom line: Looks approval ready.
Comment @argus review to re-run, or @argus re-review after pushing.
prasanna-amd
force-pushed
the
users/prasanna-amd/pin-pytorch-nightly-index
branch
from
October 23, 2026 17:58
bcaf1bb to
469bc4e
Compare
prasanna-amd
requested review from
Nikhil-Nunna,
alex-breslow-amd,
alex-mckinley,
atulkulk,
pvallem and
wenkaidu
October 23, 2026 18:00
Contributor
pr-rocm-systems-13789-oci-rccl-pr-buildBuilds
FailuresNo failing tests in this run. 183 tests skipped. pr-rocm-systems-13789-oci-rccl-pr-smokeBuilds
Failures
15 tests skipped. |
Contributor
pr-rocm-systems-13789-ruby64-rccl-pr-buildBuilds
FailuresNo failing tests in this run. 183 tests skipped. pr-rocm-systems-13789-ruby64-rccl-pr-smokeBuilds
Failures
15 tests skipped. |
cmcknigh
reviewed
Oct 23, 2026
cmcknigh
left a comment
cmcknigh
left a comment
Contributor
There was a problem hiding this comment.
P3 (minor): The all-scope step "Run PyTorch c10d NCCL tests" took 119 min 09 s on the new nightly, against timeout-minutes: 120 at .github/workflows/therock-rccl-test-pytorch-distributed.yml:120. Known hangs still run to completion before they are dropped from the verdict, so a slightly slower node kills the step before pytest writes pytorch_c10d_results.xml.
On the 2026-10-20 dispatch (run 39347652693, job 118384041919) the step ran 02:55:55 → 04:55:04. pytest reported 31 failed, 262 passed, 20 skipped in 7128.38s (313 items; the previous wheel collected 288). The 2026-10-17 develop run on the old wheel (job 117342519675) took 117 min 55 s. Known failures alone are about 77 minutes: six NCCLTraceTest::test_trace_while_* at 600 s, plus test_error_detection_and_propagation, test_restart_pg_after_error, and test_nccl_backend_4gpu_module at 300 s each.
If the step is killed, the summary, email, and webhook steps are skipped and the job reports a timeout instead of results.
Would you raise the all timeout (for example to 150), or deselect the known hangs with -k in run_tests() so they do not consume the step budget?
Contributor
Author
|
@cmcknigh thanks — this was right, and it was worse than the single data point suggested. Addressed in c83c86c. Raised the A second near-miss you did not cite: job 119021568812 on 2026-10-21 ran 117 m 44 s, a 2 m 16 s margin. So it is two runs inside ~2.5 min of the ceiling, not one — the ceiling was already being hit, not approaching. One correction: there are seven 150 is headroom, not a fix, and I did not want to leave it looking like one. Filed AICOMRCCL-3178 for the actual work: cap the known hangs via Separately in the same commit, the three Argus re-reviewed at c83c86c and came back LGTM. CI is re-running on the rebased head. |
Contributor
Author
|
@pvallem @alex-breslow-amd @wenkaidu @alex-mckinley @atulkulk @Nikhil-Nunna — an approval on this one when you get a moment. Scope is small and CI-only: the c10d workflow stops installing PyTorch from the frozen The suite is currently running against a wheel index that no longer tracks ROCm, so every day this sits is a day the c10d signal is stale. |
cmcknigh
approved these changes
Oct 23, 2026
Contributor
pr-rocm-systems-13789-oci-rccl-pr-buildBuilds
FailuresNo failing tests in this run. 183 tests skipped. pr-rocm-systems-13789-oci-rccl-pr-smokeBuilds
Failures
15 tests skipped. |
Contributor
pr-rocm-systems-13789-ruby64-rccl-pr-smokeBuilds
FailuresNo failing tests in this run. 15 tests skipped. |
pvallem
approved these changes
Oct 24, 2026
The whl-multi-arch index on rocm.nightlies.amd.com stopped publishing torch on 2026-08-22. Because uv resolves to the newest wheel the index offers, the job has silently installed the same frozen build ever since — every pytorch_c10d run since 2026-09-12 reports 2.15.0a0+rocm10.1.0a20260822. Nightlies now ship from nightly.repo.amd.com/rocm/pytorch/whl-next/, which currently resolves 2.16.0a0+rocm10.2.0a20261019. The device-gfx942 extra resolves rocm-sdk-device-gfx942 from the separate core/whl-next index, so that is added as an extra index; without it the resolve fails outright. test_pytorch_c10d.py date-pins the cloned test sources off the installed wheel's version string, so the test tree moves forward with the wheel rather than needing a separate bump. This also unblocks AICOMRCCL-2467: the shrink_group teardown hang is fixed upstream by pytorch/pytorch#194775 (landed 2026-10-13), which the frozen August wheel cannot contain.
Moving this suite onto the current ROCm nightly surfaced two tests that keep the job red for reasons unrelated to the index change this PR makes. test_comm_recursive_split_group fails because upstream PyTorch #195758 (e8430bed0) made split_group reject a parent process group bound to a device_id that has not yet run a collective, and the test splits recursively without issuing one. The suite was previously pinned to a wheel predating that guard, which also froze the test sources, so the change only became visible once the pin was lifted. Tracked as AICOMRCCL-3107. test_nccl_backend_4gpu_module hangs with both ranks unresponsive and is killed at the 300s timeout. This one is an active defect rather than a feature gap and is called out as such in the category text: it is suppressed only so the suite's signal stays readable, and the entry should come out as soon as AICOMRCCL-2468 is fixed. Both category descriptions cite their Jira key, following the precedent set by extra_context_heuristic.
The all scope has come within ~2 minutes of the 120 minute ceiling on two separate runs (117m44s and 119m09s). Overrunning is worse than a slow run: the step is killed before pytest writes pytorch_c10d_results.xml, so the summary, email and webhook steps all skip and the job reports a timeout rather than any result at all. Raise the ceiling to 150. This is headroom, not a fix. Roughly 75 of the 120 minutes is spent on known-failure hangs running to their own 600s/300s timeouts, and the suite has no intra-job parallelism. Both are tracked separately. Also addresses review feedback on known_failures.json: - The file header claimed every listed test was "not a regression", which contradicted the nccl_backend_4gpu_hang entry declaring itself an active defect. Reword the header to allow for both, and to say plainly that exclusion from the verdict is not a claim that a test should fail forever. - Drop the duplicated AICOMRCCL-2468 reference. - split_group_parent_guard asserted the failure was "not pure upstream drift" without saying what the ROCm-side part was. State what is actually known -- sibling split_group tests pass, so the guard is satisfiable here -- and say explicitly that the specific cause is still unidentified.
prasanna-amd
force-pushed
the
users/prasanna-amd/pin-pytorch-nightly-index
branch
from
October 26, 2026 23:27
c83c86c to
d0596c6
Compare
Contributor
pr-rocm-systems-13789-oci-rccl-pr-smokeBuilds
FailuresNo failing tests in this run. 15 tests skipped. |
Contributor
pr-rocm-systems-13789-ruby64-rccl-pr-smokeBuilds
FailuresNo failing tests in this run. 15 tests skipped. |
prasanna-amd
added a commit
that referenced
this pull request
Oct 27, 2026
…xes (#13790) JIRA ID: AICOMRCCL-2468 ## Summary > **Correction (2026-10-20):** an earlier version of this description claimed the job installs `jax-rocm7-plugin 0.9.1+rocm7.15.0a20260728`. That was wrong — it was inferred from the index listing rather than from a CI log. The actual logs show something different, and worse. Corrected below. The JAX job's install is broken in two ways, neither of which has ever produced a red build. **1. The ROCm nightly plugin is not being installed at all — PyPI's is.** `uv` ranks `--extra-index-url` *above* `--index-url`. This job passes the nightly index as the base and PyPI as the extra, so the plain-PyPI `jax-rocm7-plugin` outranks the ROCm nightly build and is installed in its place. Every mirrored CI log back to **2026-08-16** (the oldest retained) shows a bare PyPI version with no `+rocm` local tag: ``` + jax==0.11.2 + jax-rocm7-pjrt==0.11.2 + jax-rocm7-plugin==0.11.2 <- PyPI stable, not a ROCm nightly + jaxlib==0.11.2 ``` Because PyPI's version floats, this *looks* healthy — the number moves (`0.11.0.post1` -> `0.11.1` -> `0.11.2`). It is just never the ROCm build. So the suite has not been exercising ROCm nightly JAX at all. **2. The ROCm SDK is frozen four months back.** `rocm-sdk-core` comes from the `v2/<family>/` index, which still returns 200 and still serves wheels — it just stopped publishing. Identical in every log since 2026-08-16: ``` + rocm-sdk-core==7.14.0a20260612 + rocm-sdk-libraries-gfx94x-dcgpu==7.14.0a20260612 ``` `2026-06-12`. That is the genuinely stale component. Nightlies now ship from `nightly.repo.amd.com`, and the plugin packages were renamed `rocm7` -> `rocm10`. Current nightly: `0.11.2+rocm10.2.0a20261019`. ## What this PR changes Index order is inverted so the nightly build actually wins, and the SDK is taken from the new core index. ``` # before: nightly as base, pypi as extra -> PyPI wins + jax-rocm10-plugin==0.12.0 # after: pypi as base, nightly as extra -> nightly wins + jax-rocm10-plugin==0.11.2+rocm10.2.0a20261019 ``` The order is load-bearing, not cosmetic. **`jax`/`jaxlib` are held to `0.11.*`** because `jax-rocm10-plugin` declares no `jax` or `jaxlib` dependency at all — unconstrained, the resolve pairs `jax 0.12.0` with the `0.11.2` plugin. The ROCm nightly date still floats. **The plugin's new `[with-rocm]` extra is not usable yet.** It requires `amd_rccl_instinct`, which 404s on every `nightly.repo.amd.com` index and on PyPI (though `amd-rocm-hip-runtime-instinct` *is* on PyPI — the set is partial). So `rocm-sdk-core` / `rocm-sdk-device-gfx942` from the core index are used instead, the same role the retired `v2/<family>/` index served. Worth revisiting once those publish. ## Verified resolve ``` + rocm-sdk-core==10.2.0a20261019 + rocm-sdk-device-gfx942==10.2.0a20261019 + rocm-sdk-libraries==10.2.0a20261019 + jax==0.11.2 + jaxlib==0.11.2 + jax-rocm10-plugin==0.11.2+rocm10.2.0a20261019 + jax-rocm10-pjrt==0.11.2+rocm10.2.0a20261019 ``` Consistent ROCm nightly date across SDK and plugin. ## Note on `rocm-sdk-device-gfx942` `rocm-sdk-libraries-<family>` has no equivalent under the new core index naming; it is `rocm-sdk-device-gfx942`. `gfx942` is hardcoded, matching the already-hardcoded family in the PyTorch job — these suites only run on gfx942 runners and gfx950 is excluded. Needs revisiting if this job is ever extended to another family. ## Risk Higher than #13789, and the real jump is on the ROCm side: SDK `7.14` -> `10.2` (four months), and the plugin moves from a PyPI stable build to a ROCm nightly one for the first time. The JAX frontend itself stays on `0.11.2`, which is what CI already runs, so the Python-level API risk is low. The plugin/PJRT runtime wiring is the part to watch. ## Test plan - [ ] `therock-rccl-test-jax-collective.yml` dispatched on this branch - [ ] Confirm the log shows `jax-rocm10-plugin==0.11.2+rocm10.2.0a2026101x` **with** the `+rocm` tag — a bare `0.11.2` means PyPI won again and the fix did not take - [ ] Confirm `rocm-sdk-core==10.2.0a2026101x`, not `7.14.0a20260612` - [ ] Confirm the plugin loads and resolves math libs (no RUNPATH / `_rocm_sdk_libraries` failure) - [ ] Confirm all 10 collective smoke tests pass Co-authored-by: renjithravindrankannath <94420380+renjithravindrankannath@users.noreply.github.com>
Contributor
pr-rocm-systems-13789-oci-rccl-pr-smokeBuilds
FailuresNo failing tests in this run. 15 tests skipped. |
Contributor
pr-rocm-systems-13789-ruby64-rccl-pr-buildBuilds
FailuresNo failing tests in this run. 183 tests skipped. pr-rocm-systems-13789-ruby64-rccl-pr-smokeBuilds
Failures
15 tests skipped. |
ywang103-amd
pushed a commit
that referenced
this pull request
Oct 28, 2026
…xes (#13790) JIRA ID: AICOMRCCL-2468 ## Summary > **Correction (2026-10-20):** an earlier version of this description claimed the job installs `jax-rocm7-plugin 0.9.1+rocm7.15.0a20260728`. That was wrong — it was inferred from the index listing rather than from a CI log. The actual logs show something different, and worse. Corrected below. The JAX job's install is broken in two ways, neither of which has ever produced a red build. **1. The ROCm nightly plugin is not being installed at all — PyPI's is.** `uv` ranks `--extra-index-url` *above* `--index-url`. This job passes the nightly index as the base and PyPI as the extra, so the plain-PyPI `jax-rocm7-plugin` outranks the ROCm nightly build and is installed in its place. Every mirrored CI log back to **2026-08-16** (the oldest retained) shows a bare PyPI version with no `+rocm` local tag: ``` + jax==0.11.2 + jax-rocm7-pjrt==0.11.2 + jax-rocm7-plugin==0.11.2 <- PyPI stable, not a ROCm nightly + jaxlib==0.11.2 ``` Because PyPI's version floats, this *looks* healthy — the number moves (`0.11.0.post1` -> `0.11.1` -> `0.11.2`). It is just never the ROCm build. So the suite has not been exercising ROCm nightly JAX at all. **2. The ROCm SDK is frozen four months back.** `rocm-sdk-core` comes from the `v2/<family>/` index, which still returns 200 and still serves wheels — it just stopped publishing. Identical in every log since 2026-08-16: ``` + rocm-sdk-core==7.14.0a20260612 + rocm-sdk-libraries-gfx94x-dcgpu==7.14.0a20260612 ``` `2026-06-12`. That is the genuinely stale component. Nightlies now ship from `nightly.repo.amd.com`, and the plugin packages were renamed `rocm7` -> `rocm10`. Current nightly: `0.11.2+rocm10.2.0a20261019`. ## What this PR changes Index order is inverted so the nightly build actually wins, and the SDK is taken from the new core index. ``` # before: nightly as base, pypi as extra -> PyPI wins + jax-rocm10-plugin==0.12.0 # after: pypi as base, nightly as extra -> nightly wins + jax-rocm10-plugin==0.11.2+rocm10.2.0a20261019 ``` The order is load-bearing, not cosmetic. **`jax`/`jaxlib` are held to `0.11.*`** because `jax-rocm10-plugin` declares no `jax` or `jaxlib` dependency at all — unconstrained, the resolve pairs `jax 0.12.0` with the `0.11.2` plugin. The ROCm nightly date still floats. **The plugin's new `[with-rocm]` extra is not usable yet.** It requires `amd_rccl_instinct`, which 404s on every `nightly.repo.amd.com` index and on PyPI (though `amd-rocm-hip-runtime-instinct` *is* on PyPI — the set is partial). So `rocm-sdk-core` / `rocm-sdk-device-gfx942` from the core index are used instead, the same role the retired `v2/<family>/` index served. Worth revisiting once those publish. ## Verified resolve ``` + rocm-sdk-core==10.2.0a20261019 + rocm-sdk-device-gfx942==10.2.0a20261019 + rocm-sdk-libraries==10.2.0a20261019 + jax==0.11.2 + jaxlib==0.11.2 + jax-rocm10-plugin==0.11.2+rocm10.2.0a20261019 + jax-rocm10-pjrt==0.11.2+rocm10.2.0a20261019 ``` Consistent ROCm nightly date across SDK and plugin. ## Note on `rocm-sdk-device-gfx942` `rocm-sdk-libraries-<family>` has no equivalent under the new core index naming; it is `rocm-sdk-device-gfx942`. `gfx942` is hardcoded, matching the already-hardcoded family in the PyTorch job — these suites only run on gfx942 runners and gfx950 is excluded. Needs revisiting if this job is ever extended to another family. ## Risk Higher than #13789, and the real jump is on the ROCm side: SDK `7.14` -> `10.2` (four months), and the plugin moves from a PyPI stable build to a ROCm nightly one for the first time. The JAX frontend itself stays on `0.11.2`, which is what CI already runs, so the Python-level API risk is low. The plugin/PJRT runtime wiring is the part to watch. ## Test plan - [ ] `therock-rccl-test-jax-collective.yml` dispatched on this branch - [ ] Confirm the log shows `jax-rocm10-plugin==0.11.2+rocm10.2.0a2026101x` **with** the `+rocm` tag — a bare `0.11.2` means PyPI won again and the fix did not take - [ ] Confirm `rocm-sdk-core==10.2.0a2026101x`, not `7.14.0a20260612` - [ ] Confirm the plugin loads and resolves math libs (no RUNPATH / `_rocm_sdk_libraries` failure) - [ ] Confirm all 10 collective smoke tests pass Co-authored-by: renjithravindrankannath <94420380+renjithravindrankannath@users.noreply.github.com>
Contributor
pr-rocm-systems-13789-oci-rccl-pr-smokeBuilds
FailuresNo failing tests in this run. 15 tests skipped. |
Contributor
pr-rocm-systems-13789-ruby64-rccl-pr-buildBuilds
FailuresNo failing tests in this run. 183 tests skipped. pr-rocm-systems-13789-ruby64-rccl-pr-smokeBuilds
Failures
15 tests skipped. |
Contributor
pr-rocm-systems-13789-ruby64-rccl-pr-smokeBuilds
FailuresNo failing tests in this run. 15 tests skipped. |
systems-assistant Bot
pushed a commit
to ROCm/rccl
that referenced
this pull request
Oct 29, 2026
nightly index (#13789) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit JIRA ID: AICOMRCCL-2468 ## Summary The `whl-multi-arch` index on `rocm.nightlies.amd.com` **stopped publishing torch on 2026-08-22**. It still returns 200 and still serves wheels, so nothing failed — `uv` just resolves to the newest wheel the index offers, which has not moved since August. Every `pytorch_c10d` CI run since 2026-09-12 has installed the identical build: ``` 2.15.0a0+rocm10.1.0a20260822 ``` So for two months this job has been reporting green against a two-month-old PyTorch while appearing to test nightlies. This points the install at the current publishing location, `nightly.repo.amd.com/rocm/pytorch/whl-next/`, which today resolves `2.16.0a0+rocm10.2.0a20261019`. ## Why the extra index is required The `device-gfx942` extra resolves `rocm-sdk-device-gfx942`, which on the new host is published to `rocm/core/whl-next/`, **not** `rocm/pytorch/whl-next/`. With a single `--index-url` the resolve fails outright: ``` And because you require torch[device-gfx942], we can conclude that your requirements are unsatisfiable. ``` With both indexes it resolves cleanly (verified via `uv pip install --dry-run`): ``` + torch==2.16.0a0+rocm10.2.0a20261019 + rocm-sdk-core==10.2.0a20261019 + rocm-sdk-device-gfx942==10.2.0a20261019 + rocm-sdk-libraries==10.2.0a20261019 + triton==3.9.0+gitfc8d31a5.rocm10.2.0a20261019 ``` ## Test sources follow automatically `projects/rccl/ci/scripts/test_pytorch_c10d.py::clone_pytorch_test_sources` date-pins the cloned PyTorch test tree off the installed wheel's version string (the `--shallow-since` path). The test tree therefore moves forward with the wheel — no separate bump needed here. ## Relationship to AICOMRCCL-2467 `test_shrink_group_flags` has been timing out at 300 s in CI. The upstream teardown fix is pytorch/pytorch#194775 (landed 2026-10-13), which the frozen August wheel cannot possibly contain. Because the test sources are date-pinned off the wheel, this change pulls in both the fixed runtime and the updated test. This is complementary to #12446 (the RCCL-side destroy-path barrier fix), not a replacement for it. ## Deliberately floating, not pinned This tracks the latest nightly rather than pinning an exact build, which restores the originally intended behaviour of this job. The tradeoff is that an upstream nightly regression can turn CI red; that is preferred here over silently testing stale wheels, which is the failure mode this PR exists to fix. ## Test plan - [ ] `therock-rccl-test-pytorch-distributed.yml` dispatched on this branch, `test_scope: all` - [ ] Confirm the run log reports a `2.16.0a0+rocm10.2.0a2026101x` version rather than `...20260822` - [ ] Confirm `test_shrink_group_flags` no longer times out [rocm-systems] ROCm/rocm-systems#13789 (commit 69dc7a7)
Contributor
TheRock Submodule Bump ActivityNewest first
|
Merged
2 of 3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
JIRA ID: AICOMRCCL-2468
Summary
The
whl-multi-archindex onrocm.nightlies.amd.comstopped publishing torch on 2026-08-22. It still returns 200 and still serves wheels, so nothing failed —uvjust resolves to the newest wheel the index offers, which has not moved since August.Every
pytorch_c10dCI run since 2026-09-12 has installed the identical build:So for two months this job has been reporting green against a two-month-old PyTorch while appearing to test nightlies.
This points the install at the current publishing location,
nightly.repo.amd.com/rocm/pytorch/whl-next/, which today resolves2.16.0a0+rocm10.2.0a20261019.Why the extra index is required
The
device-gfx942extra resolvesrocm-sdk-device-gfx942, which on the new host is published torocm/core/whl-next/, notrocm/pytorch/whl-next/. With a single--index-urlthe resolve fails outright:With both indexes it resolves cleanly (verified via
uv pip install --dry-run):Test sources follow automatically
projects/rccl/ci/scripts/test_pytorch_c10d.py::clone_pytorch_test_sourcesdate-pins the cloned PyTorch test tree off the installed wheel's version string (the--shallow-sincepath). The test tree therefore moves forward with the wheel — no separate bump needed here.Relationship to AICOMRCCL-2467
test_shrink_group_flagshas been timing out at 300 s in CI. The upstream teardown fix is pytorch/pytorch#194775 (landed 2026-10-13), which the frozen August wheel cannot possibly contain. Because the test sources are date-pinned off the wheel, this change pulls in both the fixed runtime and the updated test.This is complementary to #12446 (the RCCL-side destroy-path barrier fix), not a replacement for it.
Deliberately floating, not pinned
This tracks the latest nightly rather than pinning an exact build, which restores the originally intended behaviour of this job. The tradeoff is that an upstream nightly regression can turn CI red; that is preferred here over silently testing stale wheels, which is the failure mode this PR exists to fix.
Test plan
therock-rccl-test-pytorch-distributed.ymldispatched on this branch,test_scope: all2.16.0a0+rocm10.2.0a2026101xversion rather than...20260822test_shrink_group_flagsno longer times out