Repository navigation
ci(rccl): track torch 2.16's renamed pg_timeout test and drop two fixed known failures - #14675
Conversation
…ed 4gpu_module known failure torch 2.16 renamed the c10d parametrization test_extend_nccl_pg_timeout_backend_nccl to ..._backend_nccl-legacy. The pg_timeout_semantics failure is unchanged (timedelta(seconds=18) != timedelta(seconds=8) at test_c10d_nccl.py:1182), but the old key no longer matches, so the harness now reports it as unexpected and every develop c10d all-scope run fails. test_nccl_backend_4gpu_module passes on torch 2.16 (38.88 s) and AICOMRCCL-2468 is closed. Remove it and its nccl_backend_4gpu_hang category, as the entry itself asked. JIRA ID: AICOMRCCL-3499
✅ All Policy Checks Passed
📖 Need help? See the Policy FAQ for details on every check and how to fix failures. |
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues are recorded for the RCCL known-failure list update.
No open issues 🎉
What changed in this PR
Updates RCCL CI’s PyTorch c10d known-failure list to recognize the renamed timeout test in torch 2.16 and stop suppressing the resolved 4-GPU module failure.
Changes:
- Renames the timeout test entry to
test_extend_nccl_pg_timeout_backend_nccl-legacy, retaining its existing category. - Removes
test_nccl_backend_4gpu_moduleand its now-unusednccl_backend_4gpu_hangcategory. - Leaves the other known failures unchanged pending further confirmation.
| File | Description |
|---|---|
projects/rccl/ci/scripts/known_failures.json |
Updates the timeout test name and removes the resolved 4-GPU failure exclusion. |
Review effort: Balanced
The failure was a PyTorch bug, not a ROCm one. pytorch/pytorch#195758 made split_group reject a parent whose backend is not initialized, and NCCLComm::split never set initialized_ for blocking communicators. A split child therefore could not itself be split until a collective had run on it, which is exactly what this test does. pytorch/pytorch#202072 (1aa1528f66, landed 2026-10-27) sets the flag and names this test. History: passed on every torch 2.15 run, failed once on the 2.16 20261019 build, and passed (14.5 s) on the 20261029 build. JIRA ID: AICOMRCCL-3499
prasanna-amd
changed the title
prasanna-amd
changed the title
![amd-argus[bot]](https://avatars.githubusercontent.com/in/4740020?s=60&v=4){kind=small}
There was a problem hiding this comment.
ARGUS
Findings
1 more finding(s), not posted inline
projects/rccl/ci/scripts/known_failures.json:2(Low) _comment still says 'A minority are active defects' but no remaining entry is one after nccl_backend_4gpu_hang is removed; reword it as a rule for future entries.
Bottom line: LGTM.
Comment @argus review to re-run, or @argus re-review after pushing.
pr-rocm-systems-14675-oci-rccl-pr-smokeBuilds
FailuresNo failing tests in this run. 15 tests skipped. |
pr-rocm-systems-14675-ruby64-rccl-pr-smokeBuilds
FailuresNo failing tests in this run. 15 tests skipped. |
two fixed known failures (#14675) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit JIRA ID: AICOMRCCL-3499 ## Summary Every PyTorch c10d **all-scope** run on develop fails on one unexpected test, `test_extend_nccl_pg_timeout_backend_nccl-legacy`. It is not a new failure. It is the existing `pg_timeout_semantics` known failure, which torch 2.16 renamed from `test_extend_nccl_pg_timeout_backend_nccl`. `known_failures.json` still lists only the old name. The assertion is the same one this test has hit on every all-scope run since 2026-09-12: ``` test_c10d_nccl.py:1182 self.assertEqual(w.timeout, timedelta(seconds=8)) AssertionError: Object comparison failed: datetime.timedelta(seconds=18) != datetime.timedelta(seconds=8) ``` Observed on run [40941007532](https://github.com/ROCm/rocm-systems/actions/runs/40941007532/job/123684072400): develop RCCL artifacts from nightly 40829635440, torch `2.16.0a0+rocm10.3.0a20261029`. Result: `264 passed, 1 unexpected failures, 28 known failures -> FAILED`. ## Changes - **Rename** the key `test_extend_nccl_pg_timeout_backend_nccl` → `test_extend_nccl_pg_timeout_backend_nccl-legacy`. The category is unchanged. - **Remove** `test_nccl_backend_4gpu_module` and its `nccl_backend_4gpu_hang` category. The same run passed it in 38.88 s, after it had timed out at 300 s on every all-scope run from 9/19 to 10/24. AICOMRCCL-2468 was verified and closed on 2026-10-30, and the entry itself said to remove it at that point. - **Remove** `test_comm_recursive_split_group` and its `split_group_parent_guard` category (AICOMRCCL-3107). This was a PyTorch bug, not a ROCm one. pytorch/pytorch#195758 made `split_group` reject a parent whose backend is not initialized, and `NCCLComm::split` never set `initialized_` for blocking communicators. So a split child could not itself be split until a collective had run on it, which is exactly what this test does. [pytorch/pytorch#202072](pytorch/pytorch#202072) (`1aa1528f66`, landed 2026-10-27) sets the flag and names this test. History: passed on every torch 2.15 run (13), failed once on the 2.16 `20261019` build, and passed in 14.5 s on the `20261029` build. ## Deliberately not changed `test_extend_nccl_pg_timeout_backend0` also passed in that run, but only once, and no upstream fix has been identified for it. It stays listed until a second all-scope run confirms it. ## Verification I replayed the 29 failures and 264 passes from run 40941007532 through the harness's classification logic (`test_pytorch_c10d.py`, the `known_set` / `unexpected_failed` / `now_passing` block): | `known_failures.json` | unexpected | known | now passing | |---|---|---|---| | develop | `…_backend_nccl-legacy` | 28 | `test_comm_recursive_split_group`, `test_extend_nccl_pg_timeout_backend0`, `test_nccl_backend_4gpu_module` | | this PR | — | 29 | `test_extend_nccl_pg_timeout_backend0` | On the develop file, the replay reproduces the harness's own reported result exactly. The JSON parses, and every category referenced under `tests` is defined, with none left unused. ## Test plan - [x] Replay of run 40941007532 against this file: 0 unexpected failures - [x] JSON valid; no orphaned or undefined categories - [ ] Next c10d all-scope run on develop reports no unexpected failures [rocm-systems] ROCm/rocm-systems#14675 (commit 59bd3ca)
PR Pundit review(Testing PR Pundit Code Review agent. Not all comments may be relevant. This is a one-time automated pass: please address what is useful, no need to request a re-review. Please reply to each point, whether it applies or not: we harvest these replies, and both kinds are enormously helpful for tuning the agent. Reach out to @peymanr for suggestions.)
1. 🟠 Should fix: Nightly
|
TheRock Submodule Bump ActivityNewest first
|
JIRA ID: AICOMRCCL-3499
Summary
Every PyTorch c10d all-scope run on develop fails on one unexpected test,
test_extend_nccl_pg_timeout_backend_nccl-legacy. It is not a new failure. It is the existingpg_timeout_semanticsknown failure, which torch 2.16 renamed fromtest_extend_nccl_pg_timeout_backend_nccl.known_failures.jsonstill lists only the old name.The assertion is the same one this test has hit on every all-scope run since 2026-09-12:
Observed on run 40941007532: develop RCCL artifacts from nightly 40829635440, torch
2.16.0a0+rocm10.3.0a20261029. Result:264 passed, 1 unexpected failures, 28 known failures -> FAILED.Changes
test_extend_nccl_pg_timeout_backend_nccl→test_extend_nccl_pg_timeout_backend_nccl-legacy. The category is unchanged.test_nccl_backend_4gpu_moduleand itsnccl_backend_4gpu_hangcategory. The same run passed it in 38.88 s, after it had timed out at 300 s on every all-scope run from 9/19 to 10/24. AICOMRCCL-2468 was verified and closed on 2026-10-30, and the entry itself said to remove it at that point.test_comm_recursive_split_groupand itssplit_group_parent_guardcategory (AICOMRCCL-3107). This was a PyTorch bug, not a ROCm one. [distributed] Validate the parent backend before split_group pytorch/pytorch#195758 madesplit_groupreject a parent whose backend is not initialized, andNCCLComm::splitnever setinitialized_for blocking communicators. So a split child could not itself be split until a collective had run on it, which is exactly what this test does. pytorch/pytorch#202072 (1aa1528f66, landed 2026-10-27) sets the flag and names this test. History: passed on every torch 2.15 run (13), failed once on the 2.1620261019build, and passed in 14.5 s on the20261029build.Deliberately not changed
test_extend_nccl_pg_timeout_backend0also passed in that run, but only once, and no upstream fix has been identified for it. It stays listed until a second all-scope run confirms it.Verification
I replayed the 29 failures and 264 passes from run 40941007532 through the harness's classification logic (
test_pytorch_c10d.py, theknown_set/unexpected_failed/now_passingblock):known_failures.json…_backend_nccl-legacytest_comm_recursive_split_group,test_extend_nccl_pg_timeout_backend0,test_nccl_backend_4gpu_moduletest_extend_nccl_pg_timeout_backend0On the develop file, the replay reproduces the harness's own reported result exactly. The JSON parses, and every category referenced under
testsis defined, with none left unused.Test plan