Repository navigation
fix(rccl): Redirect P2P SendRecv protocol logs to NCCL_COLL debug subsystem - #11010
Merged
Merged
Conversation
Signed-off-by: Rahul Vaidya <ravaidya@amd.com>
✅ All Policy Checks Passed
📖 Need help? See the Policy FAQ for details on every check and how to fix failures. |
rahulvaidya20
marked this pull request as ready for review
August 31, 2026 23:35
rahulvaidya20
changed the title
wenkaidu
approved these changes
Aug 31, 2026
Contributor
There was a problem hiding this comment.
Pull request overview
This PR reduces default NCCL_DEBUG=INFO log volume by retagging the per-op P2P SendRecv protocol-selection log line from the default-enabled NCCL_INIT subsystem to the opt-in NCCL_COLL subsystem, and updates the associated unit tests accordingly.
Changes:
- Retag
RCCL P2P SendRecv protocol=...logging fromINFO(NCCL_INIT, ...)toINFO(NCCL_COLL, ...)in the P2P enqueue path. - Update SendRecv unit tests to enable
NCCL_DEBUG_SUBSYS=COLLwhen scraping logs for protocol selection. - Adjust related comments to reflect the new debug subsystem requirement.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| projects/rccl/src/enqueue.cc | Moves the P2P SendRecv protocol-selection INFO log from NCCL_INIT to NCCL_COLL so it’s no longer emitted under the default debug-subsystem mask. |
| projects/rccl/test/SendRecvTests.cpp | Updates SendRecv protocol-scraping tests to set NCCL_DEBUG_SUBSYS=COLL so they continue to observe the protocol-selection log line. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
nileshnegi
approved these changes
Aug 31, 2026
Contributor
TheRock Submodule Bump ActivityNewest first
|
corey-derochie-amd
pushed a commit
that referenced
this pull request
Oct 14, 2026
…system (#11010) ## Motivation PR #10243 added `INFO(NCCL_INIT, "RCCL P2P SendRecv protocol=...")` in the P2P enqueue path. Because `NCCL_INIT` is in the default debug subsystem mask, this fires per send/recv direction per op under plain `NCCL_DEBUG=INFO`. For a 1 GB alltoall across 8 GPUs with `-n 1000`, this produces ~250k extra log lines (12.9× blowup) per run. ## Technical Details Move the subsystem tag from `NCCL_INIT` to `NCCL_COLL`. `NCCL_COLL` is not in the default mask (`NCCL_INIT | NCCL_BOOTSTRAP | NCCL_ENV`), so the line becomes opt-in via `NCCL_DEBUG_SUBSYS=COLL`. The log line itself and the string it emits are unchanged; tests are updated to set `NCCL_DEBUG_SUBSYS=COLL` instead of `INIT`. ## Issue Tracking JIRA ID: ROCM-30236 ## Test Plan Run `SendRecv.*` unit tests (the 7 LL128/LegacyLL protocol-scraping tests introduced by #10243) with `UT_POW2_GPUS=1` on MI355X gfx950. ## Test Result All 7 tests pass on MI355X (gfx950): - `SendRecv.LegacyLLIntranode` — PASS - `SendRecv.LL128NetBuffersDisabled` — PASS - `SendRecv.LL128NetBuffersEnabled` — PASS - `SendRecv.Ll128DisabledDoesNotEnableP2pLL128` — PASS - `SendRecv.Ll128EnabledDoesNotEnableP2pLL128` — PASS - `SendRecv.SeparateThresholdLL128UsesLL128Knob` — PASS - `SendRecv.SeparateThresholdLegacyLLUsesLLKnob` — PASS ## Submission Checklist - [ ] Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests. Signed-off-by: Rahul Vaidya <ravaidya@amd.com>
corey-derochie-amd
pushed a commit
that referenced
this pull request
Oct 15, 2026
…system (#11010) ## Motivation PR #10243 added `INFO(NCCL_INIT, "RCCL P2P SendRecv protocol=...")` in the P2P enqueue path. Because `NCCL_INIT` is in the default debug subsystem mask, this fires per send/recv direction per op under plain `NCCL_DEBUG=INFO`. For a 1 GB alltoall across 8 GPUs with `-n 1000`, this produces ~250k extra log lines (12.9× blowup) per run. ## Technical Details Move the subsystem tag from `NCCL_INIT` to `NCCL_COLL`. `NCCL_COLL` is not in the default mask (`NCCL_INIT | NCCL_BOOTSTRAP | NCCL_ENV`), so the line becomes opt-in via `NCCL_DEBUG_SUBSYS=COLL`. The log line itself and the string it emits are unchanged; tests are updated to set `NCCL_DEBUG_SUBSYS=COLL` instead of `INIT`. ## Issue Tracking JIRA ID: ROCM-30236 ## Test Plan Run `SendRecv.*` unit tests (the 7 LL128/LegacyLL protocol-scraping tests introduced by #10243) with `UT_POW2_GPUS=1` on MI355X gfx950. ## Test Result All 7 tests pass on MI355X (gfx950): - `SendRecv.LegacyLLIntranode` — PASS - `SendRecv.LL128NetBuffersDisabled` — PASS - `SendRecv.LL128NetBuffersEnabled` — PASS - `SendRecv.Ll128DisabledDoesNotEnableP2pLL128` — PASS - `SendRecv.Ll128EnabledDoesNotEnableP2pLL128` — PASS - `SendRecv.SeparateThresholdLL128UsesLL128Knob` — PASS - `SendRecv.SeparateThresholdLegacyLLUsesLLKnob` — PASS ## Submission Checklist - [ ] Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests. Signed-off-by: Rahul Vaidya <ravaidya@amd.com>
corey-derochie-amd
pushed a commit
that referenced
this pull request
Oct 15, 2026
…system (#11010) ## Motivation PR #10243 added `INFO(NCCL_INIT, "RCCL P2P SendRecv protocol=...")` in the P2P enqueue path. Because `NCCL_INIT` is in the default debug subsystem mask, this fires per send/recv direction per op under plain `NCCL_DEBUG=INFO`. For a 1 GB alltoall across 8 GPUs with `-n 1000`, this produces ~250k extra log lines (12.9× blowup) per run. ## Technical Details Move the subsystem tag from `NCCL_INIT` to `NCCL_COLL`. `NCCL_COLL` is not in the default mask (`NCCL_INIT | NCCL_BOOTSTRAP | NCCL_ENV`), so the line becomes opt-in via `NCCL_DEBUG_SUBSYS=COLL`. The log line itself and the string it emits are unchanged; tests are updated to set `NCCL_DEBUG_SUBSYS=COLL` instead of `INIT`. ## Issue Tracking JIRA ID: ROCM-30236 ## Test Plan Run `SendRecv.*` unit tests (the 7 LL128/LegacyLL protocol-scraping tests introduced by #10243) with `UT_POW2_GPUS=1` on MI355X gfx950. ## Test Result All 7 tests pass on MI355X (gfx950): - `SendRecv.LegacyLLIntranode` — PASS - `SendRecv.LL128NetBuffersDisabled` — PASS - `SendRecv.LL128NetBuffersEnabled` — PASS - `SendRecv.Ll128DisabledDoesNotEnableP2pLL128` — PASS - `SendRecv.Ll128EnabledDoesNotEnableP2pLL128` — PASS - `SendRecv.SeparateThresholdLL128UsesLL128Knob` — PASS - `SendRecv.SeparateThresholdLegacyLLUsesLLKnob` — PASS ## Submission Checklist - [ ] Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests. Signed-off-by: Rahul Vaidya <ravaidya@amd.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
PR #10243 added
INFO(NCCL_INIT, "RCCL P2P SendRecv protocol=...")in the P2P enqueue path. BecauseNCCL_INITis in the default debug subsystem mask, this fires per send/recv direction per op under plainNCCL_DEBUG=INFO. For a 1 GB alltoall across 8 GPUs with-n 1000, this produces ~250k extra log lines (12.9× blowup) per run.Technical Details
Move the subsystem tag from
NCCL_INITtoNCCL_COLL.NCCL_COLLis not in the default mask (NCCL_INIT | NCCL_BOOTSTRAP | NCCL_ENV), so the line becomes opt-in viaNCCL_DEBUG_SUBSYS=COLL. The log line itself and the string it emits are unchanged; tests are updated to setNCCL_DEBUG_SUBSYS=COLLinstead ofINIT.Issue Tracking
JIRA ID: ROCM-30236
Test Plan
Run
SendRecv.*unit tests (the 7 LL128/LegacyLL protocol-scraping tests introduced by #10243) withUT_POW2_GPUS=1on MI355X gfx950.Test Result
All 7 tests pass on MI355X (gfx950):
SendRecv.LegacyLLIntranode— PASSSendRecv.LL128NetBuffersDisabled— PASSSendRecv.LL128NetBuffersEnabled— PASSSendRecv.Ll128DisabledDoesNotEnableP2pLL128— PASSSendRecv.Ll128EnabledDoesNotEnableP2pLL128— PASSSendRecv.SeparateThresholdLL128UsesLL128Knob— PASSSendRecv.SeparateThresholdLegacyLLUsesLLKnob— PASSSubmission Checklist