Repository navigation
rccl: add CAST environment variables to docs - #8613
Conversation
f6e019f to
c8074cd
Compare
✅ All Policy Checks Passed
📖 Need help? See the Policy FAQ for details on every check and how to fix failures. |
|
🚫 Please fix the failed policies before requesting reviews. The following policy checks failed:
The |
There was a problem hiding this comment.
Pull request overview
Adds documentation for CAST (Congestion aware sprayed traffic) QP scheduling tunables in the RCCL environment variables reference, describing how to enable the scheduler and configure its update/reset behavior, weighting, split-data mode, and logging.
Changes:
- Introduces a new “QP scheduling (CAST)” section in the env var reference.
- Documents CAST scheduler enable/WRR, timing/weighting knobs, split-data chunk minimum, and log path/interval.
- Adds guidance for
NCCL_IB_SPLIT_DATA_ON_QPSbehavior when distributing traffic across QPs.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
c8074cd to
7d1f4b6
Compare
3f7fac8 to
73140d7
Compare
|
This pull request has been inactive for 25 days and will be marked as stale. If you would like to keep this PR open, please:
This PR will be automatically closed in 5 days if no further activity occurs. |
73140d7 to
4e7cccf
Compare
![amd-argus[bot]](https://avatars.githubusercontent.com/in/4740020?s=60&v=4){kind=small}
There was a problem hiding this comment.
ARGUS
Bottom line: Worth settling before approval: the RCCL_IB_QP_SCHED_ENABLE row says NCCL_NET=IB-CAST forces the scheduler on, which does not hold on AINIC, where the matching scheme defaults to BY_ORDER.
Comment @argus review to re-run, or @argus re-review after pushing.
|
@argus re-review |
|
@VadimKutovoi the current head ( |
4e7cccf to
75b382b
Compare
|
@argus re-review |
|
@VadimKutovoi the current head ( |
75b382b to
2aed6c1
Compare
Fix the default-enablement claim (it's disabled by default on AINIC, not enabled everywhere except when forced), scope the port failover/QP-sharing override to apply regardless of the enable value, and explain concretely when a connection resolves to BY_ORDER receive matching. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
List accepted values flat up front, man-page style, and describe when the scheduler has no effect on AINIC in terms of other documented knobs (NCCL_IB_OOO_RQ, NCCL_IB_RECEIVER_SIDE_MATCHING_SCHEME) instead of the internal, undocumented BY_ORDER/CTS-offload machinery. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
There was a problem hiding this comment.
ARGUS
Bottom line: Worth settling before approval: the RCCL_IB_QP_SCHED_ENABLE auto rule at lines 811-814 does not match rcclUseIbCastQpSched for NCCL_NET=ROCM-IB on non-AINIC NICs, or for NCCL_NET=IB-CAST on AINIC while CTS offload is on.
Comment @argus review to re-run, or @argus re-review after pushing.
5371907 to
09a519a
Compare
corey-derochie-amd
left a comment
corey-derochie-amd
left a comment
There was a problem hiding this comment.
Approved, but the wording is slightly misleading on a couple, recommend following the copilot suggestion.
Co-authored-by: VadimKutovoi <32107368+VadimKutovoi@users.noreply.github.com>
TheRock Submodule Bump ActivityNewest first
|
Motivation
This pull request adds documentation for the CAST (Congestion aware sprayed traffic) QP (Queue Pair) scheduler in the RCCL library. These variables allow users to tune the scheduler's operation, logging, and message distribution modes.
QP scheduling (CAST) documentation:
RCCL_orNCCL_prefixes), including enabling/disabling the scheduler, WRR scheduling, intervals for statistics reset and update, EMA weighting, minimum split size, logging options, and message distribution modes.Technical Details
JIRA ID
JIRA ID AICOMNET-37
Test Plan
Test Result
Submission Checklist