Repository navigation
Conversation
…llectives rcclUpdateCollectiveProtocol() calls rcclGetProtoForGfx12() for single-node AllReduce/AllGather/ReduceScatter on gfx12, which can select NCCL_PROTO_LL for messages under ~890KB/~96KB. LL primitives (src/device/prims_ll.h) never received the gfx12 memory-ordering fixes that Simple primitives (src/device/prims_simple.h) got, so selecting LL on gfx1201 deadlocks during multi-GPU collectives (observed via vLLM TP=2 on dual R9700). P2P send/recv already disables LL for gfx12 explicitly (src/enqueue.cc:1012); this closes the one remaining gap in the collective path. Equivalent to setting NCCL_PROTO=Simple, but baked into the build so it doesn't need to be set at runtime. Issue: ROCm/rocm-systems#5480
Closed
1 task done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
rcclUpdateCollectiveProtocol()callsrcclGetProtoForGfx12()for single-node AllReduce/AllGather/ReduceScatter on gfx12, which can selectNCCL_PROTO_LLfor messages under the size thresholds (~890KB for AllReduce, ~96KB for AllGather/ReduceScatter).src/device/prims_ll.hnever received the gfx12 memory-ordering fixes thatsrc/device/prims_simple.hgot (introduced for gfx12 atomic acquire semantics), so selecting LL on gfx1201 deadlocks during multi-GPU collectives.P2P send/recv already disables LL for gfx12 explicitly (
src/enqueue.cc:1012); this closes the one remaining gap in the collective path by forcingNCCL_PROTO_SIMPLEinstead of callingrcclGetProtoForGfx12().This is functionally equivalent to setting
NCCL_PROTO=Simpleat runtime (the confirmed workaround), but bakes the fix into the build so downstream users don't need to set the env var.Fixes #5480 (reported as an RCCL deadlock during vLLM TP=2 inference on dual R9700 / gfx1201).
Root cause
Introduced by 9545ae0 ("Navi4 LL enablement and tuning"), which added
rcclGetProtoForGfx12()and removed the blanket LL disable for gfx12 in the tuning table, without adding the corresponding gfx12 device-side fixes to the LL primitives.Test plan
NCCL_PROTO=Simple)Longer term, LL should be re-enabled for gfx12 once
prims_ll.hgets the same gfx12 atomic-ordering treatment asprims_simple.h— this patch is a safe stopgap, not a permanent fix.