[dlamd1dai]

JIRA: AICOMRCCL-2880 · driver defect: ROCM-32598

Based on develop. #12841 and #13235 have both merged, and this branch is one commit on top of them. It relies on #13235's node-local agreement, which turns a connect failure into the IPC fallback on every rank.

Problem

Anvil rings every SDMA doorbell from a GPU kernel, and some doorbell slots never see a write made that way. Host-written doorbells always work, so ROCr's own SDMA queues are unaffected. Before #12841, rocSHMEM alltoall and put hung as soon as a transfer took the SDMA path on CPX (MI300X and MI355X) and on DPX (MI300X), including with the default ROCSHMEM_SDMA_NUM_CHANNELS=1. #12841 avoids the problem by using even engines only. This PR replaces that restriction with the slot-level rule below, so odd engines carry traffic again.

A multi-queue probe (attached to ROCM-32598) on MI300X SPX, covering all 16 engines x 8 slots, shows the pattern:

Engine	Slots 0-3	Slots 4-7
even (0, 2, ..., 14)	work	work
odd (1, 3, ..., 15)	never serviced	work

A GPU-written doorbell reaches the SDMA instance that owns the start of its 32-byte block, while a host write reaches the instance that owns its exact address. amdgpu gives each instance 0x50 bytes of doorbells (sdma_doorbell_range = 20 dwords), which isn't a multiple of 32, so the first 16 bytes of every odd instance's range share a block with the previous instance. KFD puts slots 0-3 of each engine there (slot q at base + (q & 1) * 0x1000 + (q >> 1) * 8). All 128 slots match this rule, and DPX, QPX and CPX use the same layout. The driver-side fix (a 32-byte-aligned doorbell layout) is tracked in ROCM-32598. This PR is the rocSHMEM workaround until then.

Summary

• gpuDoorbellMisrouted() (pure, in anvil_engine_map.hpp) decides from a queue's doorbell address whether a GPU write reaches it.
• After every engine-pinned create, SdmaQueue checks its doorbell. A queue on an unreachable slot is kept parked: it holds the slot but gets no device handles. KFD hands out an engine's first free slot, so the next create on that engine lands on a reachable one. Parked queues are released by disconnect() and at teardown.
• Removes fix(rocshmem): make Anvil SDMA usable on MI300X CPX and QPX, DPX up to two OAMs #12841's even-engine restriction (isUsableSdmaEngine, usableSdmaEngines, preferredUsableEngine, and the odd-id refusal in connect()). Engine selection uses the HSA preferred engine, the OAM map, and the partition fold over all engines. Odd engines are used again, but only through slots 4-7.
• The queue budget counts the reachable slots of every engine (sdmaQueueBudget): 96 on SPX, 48 on DPX, 24 on QPX and 12 on CPX, instead of fix(rocshmem): make Anvil SDMA usable on MI300X CPX and QPX, DPX up to two OAMs #12841's 64/32/16/8 for even engines only.
• As in fix(rocshmem): make Anvil SDMA usable on MI300X CPX and QPX, DPX up to two OAMs #12841, a rejected engine-pinned queue doesn't retry as a generic HSA_QUEUE_SDMA in any mode, because KFD would pick that queue's slot. The connect fails, and fix(rocshmem): agree on the SDMA fallback across node-local ranks #13235 turns that into the agreed IPC fallback.
• Unit tests: gpuDoorbellMisrouted against all 128 measured MI300X slots, page-offset handling, reachable slots per engine, and the budget per mode.

Cost: an odd engine gives up 4 of its 8 slots while this is in place, and those slots stay held by the process. SPX is unaffected, because its HSA-preferred engines are all even.

Known issue: intermittent 16-rank DPX hang with two workgroups per queue

Unrelated to slot choice, and also seen with the even-only revision: in full-node 16-rank DPX alltoall with -w 2, about 1 run in 13 strands one queue with RB_ENABLE = 0 and CONTEXT_STATUS = 0x44 on a reachable slot. A standalone probe reproduces it in SPX without rocSHMEM when a second doorbell arrives while a 256 KiB copy is in flight. It's tracked as ROCM-32719. -w 1 makes it much rarer, and ROCSHMEM_SDMA_ENABLED=0 avoids it.

Separately, SdmaImpl::sdmaDirty is a 64-bit mask indexed by local_pe * numChannels + ch. A full DPX node (16 local PEs) overflows it at more than 4 channels, and a full QPX node (32) at more than 2. The budget or KFD refuses those configurations today.

Test plan

Results were taken at fe53038ed9, before the stack was rebased onto the current #12841 tip. The current head add560d850 (rebased onto develop at d6d2046bef, after #13235 merged; the patch is unchanged from a674a04501) has the same Anvil code apart from #12841's quieter partition-engine log (LOG_TRACE instead of LOG_INFO), and has not been rerun on hardware.

MI300X banff-cyxtera-cx56-5 (8 GPUs, NPS1, gfx942, ROCm 7.2.0, amdgpu 6.16.13, Open MPI 4.1.6). USE_IPC=ON USE_SDMA=ON, functional test 19 (alltoall) with -w 1 -z 64 -s 1048576. Each mode was set with amd-smi set --compute-partition, and the node was left in SPX. "Even only" is an earlier even-engine revision of this stack (ea1c0c075e); #12841's own results at b47b4f7ccf are in its description. Bandwidth is the last row (8 MiB per rank pair).

Mode	Case	Even only (ea1c0c075e)	This PR
SPX (8)	8 ranks, 1 / 2 / 4 / 8 channels, MPI and TCP	pass, 1.55-1.73 TB/s	pass, 1.54-1.72 TB/s; per-pair engine map identical (even engines, nothing parked)
DPX (16)	8 ranks, 1 / 2 channels, MPI and TCP	pass, 384 / 454 GB/s	pass, SDMA kept, 379 / 377 GB/s (TCP 425); engines 0, 1, 2, 5, 6, 7
DPX (16)	16 ranks, 1 / 2 channels, MPI and TCP	pass, 1.05 TB/s / 994 GB/s	pass, SDMA kept, 1.12 / 1.09 TB/s (TCP 1.14 TB/s)
DPX (16)	8 and 16 ranks, 4 and 8 channels	fallback	fallback (pinned rejection), no generic retries
QPX (32)	8 ranks, 1 / 2 channels	pass, 172 GB/s	pass, 173 GB/s (engines 0 and 2, nothing parked)
QPX (32)	32 ranks, any channels	budget refused (16), fallback, 209 GB/s	budget (24) or pinned rejection, fallback, 208 GB/s
CPX (63 seen)	8 ranks, 1 channel, MPI and TCP	pass, engine 0 only, 323 GB/s	pass, engines 0 and 1, 369 GB/s
CPX (63 seen)	8 ranks 2+ channels; 63 ranks any channels	fallback	fallback, 52 / 317-332 GB/s
CPX	2-rank tests 161, 2, 3, 26, 19 on partitions 0 and 1	pass	pass, using engine 1's slots 4-7
all	ROCSHMEM_SDMA_FAIL_CONNECT_LOCAL_RANK=3, ROCSHMEM_SDMA_ENABLED=0, 2-rank tests	pass	pass

Engine choice was checked with BUILD_DEBUG_TRACE_HOST=ON builds, 8 ranks at 1 channel. No run logged a generic-queue retry or an abort.

• rocshmem_unit_tests --gtest_filter='AnvilEngineMap.*:SdmaAgreement.*': 26 passed in every mode (18 AnvilEngineMap, 8 SdmaAgreement)
• mpirun -np 2 rocshmem_unit_tests --gtest_filter='Sdma*' on MI300X SPX: 88 passed, no exclusions (same as even only)
• MI300X DPX, QPX and CPX: the Sdma* suite stops with the same hipMalloc out-of-memory abort after 110, 62 and 38 passing tests, identical to the even-only build and to the fix(rocshmem): agree on the SDMA fallback across node-local ranks #13235 head
• Standalone slot probe: all 16 engines x 8 slots in SPX match gpuDoorbellMisrouted
• RCCL's AnvilOamMapEngine.* and AnvilPciFunctionBus.* cases from fix(rccl): GIN gate and cuMem window registration for MI300X CPX #12742 pass against this PR's anvil_engine_map.hpp (standalone compile)
• MI355X: not rerun with slot parking (the engine-0-only build passed in CPX). The probe there showed the same engine-1 failure, which matches the shared layout.
• MI300X DPX 16 ranks with -w 2: intermittent hang, ROCM-32719 (known issue above)

Made with Cursor

[amd-argus]

ARGUS

Bottom line: Worth settling before approval: none of the three behaviour changes has a test, though anvil_engine_map.hpp already holds the pure-helper seam that would give them one.

_{Comment @argus review to re-run, or @argus re-review after pushing.}

[amd-argus]

ARGUS

Findings

3 finding(s) that could not be posted as inline comments

• projects/rocshmem/tests/unit_tests/anvil_engine_map_gtest.cpp:53 (Low) usableSdmaEngines' round-up is never exercised because every total the suite passes is even Every total this suite passes to usableSdmaEngines is even (16, 8, 4, 2, 0), so the + 1 round-up at anvil_engine_map.hpp:43 is never exercised and a plain numSdmaEnginesTotal / 2 would keep the file green. That round-up is what guarantees usableSdmaEngines is at least 1 whenever oamMapEngineNeedsFold returns true, which is what keeps foldOamMapEngine's modulo off zero, so it is worth one assertion of its own.
• projects/rocshmem/src/sdma/anvil.cpp:101 (Low) describeSelection still derives preferredEngine with __builtin_ctz, so the log names a different engine than the one selected describeSelection in projects/rocshmem/src/sdma/anvil.cpp, line 101, still derives preferredEngine with __builtin_ctz(s.preferredMask), but getSdmaEngineId at anvil.cpp:716 now selects with preferredUsableEngine. The two disagree whenever the mask's lowest set bit is odd: for mask 0x6 the log prints preferredEngine=1 while the queue is pinned to engine 2. Both consumers of this string are failure logs (the engine-pinned create error at anvil.cpp:230 and the missing xGMI physical id error at anvil.cpp:653), so the one diagnostic that explains an engine choice names an engine that was never used. Calling preferredUsableEngine(s.preferredMask) at line 101 would keep the log and the selection on the same helper.
• projects/rocshmem/tests/unit_tests/anvil_engine_map_gtest.cpp:151 (Low) Three retained budget cases still pass a raw engine total into the parameter this PR redefined as the usable even-engine count This PR redefined queueBudgetExceeded's third parameter from numEnginesTotal to the usable (even) engine count (anvil_engine_map.hpp:97-100) and updated QueueBudgetRefusesOverSubscription to a literal 2, but three cases still pass kCpxTotal there: anvil_engine_map_gtest.cpp lines 151, 155 and 156. The assertions do not change, since 2 is a valid value under either reading, so the file simply documents two meanings for one parameter. Passing usableSdmaEngines(kCpxTotal) or a literal 1 at those three lines would make them say what the parameter now means.

Bottom line: Looks approval ready. The one open point is from the previous round: the parity fix at anvil.cpp:716 has no test behind it, so reverting preferredUsableEngine to __builtin_ctz keeps the suite green.

_{Comment @argus review to re-run, or @argus re-review after pushing.}

[amd-argus]

ARGUS

Bottom line: LGTM.

_{Comment @argus review to re-run, or @argus re-review after pushing.}

[amd-argus]

ARGUS

Bottom line: LGTM.

_{Comment @argus review to re-run, or @argus re-review after pushing.}

[dlamd1dai]

Thanks for the three findings in the ea1c0c075e review that could not be anchored inline. The final rework (fe53038ed9, now restacked as 92bc7a33bd) dropped the even-engine restriction, so none of the three applies to the current head:

• usableSdmaEngines rounding a non-power-of-two total up: the helper is gone. Selection uses all engines again (HSA preferred engine, OAM map, then the partition fold over numSdmaEnginesTotal).
• describeSelection logging a different preferred engine from the one selection uses: preferredUsableEngine is gone. describeSelection and getSdmaEngineId both take the preferred engine as __builtin_ctz(preferredMask), so the log and the selection agree.
• queueBudgetExceeded parameter meaning two different things: it now takes the budget directly, computed by sdmaQueueBudget(total, queuesPerEngine) (96/48/24/12 GPU-reachable slots for SPX/DPX/QPX/CPX), and the tests pass the same value.

[dlamd1dai]

@abouteiller, this is stacked on #13235, so please review that one first; this PR adds a single commit on top of it. All Argus threads are addressed.

It replaces #12841's even-engine-only restriction with a slot-level rule: only doorbell slots 0-3 of odd SDMA engines are unreachable from GPU-written doorbells (ROCM-32598), so odd engines carry traffic again. On MI300X it passed alltoall in SPX, DPX and CPX against the even-only build; CPX bandwidth rose from 323 to 369 GB/s, and SPX is unchanged. The results were taken at fe53038ed9; the current head has the same Anvil code but has not been rerun on hardware. Details and a known, unrelated DPX hang (ROCM-32719) are in the description.

The red extra(rccl) is the same RCCL topo_expl link error seen on #13235, in code this PR does not touch.

Thank you!

[dlamd1dai]

Converting to draft while the driver team looks at ROCM-32598.

#12841 has merged with the even-engine restriction, so the CPX/DPX alltoall and put hangs are already fixed on develop. This PR is now an optimization: it lets odd engines carry traffic through their GPU-reachable slots 4-7. On MI300X that gives CPX 8 ranks 323 → 369 GB/s and DPX 16 ranks 1.05 → 1.12 TB/s, while SPX and QPX are unchanged. Every configuration that fell back to IPC with even-only still falls back.

On Oct 22 the driver team (ROCM-32598) said they will investigate and update within a week. If they commit to a 32-byte-aligned SDMA doorbell layout, I'll close this and keep even-only, which stays correct after that change. If not, this is what's left before I mark it ready:

• Rebase onto develop. This PR doesn't touch any fix(rocshmem): agree on the SDMA fallback across node-local ranks #13235 file and merges cleanly, but it should still land after fix(rocshmem): agree on the SDMA fallback across node-local ranks #13235: odd engines have only 4 usable slots, so a pinned create can be rejected where even-only succeeds, and without fix(rocshmem): agree on the SDMA fallback across node-local ranks #13235 that rejection aborts.
• Limit gpuDoorbellMisrouted() to gfx942/gfx950, where the layout was measured, and add an environment variable to turn slot parking off. The rule hard-codes the MI300 doorbell layout (base 0x800, stride 0x50, 32-byte routing), and RCCL GIN compiles the same anvil.cpp/anvil_engine_map.hpp.
• Explain or fix the DPX 8-rank 2-channel drop (454 → 377 GB/s compared with even-only).
• Rerun at the current head on MI300X SPX/DPX/QPX/CPX, MI355X CPX, and RCCL GIN Anvil in DPX/CPX.
• Retitle as an optimization rather than a fix.

The red precheckin(rocshmem) is the gda_mlx5 test failure on the gfx90a node, seen on most recent rocSHMEM PRs. extra(rccl) is the RCCL link error also seen on #13235.

@abouteiller no need to review this one yet. #13235 is the one that's ready.

[therock-pr-bot]

✅ All Policy Checks Passed

Check	Status	Details
📝 PR Description	✅ Pass	—
⛔ Forbidden Files	✅ Pass	—
🧪 Unit Test	✅ Pass	—
🚫 Draft PR	🔜 To Be Enabled	—
🚩 Feature Flag	🔜 To Be Enabled	—
📊 Code Coverage	🔜 To Be Enabled	—

🎉 All policy checks passed!

📖 Need help? See the Policy FAQ for details on every check and how to fix failures.

🙋 Wish to Override Policy?

[math-ci]

Resource usage report (4gfx1201)

Dashboard artifact

[math-ci]

Resource usage report (8gfx942)

Dashboard artifact