[JinghanHuang]

Motivation

Two problems with GIN over the proxy on AINIC hosts:

1. With NCCL_NET=ROCM-IB (what AMD's AINIC test configs set) or NCCL_NET unset, the data path uses the IB-CAST NET plugin (plugin/net.cc), but the internal RMA backend that carries GIN proxy traffic is the generic IB one, without IB-CAST's AINIC handling. plugin/rma.cc picked IB-CAST only for a literal NCCL_NET=IB-CAST.
2. On IB-CAST, all GIN and RMA proxy contexts connect with channel id 0, so on AINIC they share one uDMA pipeline, which caps GIN proxy bandwidth per NIC.

Together: GIN proxy puts on 2 MI355X nodes go from 29.0 to 46.6 GB/s per rank at 64 MiB (results below).

Technical Details

Two commits:

• IB-CAST RMA backend whenever the data path uses IB-CAST (plugin/rma.cc). The backend is IB-CAST for rcclCanonicalNetName(NCCL_NET) == "IB-CAST" (so IB-CAST and ROCM-IB), or for an unset NCCL_NET on an AINIC host, the same values that select the IB-CAST NET plugin. It is plugin/net.cc's whole rule, so a request for an external NET plugin keeps the generic IB backend, as it keeps the internal NET plugin on generic IB. The generic IB backend follows IB-CAST as a fallback, so if IB-CAST cannot initialize (for example with an older libionic), RMA and GIN run on generic IB as before rather than on the socket backend. NCCL_NET=IB opts out. The include/net.h note on rcclCanonicalNetName now lists rma.cc as a plugin-selection user. An INFO line names the chosen backend. ROCM-27070: honor NCCL_GIN_ENABLE=0 and select ib-cast GIN backend on AINIC #9085 made the same choice for GIN in plugin/gin.cc before the GIN/RMA split.
• One uDMA pipeline per context (transport/net_ib_cast/gin.cc). Each IB-CAST RMA context passes its own channel id to connect and accept, from a process-wide counter counting down from MAXCHANNELS - 1 and wrapping at MAXCHANNELS / 2. Under the default RCCL_IB_UDMA_POLICY (ChannelAffinity, fix(rccl): fixed udma balancing for p2p comms (develop) #12948) consecutive contexts then land on different uDMA pipelines; with the driver-default policy or QP sharing the id does not pick the pipeline. NET channels use the low ids and reach these only when the channel count is raised past MAXCHANNELS / 2; a NET channel and a context then share a pipeline (bandwidth, not correctness). The id only selects the local pipeline and telemetry bucket and is never sent to the peer, so ranks need not agree on it. connect.cc reads it only in AINIC mode, so other hosts are unaffected. With RCCL_TELEMETRY_ENABLE=1, AINIC devices that carry GIN or RMA proxy contexts now report per-channel ids instead of null, and num_channels (highest id + 1) reads up to MAXCHANNELS.
• Host tests for both (the new RmaInternalBackendTest suite is registered in test/test_categories_micro.yaml, so the CTest categories run it); CHANGELOG entries under Resolved issues. A third commit addresses the first review: the IB fallback, plugin/net.cc's external NET plugin rule, the channel id wrap test and shorter comments.

Issue Tracking

Related: #9085 (IB-CAST GIN backend on AINIC), #12948 (UDMA balancing).

Test Plan

1. Host tests (CPU only), with ROCm 7.0.2.2 headers: rccl-UnitTestsMicro (RmaPluginInitTest.*, RmaInternalBackendTest.*) and rccl-UnitTestsMicroTransportNetIbCastGin, with and without the source changes.
2. GPU, 2 nodes with 8x MI355X and AMD Pensando Pollara 400 (AINIC) NICs each, 16 ranks, one job, so develop and this branch run on the same two hosts. A small GIN proxy benchmark: every rank puts to its rail peer on the other node over 4 GIN contexts, the receiver checks every byte, median of 50 rounds per size. NCCL_GIN_TYPE=2 NCCL_CUMEM_ENABLE=1 NCCL_DMABUF_ENABLE=1 NCCL_IB_HCA=ionic, with the hosts' default NCCL_NET (which resolves to ROCM-IB there) and with NCCL_NET=IB-CAST. Warm-up plus 3 repeats, arm order rotated each repeat.

Test Result

1. With the changes: rccl-UnitTestsMicroTransportNetIbCastGin 50/50, RMA plugin tests 14/14. With develop's rma.cc (tests kept), exactly the five RmaInternalBackendTest cases that expect IB-CAST with its IB fallback, or the generic IB backend under an external NET plugin, fail; the two plain IB cases pass both ways. With develop's gin.cc, CreateContext_EachContextPassesItsOwnTopRangeChannelId fails, and removing the % (MAXCHANNELS / 2) wrap fails exactly CreateContext_ChannelIdsWrapWithinTheTopHalf.
2. No faults or data mismatches in any arm. Per-rank GB/s, median of 48 samples (16 ranks x 3 repeats); the per-run medians vary by at most 0.3 GB/s at 64 MiB and 2.1 GB/s at 4 MiB:

NCCL_NET	build	RMA backend for GIN	4 MiB	16 MiB	64 MiB
default (ROCM-IB)	develop	generic IB	20.9	27.0	29.0
default (ROCM-IB)	this branch	IB-CAST	28.4	41.4	46.6
IB-CAST	develop	IB-CAST	21.2	27.9	30.2
IB-CAST	this branch	IB-CAST	28.1	41.3	46.6

The first commit moves the default configuration onto IB-CAST (rows 1 to 2; rows 1 and 3 show that alone is worth about 4% at 64 MiB, 3% at 16 MiB and within noise at 4 MiB), and the second spreads the contexts over the uDMA pipelines (rows 3 to 4, +54% at 64 MiB). The backend is confirmed by the logs: RMA/Plugin: internal RMA backend IB-CAST and IbCastConnect ... isRMA=1 on this branch, generic ncclIbConnect for GIN on develop with the default NCCL_NET.

The GIN proxy arms ran with NCCL_DMABUF_ENABLE=1, as the GIN how-to (docs/how-to/device-api-gin.rst) requires.

Submission Checklist

• Look over the contributing guidelines at https://github.com/ROCm/rocm-systems/blob/develop/CONTRIBUTING.md.

[therock-pr-bot]

✅ All Policy Checks Passed

Check	Status	Details
📝 PR Description	✅ Pass	—
⛔ Forbidden Files	✅ Pass	—
🧪 Unit Test	✅ Pass	—
🚫 Draft PR	🔜 To Be Enabled	—
🚩 Feature Flag	🔜 To Be Enabled	—
📊 Code Coverage	🔜 To Be Enabled	—

🎉 All policy checks passed!

📖 Need help? See the Policy FAQ for details on every check and how to fix failures.

🙋 Wish to Override Policy?

[amd-argus]

ARGUS

Findings

3 more finding(s), not posted inline

• projects/rccl/src/transport/net_ib_cast/gin.cc:464 (Low) Top-range chId also selects the telemetry bucket, so num_channels reads 256 on AINIC jobs with a GIN context even when few NET channels run; consider reporting GIN/RMA ids separately.
• projects/rccl/test/host/transport/net_ib_cast/net-ib-cast-gin-test.cc:671 (Low) No test fails if the % (MAXCHANNELS / 2) wrap at gin.cc:464 is dropped; three contexts never reach the wrap, which keeps chId inside nccl_channel_ud_map.
• projects/rccl/test/host/transport/net_ib_cast/net-ib-cast-gin-test.cc:634 (Nit) accept_ hook line 631 is 121 columns, past the 120 ColumnLimit; wrap it like connect_.

Bottom line: Worth settling before approval: IB-CAST becomes the default internal RMA backend on AINIC with no opt-in, and if its init fails GIN falls back to the socket proxy instead of generic IB; the deliberate NCCL_NET_PLUGIN split is also untested.

_{Comment @argus review to re-run, or @argus re-review after pushing.}

[JinghanHuang]

Thanks for the review. On the three findings in the summary:

• Telemetry (gin.cc:464): leaving it as is. The channel id is what selects the uDMA pipeline, and telemetry buckets by it, so GIN/RMA contexts now get per-channel counters instead of none; num_channels reading up to MAXCHANNELS on such devices is noted in the description. Reporting them separately would change the telemetry format, which I'd rather do in its own PR if wanted.
• Wrap test: added CreateContext_ChannelIdsWrapWithinTheTopHalf; removing the % (MAXCHANNELS / 2) makes exactly that test fail.
• Line width: the accept_ hook is wrapped like connect_.

[JinghanHuang]

@argus re-review

[amd-argus]

@JinghanHuang I can't trigger on this PR yet -- your association here reads as CONTRIBUTOR, and the current policy is members. Ask a maintainer to add JinghanHuang to ARGUS's allowed-users list.

[amd-argus]

ARGUS

Bottom line: Nothing new worth raising; still open from earlier: rma.cc:235. Worth clearing before approval.

_{Comment @argus review to re-run, or @argus re-review after pushing.}