test(net): add v11 plugin compat lazy-init unit test by nikitaxgusev · Pull Request #8633 · ROCm/rocm-systems

nikitaxgusev · 2026-07-15T18:10:44Z

Motivation

The NCCL v11 network-plugin compatibility layer must initialize lazily:
getNcclNet_v11() / getNcclCollNet_v11() should populate only the init
entry point of the public ncclNet_t / ncclCollNet_t vtable, and defer
wiring up every other entry point (devices, getProperties, listen, …)
until the plugin's init() callback has actually run. A v11 plugin is only
allowed to expose its per-device entry points after init(); populating the
vtable eagerly (the old direct-cast pattern) meant RCCL could call into an
uninitialized plugin.

The fix already landed on develop
(projects/rccl/src/plugin/net/net_v11.cc), but there was no test guarding
the behavior. This PR adds that regression test and wires it into CI.

Technical Details

New test projects/rccl/test/PluginCompatV11_test.cpp:
- Defines an in-process mock v11 net/collnet plugin and exports it under the
  exact symbols the compat layer resolves via dlsym
  (ncclNetPlugin_v11 / ncclCollNetPlugin_v11), so no GPU or real
  network plugin is required.
- Hands the compat layer a dlopen(NULL) handle and asserts the lazy-init
  contract:
  1. getNcclNet_v11() returns a non-null ncclNet_t*.
  2. Before init(): only ->init is set; every other entry point is
    nullptr.
  3. After ->init() succeeds: the remaining entry points become non-null.
- Covers both the net and collnet compat layers.
projects/rccl/test/CMakeLists.txt:
- Adds PluginCompatV11_test.cpp to the Debug-only
  rccl-UnitTestsFixturesDebug target — the test calls internal symbols
  (getNcclNet_v11) that have hidden visibility in Release builds, so it
  cannot live in the always-built rccl-UnitTests (that would break the
  Release link).
- Exports only the two mock plugin symbols via
  --export-dynamic-symbol=ncclNetPlugin_v11 / =ncclCollNetPlugin_v11 (rather
  than the broad -rdynamic) so dlsym(dlopen(NULL)) can find them.
projects/rccl/tools/scripts/test_runner/configs/mi300x_mellanox_ib.json:
- Registers the two tests as a plugin_compat_v11 gtest group
  (rccl-UnitTestsFixturesDebug, 1 GPU) so they run in the MI300X + Mellanox
  IB config.

JIRA ID

JIRA ID : AICOMRCCL-1532

Test Plan

Built and ran on an AMD MI350X (gfx950) node (Debug build):

bash install.sh -l -t --debug --prefix ~/rccl-install-1532
./build/debug/test/rccl-UnitTestsFixturesDebug --gtest_filter=PluginCompatV11*

Validity of the test was confirmed by temporarily reverting
getNcclNet_v11() to the old eager/direct-cast pattern and re-running. The
tests are also registered in the mi300x_mellanox_ib test-runner config so
they execute in CI.

Test Result

Fixed (current) code — PASS:

[==========] Running 2 tests from 1 test suite.
[ RUN      ] PluginCompatV11.NetLazyInitialization
[       OK ] PluginCompatV11.NetLazyInitialization (0 ms)
[ RUN      ] PluginCompatV11.CollNetLazyInitialization
[       OK ] PluginCompatV11.CollNetLazyInitialization (0 ms)
[  PASSED  ] 2 tests.

With getNcclNet_v11() reverted to the eager pattern — FAIL (the test
catches the bug), failing exactly on the pre-init() null checks
(devices / getProperties / listen / … set before init()).
Restoring the fix returns the suite to PASS.

Submission Checklist

Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests.

therock-pr-bot · 2026-07-15T18:12:14Z

✅ All Policy Checks Passed

Check	Status	Details
🌿 Branch Name	✅ Pass	—
📝 PR Title/Description	✅ Pass	—
⛔ Forbidden Files	✅ Pass	—
🧪 Unit Test	✅ Pass	PR does not contain code files — Unit Test auto-passed
🚫 Draft PR	🔜 To Be Enabled	—
🚩 Feature Flag	🔜 To Be Enabled	—
📊 Code Coverage	🔜 To Be Enabled	—

🎉 All policy checks passed!

📖 Need help? See the Policy FAQ for details on every check and how to fix failures.

therock-pr-bot · 2026-07-15T18:12:16Z

🚫 Please fix the failed policies before requesting reviews.

The following policy checks failed:

❌ Unit Test

The Not ready to Review label has been added to this PR.
Once all policies pass, the label will be removed automatically.

Copilot

Pull request overview

Adds a new regression unit test to guard the NCCL v11 network-plugin compatibility layer’s lazy-initialization contract, ensuring only ->init is populated before initialization and all other vtable entry points are deferred until after init() succeeds.

Changes:

Introduces an in-process mock v11 net/collnet plugin and validates pre-/post-init() vtable wiring behavior via GoogleTest.
Updates rccl-UnitTests build to include the new test source and exports executable symbols for dlsym(dlopen(NULL), ...) via -rdynamic.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
projects/rccl/test/PluginCompatV11Tests.cpp	Adds regression tests for v11 net/collnet compat-layer lazy initialization using in-process exported mock plugin symbols.
projects/rccl/test/CMakeLists.txt	Includes the new test in `rccl-UnitTests` and adds `-rdynamic` so `dlsym` can resolve in-process plugin symbols.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

nikitaxgusev · 2026-07-16T15:14:51Z

%RCCL_TEST_CONFIG
mi300x_mellanox_ib.json

nikitaxgusev · 2026-07-16T15:15:08Z

%RCCL_TEST_CONFIG
mi300x_mellanox_ib.json

nikitaxgusev · 2026-07-17T12:09:15Z

%RCCL_TEST_CONFIG
mi300x_mellanox_ib.json
SUITE=plugin_compat_v11
TEST_NAME=PluginCompatV11.*

nikitaxgusev · 2026-07-17T12:13:28Z

%RCCL_TEST_CONFIG
mi300x_mellanox_ib.json
SUITE=plugin_compat_v11
TEST_NAME=PluginCompatV11.*

nikitaxgusev · 2026-07-17T12:21:55Z

%RCCL_TEST_CONFIG
mi300x_mellanox_ib.json
SUITE=plugin_compat_v11
TEST_NAME=PluginCompatV11.*

nikitaxgusev · 2026-07-17T12:31:39Z

%RCCL_TEST_CONFIG
mi300x_mellanox_ib.json
SUITE=plugin_compat_v11
TEST_NAME=PluginCompatV11.*

nikitaxgusev · 2026-07-21T10:25:20Z

Checked manually

export LD_LIBRARY_PATH=~/rocm-systems-1532/projects/rccl/build/debug:$LD_LIBRARY_PATH
python3 test_runner.py --config configs/mi300x_mellanox_ib.json \
  --suite-name 'Plugin Compat v11 - Lazy Init' \
  --no-build --skip-mpi-check --verbose
----------------------
rccl-UnitTestsFixturesDebug --gtest_filter=PluginCompatV11.NetLazyInitialization      --gtest_output=json:...
rccl-UnitTestsFixturesDebug --gtest_filter=PluginCompatV11.CollNetLazyInitialization  --gtest_output=json:...
----------------------
NCCL INFO NET/Plugin: Loaded net plugin mock_v11_net (v11)
[       OK ] PluginCompatV11.NetLazyInitialization (221 ms)
NCCL INFO NET/Plugin: Loaded collnet plugin mock_v11_collnet (v11)
[       OK ] PluginCompatV11.CollNetLazyInitialization (232 ms)
Total Tests:   2
Passed:        2
Failed:        0
Skipped:       0
Timeout:       0
Exiting: Test run completed successfully   (exit=0)

github-actions · 2026-07-21T18:57:20Z

RCCL Perf-Regression Gate: ✅ PASS

Mode: detect (reference vs candidate)
Thresholds (per-tier, calibrated): small 17.2% · mid 12.3% · large 12.9%
Keys compared: 0 · Confirmed regressions: 0 · Inconclusive: 0

Per-collective breakdown

group	keys	regressions	inconclusive
all_gather_perf-d=bfloat16-default	0	0	0
all_gather_perf-d=float-default	0	0	0
all_reduce_perf-d=bfloat16-default	0	0	0
all_reduce_perf-d=float-default	0	0	0
broadcast_perf-d=bfloat16-default	0	0	0
broadcast_perf-d=float-default	0	0	0
reduce_scatter_perf-d=bfloat16-default	0	0	0
reduce_scatter_perf-d=float-default	0	0	0

Add PluginCompatV11_test.cpp, an in-process regression test for the NCCL v11 network-plugin compat layer's lazy initialization: getNcclNet_v11() / getNcclCollNet_v11() must populate only ncclNet.init and leave the remaining entry points null until init() runs. The test defines an in-process mock v11 net/collnet plugin exported under the symbols the compat layer resolves via dlsym, so no GPU or real plugin is required. It lives in the Debug-only rccl-UnitTestsFixturesDebug target since it calls internal symbols (getNcclNet_v11) that are hidden in Release builds, and exports the two mock symbols via --export-dynamic-symbol. Resolves AICOMRCCL-1532

Register the two PluginCompatV11 tests as a plugin_compat_v11 gtest group (rccl-UnitTestsFixturesDebug, 1 GPU) so they run in the MI300X + Mellanox IB test-runner config.

The compat-layer mock symbols are exported via --export-dynamic-symbol, not -rdynamic; update the comment to match.

The plugin_compat_v11 test configuration had no matching test_suites entry, so the test_runner never enumerated the PluginCompatV11 lazy-init tests. Add the suite entry (Debug-only) so CI actually runs them.

## Motivation The NCCL v11 network-plugin compatibility layer must initialize **lazily**: `getNcclNet_v11()` / `getNcclCollNet_v11()` should populate only the `init` entry point of the public `ncclNet_t` / `ncclCollNet_t` vtable, and defer wiring up every other entry point (`devices`, `getProperties`, `listen`, …) until the plugin's `init()` callback has actually run. A v11 plugin is only allowed to expose its per-device entry points after `init()`; populating the vtable eagerly (the old direct-cast pattern) meant RCCL could call into an uninitialized plugin. The fix already landed on `develop` (`projects/rccl/src/plugin/net/net_v11.cc`), but there was no test guarding the behavior. This PR adds that regression test and wires it into CI. ## Technical Details - New test `projects/rccl/test/PluginCompatV11_test.cpp`: - Defines an in-process mock v11 net/collnet plugin and exports it under the exact symbols the compat layer resolves via `dlsym` (`ncclNetPlugin_v11` / `ncclCollNetPlugin_v11`), so **no GPU or real network plugin is required**. - Hands the compat layer a `dlopen(NULL)` handle and asserts the lazy-init contract: 1. `getNcclNet_v11()` returns a non-null `ncclNet_t*`. 2. Before `init()`: only `->init` is set; every other entry point is `nullptr`. 3. After `->init()` succeeds: the remaining entry points become non-null. - Covers both the net and collnet compat layers. - `projects/rccl/test/CMakeLists.txt`: - Adds `PluginCompatV11_test.cpp` to the **Debug-only** `rccl-UnitTestsFixturesDebug` target — the test calls internal symbols (`getNcclNet_v11`) that have hidden visibility in Release builds, so it cannot live in the always-built `rccl-UnitTests` (that would break the Release link). - Exports only the two mock plugin symbols via `--export-dynamic-symbol=ncclNetPlugin_v11 / =ncclCollNetPlugin_v11` (rather than the broad `-rdynamic`) so `dlsym`(`dlopen(NULL)`) can find them. - `projects/rccl/tools/scripts/test_runner/configs/mi300x_mellanox_ib.json`: - Registers the two tests as a `plugin_compat_v11` gtest group (`rccl-UnitTestsFixturesDebug`, 1 GPU) so they run in the MI300X + Mellanox IB config. ## JIRA ID JIRA ID : AICOMRCCL-1532 ## Test Plan Built and ran on an AMD MI350X (gfx950) node (Debug build): ``` bash install.sh -l -t --debug --prefix ~/rccl-install-1532 ./build/debug/test/rccl-UnitTestsFixturesDebug --gtest_filter=PluginCompatV11* ``` Validity of the test was confirmed by temporarily reverting `getNcclNet_v11()` to the old eager/direct-cast pattern and re-running. The tests are also registered in the `mi300x_mellanox_ib` test-runner config so they execute in CI. ## Test Result - Fixed (current) code — **PASS**: ``` [==========] Running 2 tests from 1 test suite. [ RUN ] PluginCompatV11.NetLazyInitialization [ OK ] PluginCompatV11.NetLazyInitialization (0 ms) [ RUN ] PluginCompatV11.CollNetLazyInitialization [ OK ] PluginCompatV11.CollNetLazyInitialization (0 ms) [ PASSED ] 2 tests. ``` - With `getNcclNet_v11()` reverted to the eager pattern — **FAIL** (the test catches the bug), failing exactly on the pre-`init()` null checks (`devices` / `getProperties` / `listen` / … set before `init()`). Restoring the fix returns the suite to PASS. ## Submission Checklist - [x] Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests.

Copilot AI review requested due to automatic review settings July 15, 2026 18:10

nikitaxgusev requested a review from a team as a code owner July 15, 2026 18:10

github-actions Bot added the project: rccl label Jul 15, 2026

Copilot started reviewing on behalf of nikitaxgusev July 15, 2026 18:11

View session

therock-pr-bot Bot added the Not ready to Review label Jul 15, 2026

Copilot AI reviewed Jul 15, 2026

View reviewed changes

Comment thread projects/rccl/test/CMakeLists.txt Resolved

nikitaxgusev force-pushed the users/nikitaxgusev/AICOMRCCL-1532-plugin-v11-test branch from ffaa5a2 to c884648 Compare July 15, 2026 18:20

systems-assistant Bot added the organization: ROCm label Jul 15, 2026

nikitaxgusev force-pushed the users/nikitaxgusev/AICOMRCCL-1532-plugin-v11-test branch from c884648 to 983952e Compare July 15, 2026 18:52

amd-arozanov reviewed Jul 16, 2026

View reviewed changes

Comment thread projects/rccl/test/PluginCompatV11_test.cpp Resolved

nikitaxgusev removed the Not ready to Review label Jul 16, 2026

amd-arozanov approved these changes Jul 16, 2026

View reviewed changes

thomas-huber approved these changes Jul 16, 2026

View reviewed changes

nikitaxgusev added the ci: code-coverage label Jul 16, 2026

nikitaxgusev added ci: code-coverage and removed ci: code-coverage labels Jul 17, 2026

nikitaxgusev force-pushed the users/nikitaxgusev/AICOMRCCL-1532-plugin-v11-test branch 2 times, most recently from 278bc7a to a01bb82 Compare July 17, 2026 12:30

alexander-sannikov approved these changes Jul 17, 2026

View reviewed changes

nikitaxgusev added the ainic-test label Jul 22, 2026

nikitaxgusev removed the ainic-test label Jul 22, 2026

Nikita Gusev added 3 commits July 22, 2026 09:44

test(net): run v11 compat lazy-init test in mi300x_mellanox_ib CI

6b550e0

Register the two PluginCompatV11 tests as a plugin_compat_v11 gtest group (rccl-UnitTestsFixturesDebug, 1 GPU) so they run in the MI300X + Mellanox IB test-runner config.

test(net): fix stale -rdynamic reference in mock symbol comment

b4e2834

The compat-layer mock symbols are exported via --export-dynamic-symbol, not -rdynamic; update the comment to match.

nikitaxgusev force-pushed the users/nikitaxgusev/AICOMRCCL-1532-plugin-v11-test branch from 64a8798 to 1d7490c Compare July 22, 2026 09:44

nikitaxgusev added the ainic-test label Jul 22, 2026

test(net): register plugin_compat_v11 suite in mi300x_mellanox_ib CI

9448042

The plugin_compat_v11 test configuration had no matching test_suites entry, so the test_runner never enumerated the PluginCompatV11 lazy-init tests. Add the suite entry (Debug-only) so CI actually runs them.

nikitaxgusev force-pushed the users/nikitaxgusev/AICOMRCCL-1532-plugin-v11-test branch from 1d7490c to 9448042 Compare July 24, 2026 09:22

nikitaxgusev merged commit c7c3eb0 into develop Jul 24, 2026
28 of 36 checks passed

nikitaxgusev deleted the users/nikitaxgusev/AICOMRCCL-1532-plugin-v11-test branch July 24, 2026 18:57

nikitaxgusev mentioned this pull request Jul 24, 2026

ROCM-27070: honor NCCL_GIN_ENABLE=0 and select ib-cast GIN backend on AINIC #9085

Merged

1 task done

Navigation Menu

Repository navigation

test(net): add v11 plugin compat lazy-init unit test - #8633

test(net): add v11 plugin compat lazy-init unit test#8633
nikitaxgusev merged 4 commits into
developfrom
users/nikitaxgusev/AICOMRCCL-1532-plugin-v11-test

nikitaxgusev commented Jul 15, 2026 •

edited

Loading

Uh oh!

therock-pr-bot Bot commented Jul 15, 2026 •

edited

Loading

Uh oh!

therock-pr-bot Bot commented Jul 15, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

nikitaxgusev commented Jul 16, 2026

Uh oh!

nikitaxgusev commented Jul 16, 2026

Uh oh!

nikitaxgusev commented Jul 17, 2026

Uh oh!

nikitaxgusev commented Jul 17, 2026

Uh oh!

nikitaxgusev commented Jul 17, 2026

Uh oh!

nikitaxgusev commented Jul 17, 2026

Uh oh!

nikitaxgusev commented Jul 21, 2026

Uh oh!

github-actions Bot commented Jul 21, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

nikitaxgusev commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Technical Details

JIRA ID

Test Plan

Test Result

Submission Checklist

Uh oh!

therock-pr-bot Bot commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ All Policy Checks Passed

Uh oh!

therock-pr-bot Bot commented Jul 15, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

nikitaxgusev commented Jul 16, 2026

Uh oh!

nikitaxgusev commented Jul 16, 2026

Uh oh!

nikitaxgusev commented Jul 17, 2026

Uh oh!

nikitaxgusev commented Jul 17, 2026

Uh oh!

nikitaxgusev commented Jul 17, 2026

Uh oh!

nikitaxgusev commented Jul 17, 2026

Uh oh!

nikitaxgusev commented Jul 21, 2026

Uh oh!

github-actions Bot commented Jul 21, 2026

RCCL Perf-Regression Gate: ✅ PASS

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

nikitaxgusev commented Jul 15, 2026 •

edited

Loading

therock-pr-bot Bot commented Jul 15, 2026 •

edited

Loading