[https://nvbugs/5923949][fix] Improve NCCL library load stability

[nv-lschneider]

Summary by CodeRabbit

• New Features

  • Added runtime verification for NCCL window buffer support with clearer fallback behavior when unsupported.

• Refactor

  • Replaced dynamic NCCL symbol loading with streamlined static allocation path for NCCL 2.28+.
  • Updated API to use safer memory management patterns for NCCL communicators.

• Chores

  • Updated unit tests to align with revised NCCL utility interfaces.

This also fixes: 5803120

Description

Symptom of mismatched NCCL versions at runtime was a segfault.
New behavior, graceful detection and skipping registering windows.

This PR enables stability of using NCCL symmetric related functions.

1. It removes the use of the NCCLHelper to dynamically load nccl functions from a runtime library. Instead nccl provided macros and runtime checks are used to ensure compatibility.
2. Use of shared ptr improved to avoid dead pointer access.

Test Coverage

Is covered in existing

PR Checklist

Please review the following before submitting your PR:

• PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

• PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

• Test cases are provided for new code paths (see test instructions)

• Any new dependencies have been scanned for license and vulnerabilities

• CODEOWNERS updated if ownership changes

• Documentation updated as needed

• Update tava architecture diagram if there is a significant design change in PR.

• The reviewers assigned automatically/manually are appropriate for the PR.

• Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

[coderabbitai]

📝 Walkthrough

Walkthrough

Replace NCCLHelper-based dynamic symbol loading with NCCLWindowAllocator-based implementation gated by NCCL 2.28+ runtime checks. Migrate API signatures from raw ncclComm_t to std::shared_ptr<ncclComm_t> for safer resource management across header, implementation, and test files.

Changes

Cohort / File(s)	Summary
NCCL Utilities Refactoring cpp/tensorrt_llm/common/ncclUtils.h, cpp/tensorrt_llm/common/ncclUtils.cpp	Removed NCCLHelper dynamic library loader; introduced runtime NCCL version checks (isNcclWindowSupported). Replaced NCCLHelper with NCCLWindowAllocator singleton. Migrated ScopedNCCLWindowBuffer and createNCCLWindowTensor to accept std::shared_ptr<ncclComm_t> instead of raw ncclComm_t. Updated allocator/deleter interactions to dereference shared_ptr.
Consumer Code cpp/tensorrt_llm/thop/allreduceOp.cpp	Updated createNCCLWindowTensor calls to pass rawComm instead of local comm variable for symmetric NCCL path window tensor creation.
Unit Tests cpp/tests/unit_tests/multi_gpu/ncclUtilsTest.cpp	Replaced runtime symmetric memory check with isNcclWindowSupported(). Updated test calls to pass shared_ptr<ncclComm_t> (mComm) instead of dereferenced raw pointer (*mComm) to ScopedNCCLWindowBuffer and createNCCLWindowTensor.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

🚥 Pre-merge checks | ✅ 1 | ❌ 2

❌ Failed checks (1 warning, 1 inconclusive)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 15.79% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.
Description check	❓ Inconclusive	PR description follows the template structure with title, description, test coverage, and checklist sections, though some details are incomplete.	Clarify which specific existing tests provide coverage and expand the description to detail the shared_ptr lifetime improvements mentioned in the objectives.

✅ Passed checks (1 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title accurately summarizes the main change: removing dynamic NCCL library loading (NCCLHelper) and replacing it with NCCL-provided macros and runtime checks to improve stability.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@cpp/tensorrt_llm/common/ncclUtils.cpp`:
- Around line 225-245: The runtime check in NCCLWindowAllocator::requestBuffer
uses ncclGetVersion and wrongly targets NCCL 2.28.0; update the minimum required
version to 2.27.3 and, more importantly, remove hard references to symbols that
may not exist at link time (ncclMemAlloc, ncclCommWindowRegister) by resolving
them dynamically (dlopen + dlvsym/dlsym) or marking them weak so the shared
object can load on older libnccl; in practice update the version check to 2.27.3
in requestBuffer and implement late-binding for
ncclMemAlloc/ncclCommWindowRegister (or use weak symbol declarations) inside the
NCCLWindowAllocator so the code can gracefully fall back to regular tensors when
those symbols are absent rather than crashing at load time.

In `@cpp/tensorrt_llm/common/ncclUtils.h`:
- Around line 282-285: The constructor ScopedNCCLWindowBuffer currently
dereferences mComm when calling
NCCLWindowAllocator::getInstance().requestBuffer(*mComm, size), which can
segfault if the incoming std::shared_ptr<ncclComm_t> comm is null or points to a
null handle; update the constructor (and the similar helper in the 329-370
range) to first check that comm && *comm are valid, and only call requestBuffer
with *mComm when valid—otherwise set mBuffer to an empty/failed state or return
early so callers that treat a null comm as a normal failure continue to work
safely.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 807931b0-0046-4e70-a998-1112e4531bf9

📥 Commits

Reviewing files that changed from the base of the PR and between 02c8a94 and 887a7e8.

📒 Files selected for processing (4)

• cpp/tensorrt_llm/common/ncclUtils.cpp
• cpp/tensorrt_llm/common/ncclUtils.h
• cpp/tensorrt_llm/thop/allreduceOp.cpp
• cpp/tests/unit_tests/multi_gpu/ncclUtilsTest.cpp

[nv-lschneider]

/bot run --disable-fail-fast --add-multigpu

[nv-lschneider]

@Tabrizian is it OK to require a certain minimum NCCL version at run time?
I think it would make our lives easier if we can require it, since it fails fast if not.

[tensorrt-cicd]

PR_Github #38194 Bot args parsing error: usage: /bot [-h]
{run,kill,skip,submit,reviewers,reuse-pipeline,reuse-review} ...
/bot: error: unrecognized arguments: --add-multigpu

Link to invocation

[nv-lschneider]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #38197 [ run ] triggered by Bot. Commit: b0a827a Link to invocation

[tensorrt-cicd]

PR_Github #38197 [ run ] completed with state SUCCESS. Commit: b0a827a
/LLM/main/L0_MergeRequest_PR pipeline #29590 completed with status: 'FAILURE'

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

[nv-lschneider]

/bot run --disable-fail-fast --add-multi-gpu-test

[tensorrt-cicd]

PR_Github #38288 [ run ] triggered by Bot. Commit: 4fe79de Link to invocation

[tensorrt-cicd]

PR_Github #38288 [ run ] completed with state SUCCESS. Commit: 4fe79de
/LLM/main/L0_MergeRequest_PR pipeline #29667 completed with status: 'FAILURE'

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

[Tabrizian]

/bot run --disable-fail-fast --add-multi-gpu-test

[tensorrt-cicd]

PR_Github #38325 [ run ] triggered by Bot. Commit: 4fe79de Link to invocation

[tensorrt-cicd]

PR_Github #38325 [ run ] completed with state SUCCESS. Commit: 4fe79de
/LLM/main/L0_MergeRequest_PR pipeline #29703 completed with status: 'FAILURE'

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

[nv-lschneider]

/bot run --disable-fail-fast --add-multi-gpu-test

[tensorrt-cicd]

PR_Github #38373 [ run ] triggered by Bot. Commit: 4fe79de Link to invocation

[tensorrt-cicd]

PR_Github #38373 [ run ] completed with state SUCCESS. Commit: 4fe79de
/LLM/main/L0_MergeRequest_PR pipeline #29742 completed with status: 'ABORTED'

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

[nv-lschneider]

/bot run --disable-fail-fast --add-multi-gpu-test

[tensorrt-cicd]

PR_Github #38485 [ run ] triggered by Bot. Commit: 4fe79de Link to invocation

[nv-lschneider]

/bot run --disable-fail-fast --add-multi-gpu-test

[tensorrt-cicd]

PR_Github #38485 [ run ] completed with state SUCCESS. Commit: 4fe79de
/LLM/main/L0_MergeRequest_PR pipeline #29836 completed with status: 'FAILURE'

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

[tensorrt-cicd]

PR_Github #38491 [ run ] triggered by Bot. Commit: ead6bba Link to invocation

[tensorrt-cicd]

PR_Github #38491 [ run ] completed with state SUCCESS. Commit: ead6bba
/LLM/main/L0_MergeRequest_PR pipeline #29843 completed with status: 'FAILURE'

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

[nv-lschneider]

/bot run --disable-fail-fast --add-multi-gpu-test

[tensorrt-cicd]

PR_Github #38609 [ run ] triggered by Bot. Commit: ead6bba Link to invocation

[tensorrt-cicd]

PR_Github #38609 [ run ] completed with state SUCCESS. Commit: ead6bba
/LLM/main/L0_MergeRequest_PR pipeline #29946 completed with status: 'SUCCESS'

CI Report

Link to invocation