fix(ci): bump NCCL to 2.28+ for TensorRT-LLM compatibility

[slin1237]

Summary

• Bumps NCCL minimum from 2.27.7 to 2.28.0 to fix TRT-LLM build failure in CI

Root Cause

TRT-LLM PR #12015 (merged 2026-03-11, "Improve NCCL library load stability") wrapped NCCLWindowAllocator and createNCCLWindowTensor behind #if NCCL_VERSION_CODE >= NCCL_VERSION(2, 28, 0) in ncclUtils.h, but allreduceOp.cpp still references these symbols unconditionally. Building against NCCL 2.27 causes:

error: 'NCCLWindowAllocator' has not been declared in 'tensorrt_llm::_v1::common::nccl_util'

What changed

• scripts/ci_install_trtllm.sh: nvidia-nccl-cu13>=2.27.7 → nvidia-nccl-cu13>=2.28.0 (both cached and full-build paths). Cache version bumped to 3 to invalidate stale wheels.

Test plan

• CI e2e-1gpu-chat (trtllm) job passes

Summary by CodeRabbit

• Chores
  • Updated NCCL runtime dependency to version 2.28.0 for improved compatibility and performance.

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request addresses a critical build failure within the continuous integration environment for TensorRT-LLM by upgrading the minimum required NCCL library version. The change ensures compatibility with recent updates in TensorRT-LLM that introduced new symbols requiring a newer NCCL version, thereby stabilizing the CI pipeline and allowing successful builds.

Highlights

• NCCL Version Bump: Increased the minimum required NCCL version from 2.27.7 to 2.28.0 to resolve a TensorRT-LLM build failure in the CI pipeline.
• CI Cache Invalidation: Incremented the cache version for the TensorRT-LLM installation script to ensure that new builds pick up the updated NCCL dependency.

Changelog

• scripts/ci_install_trtllm.sh
  • Updated the nvidia-nccl-cu13 package requirement from >=2.27.7 to >=2.28.0 in both cached and full-build paths.
  • Incremented the cache version from 2 to 3 to invalidate stale cached wheels, ensuring the new NCCL version is picked up.

Activity

• No human activity has occurred on this pull request yet.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[coderabbitai]

📝 Walkthrough

Walkthrough

The CI installation script updates the NCCL runtime dependency from version 2.27.7 to 2.28.0 across two installation blocks and increments the cache version comment from 2 to 3. No logic or control flow changes.

Changes

Cohort / File(s)	Summary
Dependency Version Update scripts/ci_install_trtllm.sh	Bumped NCCL runtime dependency from 2.27.7 to 2.28.0 in two occurrences; updated cache version comment from "2" to "3".

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~2 minutes

Suggested labels

ci

Suggested reviewers

• CatherineSue
• key4ng
• XinyueZhang369

Poem

🐰 A hop and a bump, the NCCL climbs high,
From 2.27 to 2.28, reaching for the sky!
Cache version three now marks the way,
Fresh dependencies for a faster CI day! 🚀

🚥 Pre-merge checks | ✅ 3

✅ Passed checks (3 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately summarizes the main change: bumping NCCL to version 2.28+ to resolve TensorRT-LLM compatibility issues in CI.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

• 📝 Generate docstrings (stacked PR)
• 📝 Generate docstrings (commit on current branch)

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch slin/nv

📝 Coding Plan

• Generate coding plan for human review comments

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[gemini-code-assist]

Code Review

This pull request correctly bumps the minimum required NCCL version to 2.28.0 to fix a build failure with TensorRT-LLM. The changes are applied consistently for both cached and full builds. I've added one comment to suggest updating an outdated comment in the script to improve maintainability, aligning with the rule to ensure accurate external resource versions.

[coderabbitai]

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

scripts/ci_install_trtllm.sh (1)

159-163: 🧹 Nitpick | 🔵 Trivial

Update stale section header comment.

The section header on line 159 still references "NCCL 2.27" but now installs 2.28+. Update for consistency.

📝 Suggested fix

-# ── NCCL 2.27 setup ──────────────────────────────────────────────────────────
-# Use pip-installed NCCL 2.27+ which has both headers and libraries.
+# ── NCCL 2.28 setup ──────────────────────────────────────────────────────────
+# Use pip-installed NCCL 2.28+ which has both headers and libraries.
 # This matches the working installation guide approach.
 pip install --upgrade pip
 pip install --no-cache-dir "nvidia-nccl-cu13>=2.28.0"

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@scripts/ci_install_trtllm.sh` around lines 159 - 163, Update the stale
section header that currently reads "── NCCL 2.27 setup
──────────────────────────────────────────────────────────" to reflect the
actual package being installed (e.g., "── NCCL 2.28+ setup
──────────────────────────────────────────────────────────") so it matches the
pip install line that installs "nvidia-nccl-cu13>=2.28.0"; ensure the header
text near the pip install commands is consistent with the version constraint
used.

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Outside diff comments:
In `@scripts/ci_install_trtllm.sh`:
- Around line 159-163: Update the stale section header that currently reads "──
NCCL 2.27 setup ──────────────────────────────────────────────────────────" to
reflect the actual package being installed (e.g., "── NCCL 2.28+ setup
──────────────────────────────────────────────────────────") so it matches the
pip install line that installs "nvidia-nccl-cu13>=2.28.0"; ensure the header
text near the pip install commands is consistent with the version constraint
used.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: cdea1ede-ff9a-459f-96df-e717b3ec9f14

📥 Commits

Reviewing files that changed from the base of the PR and between 7b5d1dc and 877ccf9.

📒 Files selected for processing (1)

• scripts/ci_install_trtllm.sh