[TRTLLM-15314][feat] Add FP8 LoRA support for B200 by achartier · Pull Request #17521 · NVIDIA/TensorRT-LLM

achartier · 2026-08-11T21:35:46Z

Dev Engineer Review

Added native FP8 LoRA support for SM100 while preserving SM90 support.
Added architecture-specific eager and CUDA-graph grouped-GEMM dispatch.
Added supportsFp8GroupedGemm and lora_grouped_gemm_supports_fp8 capability queries.
Added shared capability gating for FP8 adapter loading and cache initialization.
Preserved dense-only homogeneous E4M3 cache and alignment requirements.
Rejected unsupported SM120 and SM121 devices.
Updated the B200 test-list configuration with LoRA test paths.
Review focus: validate CUTLASS compile guards, SM100 cluster configuration, runtime dispatch, and capability-query error handling.

QA Engineer Review

Added native FP8 capability and cache tests:
- test_supports_native_fp8_lora
- test_missing_native_fp8_lora_capability_query_warns_once
- test_native_fp8_lora_initializes_fp8_cache
- test_device_without_native_kernels_does_not_initialize_fp8_cache
- test_missing_native_fp8_lora_kernels_do_not_initialize_fp8_cache
- test_non_fp8_lora_does_not_query_device_capability
Updated grouped-GEMM regression tests for SM100 dispatch and kernel availability.
Updated LoRA manager tests for SM100 support, kernel fallback, and SM120 rejection.
Added test_lora_manager.py and _torch/lora to tests/integration/test_lists/test-db/l0_b200.yml.
The changed test functions are not individually listed in the test-list file.
Verdict: needs follow-up because CI failures require investigation and no CBTS coverage data is available.

Description

Extends dense native-FP8 LoRA support from Hopper SM90 to B200 SM100 while preserving the existing SM90 path and continuing to reject unsupported SM120/SM121 devices.

The eager and CUDA-graph grouped-GEMM paths now dispatch a dedicated CUTLASS SM100 collective with a 128x256x128 tile, a 1-SM Blackwell schedule, and CUTLASS's preferred (4, 2, 1) plus fallback (2, 1, 1) cluster shapes. The LoRA manager accepts SM100 FP8 adapters, and regression coverage verifies the architecture gates and both dispatch paths.

This change retains the existing dense-only, homogeneous E4M3 PEFT-cache contract and alignment requirements. Routed-expert MoE FP8 LoRA and SM120/SM121 support remain out of scope. Kernel tactic and cluster-shape autotuning is also left for a separate performance follow-up.

Test Coverage

Validated on one NVIDIA B200:

Incremental Release build of tensorrt_llm, bindings, and gtest_main targets.
Eager FP8 grouped GEMM matches its reference on SM100.
CUDA-graph FP8 grouped GEMM matches its reference after graph replay on SM100.
Full FP8 LoRA regression suite plus dense Qwen3 BF16-control and FP8-adapter end-to-end tests:
```
22 passed, 2 warnings in 102.78s
```

The warnings are existing environment warnings for the ModelOpt/Transformers version combination and an unknown pytest xdist_start_method option.

PR Checklist

Please review the following before submitting your PR:

PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.
Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

achartier · 2026-08-12T01:36:59Z

/bot run

coderabbitai · 2026-08-12T01:39:29Z

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

@coderabbitai resume to resume automatic reviews.
@coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

▶️ Resume reviews
🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: a2e6dbe6-9af5-41eb-a009-779f537a84dc

📥 Commits

Reviewing files that changed from the base of the PR and between 57c1675 and 956a917.

📒 Files selected for processing (1)

tensorrt_llm/_torch/pyexecutor/_util.py

🚧 Files skipped from review as they are similar to previous changes (1)

tensorrt_llm/_torch/pyexecutor/_util.py

Walkthrough

FP8 grouped GEMM now supports SM90 and SM100 through architecture-specific CUTLASS implementations. LoRA capability detection queries kernel availability and controls FP8 initialization, weight retention, and dtype fallback.

Changes

FP8 LoRA support

Layer / File(s)	Summary
Architecture-specific CUDA Graph kernels `cpp/tensorrt_llm/kernels/fp8GroupedGemmConfig.h`, `cpp/tensorrt_llm/kernels/cuda_graph_grouped_gemm.cu`, `cpp/tensorrt_llm/kernels/groupGemm.cu`	FP8 grouped GEMM uses SM90 and SM100 configuration types. Runtime dispatch selects the matching implementation and configures dynamic cluster metadata when required.
Grouped GEMM capability and operator wiring `cpp/tensorrt_llm/kernels/groupGemm.h`, `cpp/tensorrt_llm/thop/loraOp.cpp`, `cpp/tensorrt_llm/kernels/groupGemm.cu`	Grouped GEMM exposes `supportsFp8GroupedGemm`. The Torch operator forwards the SM-version capability query.
LoRA capability consumption and validation `tensorrt_llm/_torch/pyexecutor/_util.py`, `tensorrt_llm/lora_manager.py`, `tests/unittest/_torch/lora/`, `tests/unittest/others/`, `docs/source/features/lora.md`, `tests/integration/test_lists/test-db/l0_b200.yml`	LoRA initialization checks native FP8 support on SM90 and SM100. Tests cover warnings, cache initialization, dtype retention, fallback behavior, and B200 test-list coverage.

Estimated code review effort: 4 (Complex) | ~45 minutes

Mergeability Score: ⚪ Minimal · up to 956a9

The PR adds B200 FP8 LoRA support with documented regression coverage, and no actionable merge-blocking risk remains beyond normal checks and review.

Sequence Diagram(s)

sequenceDiagram
  participant LoRAManager
  participant TorchOperator
  participant GroupedGemm
  participant Sm90ConfigOrSm100Config
  LoRAManager->>TorchOperator: query native FP8 support
  TorchOperator->>GroupedGemm: pass SM version
  GroupedGemm->>Sm90ConfigOrSm100Config: select architecture configuration
  Sm90ConfigOrSm100Config-->>GroupedGemm: return compiled-kernel capability
  GroupedGemm-->>TorchOperator: return FP8 availability
  TorchOperator-->>LoRAManager: retain FP8 or use dtype fallback

Possibly related PRs

NVIDIA/TensorRT-LLM#16810: Adds the earlier FP8 LoRA grouped-GEMM implementation that this change extends to SM100.

Suggested reviewers: qijune, yuanjingx87

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 15.91% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and concisely identifies the main change: adding FP8 LoRA support for B200.
Description check	✅ Passed	The description explains the scope, implementation, exclusions, test coverage, and checklist in sufficient detail.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands.}

tensorrt-cicd · 2026-08-12T01:44:43Z

PR_Github #65451 [ run ] triggered by Bot. Commit: 2ca0da5 Link to invocation

tensorrt-cicd · 2026-08-12T03:32:08Z

PR_Github #65451 [ run ] completed with state FAILURE. Commit: 2ca0da5
/LLM/main/L0_MergeRequest_PR pipeline #53201 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

achartier · 2026-08-12T03:38:08Z

/bot run

tensorrt-cicd · 2026-08-12T03:44:13Z

PR_Github #65482 [ run ] triggered by Bot. Commit: 2ca0da5 Link to invocation

tensorrt-cicd · 2026-08-12T05:36:58Z

PR_Github #65482 [ run ] completed with state FAILURE. Commit: 2ca0da5
/LLM/main/L0_MergeRequest_PR pipeline #53227 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

achartier · 2026-08-12T13:20:11Z

/bot run

tensorrt-cicd · 2026-08-12T13:26:23Z

PR_Github #65579 [ run ] triggered by Bot. Commit: 2ca0da5 Link to invocation

tensorrt-cicd · 2026-08-12T15:27:32Z

PR_Github #65579 [ run ] completed with state SUCCESS. Commit: 2ca0da5
/LLM/main/L0_MergeRequest_PR pipeline #53312 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

achartier · 2026-08-12T17:37:30Z

/bot run

tensorrt-cicd · 2026-08-12T17:44:16Z

PR_Github #65627 [ run ] triggered by Bot. Commit: 3d3fb70 Link to invocation

brnguyen2

The feature ticket slot is [None] — extending FP8 LoRA to a new architecture is a nontrivial feature and should carry a TRTLLM JIRA for tracking, especially with the autotuning follow-up already planned.

Two non-blocking notes:

docs/source/features/lora.md doesn't document native FP8 adapter support at all (the gap predates this PR, from #16810). Since the supported-architecture story is now SM90+SM100 with explicit SM120/121 rejection, this is a good point to add a row to the support matrix — fine as a follow-up.
The capability plumbing is well done: supports_native_fp8_lora gates on what the binary actually compiled (via the new torch op) rather than device SM alone, so a build with EXCLUDE_SM_100F degrades to the bf16 conversion path instead of aborting inside the GEMM at runtime.

Main pre-merge ask is the CI test-list registration (see inline comments): as it stands, none of the SM100 kernel or manager tests execute in any CI stage.

tensorrt-cicd · 2026-08-12T18:20:25Z

PR_Github #65627 [ run ] completed with state FAILURE. Commit: 3d3fb70
/LLM/main/L0_MergeRequest_PR pipeline #53350 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

achartier · 2026-08-20T20:58:58Z

/bot run

tensorrt-cicd · 2026-08-20T21:05:27Z

PR_Github #67964 [ run ] triggered by Bot. Commit: ddd5a05 Link to invocation

tensorrt-cicd · 2026-08-20T22:26:16Z

PR_Github #67964 [ run ] completed with state SUCCESS. Commit: ddd5a05
/LLM/main/L0_MergeRequest_PR pipeline #55415 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

achartier · 2026-08-21T02:48:45Z

/bot run

tensorrt-cicd · 2026-08-21T02:55:11Z

PR_Github #68101 [ run ] triggered by Bot. Commit: ddd5a05 Link to invocation

tensorrt-cicd · 2026-08-21T03:36:56Z

PR_Github #68101 [ run ] completed with state SUCCESS. Commit: ddd5a05
/LLM/main/L0_MergeRequest_PR pipeline #55547 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

achartier · 2026-08-21T19:32:55Z

/bot run

tensorrt-cicd · 2026-08-21T19:38:56Z

PR_Github #68392 [ run ] triggered by Bot. Commit: ddd5a05 Link to invocation

tensorrt-cicd · 2026-08-21T20:44:29Z

PR_Github #68392 [ run ] completed with state SUCCESS. Commit: ddd5a05
/LLM/main/L0_MergeRequest_PR pipeline #55812 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

achartier · 2026-08-22T01:20:47Z

/bot run

tensorrt-cicd · 2026-08-22T01:27:32Z

PR_Github #68435 [ run ] triggered by Bot. Commit: ddd5a05 Link to invocation

tensorrt-cicd · 2026-08-22T02:18:20Z

PR_Github #68435 [ run ] completed with state FAILURE. Commit: ddd5a05
/LLM/main/L0_MergeRequest_PR pipeline #55855 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Add native SM100 FP8 grouped GEMM dispatch for eager and CUDA graph LoRA paths, mirror CUTLASS cluster settings, and size persistent grids using active clusters. Share the runtime kernel capability gate between adapter loading and cache initialization, preserve compute-dtype fallback for unsupported builds, and warn when the capability query is unavailable. Signed-off-by: Aurelien Chartier <2567591+achartier@users.noreply.github.com>

Signed-off-by: Aurelien Chartier <2567591+achartier@users.noreply.github.com>

achartier · 2026-08-22T03:21:17Z

/bot run

tensorrt-cicd · 2026-08-22T03:27:54Z

PR_Github #68440 [ run ] triggered by Bot. Commit: a4483f2 Link to invocation

tensorrt-cicd · 2026-08-22T17:11:18Z

PR_Github #68440 [ run ] completed with state FAILURE. Commit: a4483f2
/LLM/main/L0_MergeRequest_PR pipeline #55860 completed with status: 'UNSTABLE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

achartier · 2026-08-23T02:53:58Z

/bot run

tensorrt-cicd · 2026-08-23T02:59:49Z

PR_Github #68539 [ run ] triggered by Bot. Commit: a4483f2 Link to invocation

tensorrt-cicd · 2026-08-23T06:44:23Z

PR_Github #68539 [ run ] completed with state SUCCESS. Commit: a4483f2
/LLM/main/L0_MergeRequest_PR pipeline #55954 completed with status: 'SUCCESS'

CI Report

Link to invocation

QiJune

LGTM

github-actions Bot assigned achartier Aug 11, 2026

achartier force-pushed the fp8-lora-b200 branch from dd28d1b to 2ca0da5 Compare August 12, 2026 01:33

achartier marked this pull request as ready for review August 12, 2026 01:35

achartier requested review from a team as code owners August 12, 2026 01:35

achartier requested review from QiJune, allisonlim-nv, lori-ren, schetlur-nv and zongfeijing August 12, 2026 01:35

achartier force-pushed the fp8-lora-b200 branch from 2ca0da5 to 3d3fb70 Compare August 12, 2026 17:37

brnguyen2 reviewed Aug 12, 2026

View reviewed changes

Comment thread tests/unittest/_torch/peft/test_fp8_lora_grouped_gemm_regressions.py

Comment thread tests/unittest/others/test_lora_manager.py

Comment thread cpp/tensorrt_llm/kernels/groupGemm.cu Outdated

Comment thread tensorrt_llm/lora_manager.py Outdated

achartier changed the title ~~[None][feat] Add FP8 LoRA support for B200~~ [TRTLLM-15314][feat] Add FP8 LoRA support for B200 Aug 12, 2026

achartier requested review from a team as code owners August 12, 2026 20:56

achartier force-pushed the fp8-lora-b200 branch from f3e4131 to ddd5a05 Compare August 20, 2026 20:58

yuanjingx87 approved these changes Aug 20, 2026

View reviewed changes

allisonlim-nv approved these changes Aug 21, 2026

View reviewed changes

achartier added 5 commits August 21, 2026 20:20

fix: address FP8 LoRA review feedback

216b6d0

Signed-off-by: Aurelien Chartier <2567591+achartier@users.noreply.github.com>

style: clean up FP8 grouped GEMM constants

22a7584

Signed-off-by: Aurelien Chartier <2567591+achartier@users.noreply.github.com>

test: remove redundant FP8 LoRA capability test

c3893c2

Signed-off-by: Aurelien Chartier <2567591+achartier@users.noreply.github.com>

test: fix B200 FP8 LoRA unit test path

a4483f2

Signed-off-by: Aurelien Chartier <2567591+achartier@users.noreply.github.com>

achartier force-pushed the fp8-lora-b200 branch from ddd5a05 to a4483f2 Compare August 22, 2026 03:20

QiJune approved these changes Aug 23, 2026

View reviewed changes

QiJune merged commit aaaf659 into NVIDIA:main Aug 23, 2026
7 checks passed

achartier deleted the fp8-lora-b200 branch August 23, 2026 23:30

Uh oh!

Conversation

achartier commented Aug 11, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Dev Engineer Review

QA Engineer Review

Description

Test Coverage

PR Checklist

GitHub Bot Help

Uh oh!

achartier commented Aug 12, 2026

Uh oh!

coderabbitai Bot commented Aug 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Reviews paused

Walkthrough

Changes

Sequence Diagram(s)

❌ Failed checks (1 warning)

Uh oh!

tensorrt-cicd commented Aug 12, 2026

Uh oh!

tensorrt-cicd commented Aug 12, 2026

Uh oh!

achartier commented Aug 12, 2026

Uh oh!

tensorrt-cicd commented Aug 12, 2026

Uh oh!

tensorrt-cicd commented Aug 12, 2026

Uh oh!

achartier commented Aug 12, 2026

Uh oh!

tensorrt-cicd commented Aug 12, 2026

Uh oh!

tensorrt-cicd commented Aug 12, 2026

Uh oh!

achartier commented Aug 12, 2026

Uh oh!

tensorrt-cicd commented Aug 12, 2026

Uh oh!

brnguyen2 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tensorrt-cicd commented Aug 12, 2026

Uh oh!

achartier commented Aug 20, 2026

Uh oh!

tensorrt-cicd commented Aug 20, 2026

Uh oh!

tensorrt-cicd commented Aug 20, 2026

Uh oh!

achartier commented Aug 21, 2026

Uh oh!

tensorrt-cicd commented Aug 21, 2026

Uh oh!

tensorrt-cicd commented Aug 21, 2026

Uh oh!

achartier commented Aug 21, 2026

Uh oh!

tensorrt-cicd commented Aug 21, 2026

Uh oh!

tensorrt-cicd commented Aug 21, 2026

Uh oh!

achartier commented Aug 22, 2026

Uh oh!

tensorrt-cicd commented Aug 22, 2026

Uh oh!

tensorrt-cicd commented Aug 22, 2026

Uh oh!

achartier commented Aug 22, 2026

Uh oh!

tensorrt-cicd commented Aug 22, 2026

Uh oh!

tensorrt-cicd commented Aug 22, 2026

Uh oh!

achartier commented Aug 11, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Aug 12, 2026 •

edited

Loading