[https://nvbugs/6596064][fix] Release eager outputs before CUDA graph capture

[yuxianq]

Dev Engineer Review

• Releases the eager warmup result before CUDA graph capture in CUDAGraphRunner and EncoderCUDAGraphRunner.
• Restores the expected allocator lifetime and fixes the B200 illegal memory access.
• Removes the LagunaXS test_fp8 and test_nvfp4 waiver entries for the affected platforms.
• No public API or unrelated configuration changes found.
• B200 FP8 validation and pre-commit checks passed.

QA Engineer Review

• Modified tests/integration/test_lists/waives.txt.
• Removed five LagunaXS waiver entries for test_fp8 and test_nvfp4, including platform-specific test_fp8 entries for B200, B300, and RTXPro6000D.
• No test-db/ or qa/ files were modified.
• Verdict: needs follow-up because CBTS coverage data is unavailable.

Description

PR #16319 started retaining the eager warmup result in output across CUDA graph setup. On B200, that changed allocator liveness during capture and caused TestLagunaXS::test_fp8 to fail with an illegal memory access.

Release the eager warmup result before entering torch.cuda.graph in both CUDAGraphRunner and EncoderCUDAGraphRunner, restoring the pre-#16319 lifetime while preserving the current capture flow. Remove the corresponding NVBug 6525011 waivers for B200, B300, and RTXPro6000D.

Test Coverage

• B200: accuracy/test_llm_api_pytorch.py::TestLagunaXS::test_fp8
  • 1 passed, 4 warnings in 320.94s
  • MMLU: 73.904 (required >= 73.373)
  • GSM8K: 84.344 (required >= 83.947)
• Pre-commit checks for both changed files passed.

PR Checklist

Please review the following before submitting your PR:

• PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

• PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

• Test cases are provided for new code paths (see test instructions)

• If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.

• Any new dependencies have been scanned for license and vulnerabilities

• CODEOWNERS updated if ownership changes

• Documentation updated as needed

• Update tava architecture diagram if there is a significant design change in PR.

• The reviewers assigned automatically/manually are appropriate for the PR.

• Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

[yuxianq]

/bot run --disable-fail-fast

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 2eefa3fd-8b7d-45a2-a670-270f5e46f2db

📥 Commits

Reviewing files that changed from the base of the PR and between 3b604cd and b6936e0.

📒 Files selected for processing (1)

• tests/integration/test_lists/waives.txt

💤 Files with no reviewable changes (1)

• tests/integration/test_lists/waives.txt

Included review availability: Your plan includes up to 12 reviews per rolling hour; 11 remain after this review.

---

Walkthrough

The change clears eager warmup output references before decoder and encoder CUDA graph setup.

Changes

CUDA graph capture cleanup

Layer / File(s)	Summary
Capture cleanup tensorrt_llm/_torch/pyexecutor/cuda_graph_runner.py	The decoder and encoder capture paths clear eager warmup output references before CUDA graph setup.

Estimated code review effort: 1 (Trivial) | ~5 minutes

Merge Risk: ⚪ Minimal · up to b6936

The change restores eager-output lifetime before CUDA graph capture and removes the related test waivers; based on the supplied validation, no actionable merge-blocking risk remains beyond normal checks and review.

Possibly related PRs

• NVIDIA/TensorRT-LLM#16609: Clears stale eager warmup output references in cuda_graph_runner.py.

Suggested reviewers: cascade812, zhaoyuanh-nvidia

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly identifies the NVBug, fix type, and primary change before CUDA graph capture.
Description check	✅ Passed	The description explains the issue and solution, identifies affected tests, reports validation results, and includes the completed checklist.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[tensorrt-cicd]

PR_Github #66743 [ run ] triggered by Bot. Commit: c7f839d Link to invocation

[tensorrt-cicd]

PR_Github #66743 [ run ] completed with state SUCCESS. Commit: c7f839d
/LLM/main/L0_MergeRequest_PR pipeline #54349 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[yuxianq]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #66931 [ run ] triggered by Bot. Commit: 3b604cd Link to invocation

[YihuiLu512]

The PR itself looks good to me.

However, the unwaive operation and the CI coverage issue still require further explicit evaluation.

[yuxianq]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #66968 [ run ] triggered by Bot. Commit: b6936e0 Link to invocation

[tensorrt-cicd]

PR_Github #66931 [ run ] completed with state ABORTED. Commit: 3b604cd

Link to invocation

[tensorrt-cicd]

PR_Github #66968 [ run ] completed with state FAILURE. Commit: b6936e0
/LLM/main/L0_MergeRequest_PR pipeline #54517 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[yuxianq]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #67329 [ run ] triggered by Bot. Commit: b6936e0 Link to invocation

[tensorrt-cicd]

PR_Github #67329 [ run ] completed with state SUCCESS. Commit: b6936e0
/LLM/main/L0_MergeRequest_PR pipeline #54846 completed with status: 'UNSTABLE'

CI Report

⚠️ Multi-GPU Label Required:
Multi-GPU tests require the ci: full pre-merge approved label on this PR. Ask a member of NVIDIA/trt-llm-ci-approvers to add the label, then re-trigger CI with the same bot command (no rebase needed).

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

[yuxianq]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #67375 [ run ] triggered by Bot. Commit: b6936e0 Link to invocation

[tensorrt-cicd]

PR_Github #67375 [ run ] completed with state FAILURE. Commit: b6936e0
/LLM/main/L0_MergeRequest_PR pipeline #54884 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[yuxianq]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #67682 [ run ] triggered by Bot. Commit: b6936e0 Link to invocation

[tensorrt-cicd]

PR_Github #67682 [ run ] completed with state FAILURE. Commit: b6936e0
/LLM/main/L0_MergeRequest_PR pipeline #55168 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[yuxianq]

/bot skip --comment "the 2 failed tests also appear in other PRs and unrelated to this PR, skip CI"

[tensorrt-cicd]

PR_Github #67769 [ skip ] triggered by Bot. Commit: b6936e0 Link to invocation

[tensorrt-cicd]

PR_Github #67769 [ skip ] completed with state SUCCESS. Commit: b6936e0
Skipping testing for commit b6936e0

Link to invocation