[https://nvbugs/6619882][test] Unwaive unittest/_torch/sampler on DGX_B200 by zhaoyangwang-nvidia · Pull Request #17946 · NVIDIA/TensorRT-LLM

zhaoyangwang-nvidia · 2026-08-19T03:42:01Z

Dev Engineer Review

Removed the _torch/sampler -k "not test_speculative_d2h_parity_real_predictor" waiver from tests/integration/test_lists/waives.txt.
Restored DGX_B200 sampler coverage without changing test code or public entities.
The waiver format and scope are consistent with the objective.
A local 8xB200 run passed 1,147 tests, skipped 699, and deselected 2.

QA Engineer Review

Modified tests/integration/test_lists/waives.txt.
Removed the _torch/sampler waiver entry.
CBTS coverage data is unavailable.

Verdict: needs follow-up

Description

unittest/_torch/sampler -k "not test_speculative_d2h_parity_real_predictor" was waived on DGX_B200 by #17782 (a098858cef) because of an intermittent failure that shows up roughly 4 times per month.

The reported failure is not an assertion failure. In the CI log the inner pytest process dies without printing a summary or a traceback, right after the KV cache allocation of a test_beam_search_e2e[multi_process-...] case; the outer wrapper then only surfaces the generic AssertionError: failure reported in unittests.

Waiving the whole entry drops coverage for the entire sampler directory on Blackwell, which is far broader than the intermittent failure itself. This PR removes the waive so the case runs in CI again and the intermittent failure can be characterized on current main.

Test Coverage

unittest/_torch/sampler -k "not test_speculative_d2h_parity_real_predictor" (DGX_B200-PyTorch-1) is re-enabled; no other test is affected.

Local run on an 8xB200 node at bd90276f6b, bypassing the waive list:

1147 passed, 699 skipped, 2 deselected in 995.93s (0:16:35)

PR Checklist

Please review the following before submitting your PR:

PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.
Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

coderabbitai · 2026-08-19T03:43:40Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 991e4d71-a875-4a39-abe1-b52a91140462

📥 Commits

Reviewing files that changed from the base of the PR and between c82fabf and 494ed57.

📒 Files selected for processing (1)

tests/integration/test_lists/waives.txt

💤 Files with no reviewable changes (1)

tests/integration/test_lists/waives.txt

Included review availability: Your plan provides up to 12 included reviews per hour; 11 remain after this review.

Walkthrough

The integration waiver list removes the skip entry for test_speculative_d2h_parity_real_predictor in the _torch/sampler test group.

Changes

Sampler waiver update

Layer / File(s)	Summary
Remove sampler test skip `tests/integration/test_lists/waives.txt`	Removes the waiver that excluded `test_speculative_d2h_parity_real_predictor` from the `_torch/sampler` test group.

Estimated code review effort: 1 (Trivial) | ~2 minutes

Merge Risk: ⚪ Minimal · up to 494ed

This change only re-enables the existing sampler test suite on DGX_B200; no actionable merge-blocking risk remains beyond normal checks and review.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description check	✅ Passed	The description explains why the waiver was removed, identifies the affected test, and provides relevant local test coverage.
Title check	✅ Passed	The title clearly identifies the test waiver removal and the DGX_B200 target.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0 files.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands.}

zhaoyangwang-nvidia · 2026-08-19T05:35:08Z

/bot run

zhaoyangwang-nvidia · 2026-08-19T05:39:31Z

/bot run --stage-list "DGX_B200-PyTorch-1"

tensorrt-cicd · 2026-08-19T05:41:32Z

PR_Github #67299 [ run ] triggered by Bot. Commit: 33a6a44 Link to invocation

tensorrt-cicd · 2026-08-19T05:45:05Z

PR_Github #67300 [ run ] triggered by Bot. Commit: 33a6a44 Link to invocation

tensorrt-cicd · 2026-08-19T05:48:53Z

PR_Github #67299 [ run ] completed with state ABORTED. Commit: 33a6a44

Link to invocation

tensorrt-cicd · 2026-08-19T07:44:21Z

PR_Github #67300 [ run ] completed with state SUCCESS. Commit: 33a6a44
/LLM/main/L0_MergeRequest_PR pipeline #54821 (Partly Tested) completed with status: 'SUCCESS'

CI Report

Link to invocation

zhaoyangwang-nvidia · 2026-08-19T08:50:28Z

/bot run --stage-list "DGX_B200-PyTorch-1"

tensorrt-cicd · 2026-08-19T08:56:12Z

PR_Github #67374 [ run ] triggered by Bot. Commit: 33a6a44 Link to invocation

tensorrt-cicd · 2026-08-20T08:56:05Z

PR_Github #67374 [ run ] completed with state ABORTED. Commit: 33a6a44

Link to invocation

zhaoyangwang-nvidia · 2026-08-24T01:28:10Z

/bot run --stage-list "DGX_B200-PyTorch-1"

tensorrt-cicd · 2026-08-24T01:33:08Z

PR_Github #68621 [ run ] triggered by Bot. Commit: 33a6a44 Link to invocation

tensorrt-cicd · 2026-08-24T02:45:53Z

PR_Github #68621 [ run ] completed with state SUCCESS. Commit: 33a6a44
/LLM/main/L0_MergeRequest_PR pipeline #56030 (Partly Tested) completed with status: 'SUCCESS'

CI Report

Link to invocation

zhaoyangwang-nvidia · 2026-08-24T03:00:24Z

/bot run

tensorrt-cicd · 2026-08-24T03:06:08Z

PR_Github #68654 [ run ] triggered by Bot. Commit: 33a6a44 Link to invocation

tensorrt-cicd · 2026-08-24T03:58:03Z

PR_Github #68654 [ run ] completed with state FAILURE. Commit: 33a6a44
/LLM/main/L0_MergeRequest_PR pipeline #56060 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

zhaoyangwang-nvidia · 2026-08-24T07:02:54Z

/bot run

tensorrt-cicd · 2026-08-24T07:09:12Z

PR_Github #68725 [ run ] triggered by Bot. Commit: 33a6a44 Link to invocation

tensorrt-cicd · 2026-08-24T07:57:50Z

PR_Github #68725 [ run ] completed with state FAILURE. Commit: 33a6a44
/LLM/main/L0_MergeRequest_PR pipeline #56122 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

zhaoyangwang-nvidia · 2026-08-24T08:43:16Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-08-24T08:49:14Z

PR_Github #68768 [ run ] triggered by Bot. Commit: 33a6a44 Link to invocation

tensorrt-cicd · 2026-08-24T10:15:30Z

PR_Github #68768 [ run ] completed with state FAILURE. Commit: 33a6a44
/LLM/main/L0_MergeRequest_PR pipeline #56164 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

…_B200 The whole `unittest/_torch/sampler` entry was waived on DGX_B200 by NVIDIA#17782 because of an intermittent failure (about 4 occurrences per month). The reported failure is not an assertion failure: the inner pytest process dies without emitting a summary or traceback, and the outer wrapper only surfaces `AssertionError: failure reported in unittests`. Waiving the whole entry removes coverage for the entire sampler directory on Blackwell, which is much broader than the intermittent failure itself. Unwaive it so the case runs in CI again and the intermittent failure can be characterized on current main. Signed-off-by: ZhaoyangWang <zhaoyangw@nvidia.com>

zhaoyangwang-nvidia · 2026-08-24T10:32:33Z

/bot run --disable-fail-fast

coderabbitai · 2026-08-24T10:32:37Z

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

tensorrt-cicd · 2026-08-24T10:38:49Z

PR_Github #68804 [ run ] triggered by Bot. Commit: 494ed57 Link to invocation

tensorrt-cicd · 2026-08-24T13:51:27Z

PR_Github #68804 [ run ] completed with state SUCCESS. Commit: 494ed57
/LLM/main/L0_MergeRequest_PR pipeline #56198 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

zhaoyangwang-nvidia · 2026-08-24T14:10:29Z

/bot run

zhaoyangwang-nvidia · 2026-08-25T01:12:36Z

/bot run

tensorrt-cicd · 2026-08-25T01:19:09Z

PR_Github #68940 [ run ] triggered by Bot. Commit: 494ed57 Link to invocation

tensorrt-cicd · 2026-08-25T02:25:50Z

PR_Github #68940 [ run ] completed with state SUCCESS. Commit: 494ed57
/LLM/main/L0_MergeRequest_PR pipeline #56323 completed with status: 'SUCCESS'

CI Report

Link to invocation

github-actions Bot assigned zhaoyangwang-nvidia Aug 19, 2026

lori-ren approved these changes Aug 19, 2026

View reviewed changes

coderabbitai Bot mentioned this pull request Aug 19, 2026

[https://nvbugs/6578853][test] Unwaive layer-wise benchmarks performance alignment test #17963

Merged

1 task

zhaoyangwang-nvidia force-pushed the unwaive-sampler-b200-6619882 branch from 33a6a44 to 494ed57 Compare August 24, 2026 10:32

zhaoyangwang-nvidia enabled auto-merge (squash) August 24, 2026 10:40

zhaoyangwang-nvidia merged commit 7e6c0b2 into NVIDIA:main Aug 25, 2026
11 checks passed

Uh oh!

Conversation

zhaoyangwang-nvidia commented Aug 19, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Dev Engineer Review

QA Engineer Review

Description

Test Coverage

PR Checklist

GitHub Bot Help

Uh oh!

coderabbitai Bot commented Aug 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Uh oh!

zhaoyangwang-nvidia commented Aug 19, 2026

Uh oh!

zhaoyangwang-nvidia commented Aug 19, 2026

Uh oh!

tensorrt-cicd commented Aug 19, 2026

Uh oh!

tensorrt-cicd commented Aug 19, 2026

Uh oh!

tensorrt-cicd commented Aug 19, 2026

Uh oh!

tensorrt-cicd commented Aug 19, 2026

Uh oh!

zhaoyangwang-nvidia commented Aug 19, 2026

Uh oh!

tensorrt-cicd commented Aug 19, 2026

Uh oh!

tensorrt-cicd commented Aug 20, 2026

Uh oh!

zhaoyangwang-nvidia commented Aug 24, 2026

Uh oh!

tensorrt-cicd commented Aug 24, 2026

Uh oh!

tensorrt-cicd commented Aug 24, 2026

Uh oh!

zhaoyangwang-nvidia commented Aug 24, 2026

Uh oh!

tensorrt-cicd commented Aug 24, 2026

Uh oh!

tensorrt-cicd commented Aug 24, 2026

Uh oh!

zhaoyangwang-nvidia commented Aug 24, 2026

Uh oh!

tensorrt-cicd commented Aug 24, 2026

Uh oh!

tensorrt-cicd commented Aug 24, 2026

Uh oh!

zhaoyangwang-nvidia commented Aug 24, 2026

Uh oh!

tensorrt-cicd commented Aug 24, 2026

Uh oh!

tensorrt-cicd commented Aug 24, 2026

Uh oh!

zhaoyangwang-nvidia commented Aug 24, 2026

Uh oh!

coderabbitai Bot commented Aug 24, 2026

Uh oh!

tensorrt-cicd commented Aug 24, 2026

Uh oh!

tensorrt-cicd commented Aug 24, 2026

Uh oh!

zhaoyangwang-nvidia commented Aug 24, 2026

Uh oh!

zhaoyangwang-nvidia commented Aug 25, 2026

Uh oh!

tensorrt-cicd commented Aug 25, 2026

Uh oh!

tensorrt-cicd commented Aug 25, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

zhaoyangwang-nvidia commented Aug 19, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Aug 19, 2026 •

edited

Loading