[https://nvbugs/6633931][fix] Chunk the MoE workspace on the SM90 branch of TestDeepSeekV32::test_fp8_blockscale

[pranav-nvidia]

Dev Engineer Review

• test_fp8_blockscale now sets MoeConfig(max_num_tokens=16384) on the SM90 path.
• The setting enables MoE workspace chunking and reduces peak CUDA memory use.
• The value matches the existing Blackwell configuration.
• The waiver for NVBugs 6633931 was removed.
• The waiver entry format and test path are correct.
• Verdict: sufficient.

QA Engineer Review

• Modified test: TestDeepSeekV32::test_fp8_blockscale[baseline].
• The test is covered through --stage-list.
• The test no longer has a waiver in tests/integration/test_lists/waives.txt.
• Verdict: sufficient.

Description

TestDeepSeekV32::test_fp8_blockscale[baseline] fails intermittently on DGX_H200-8_GPUs-PyTorch-Post-Merge-1 with a CUDA OOM during executor warmup: a single 14.02 GiB MoE workspace allocation against 10.80 GiB free.

The SM90 branch of the test builds MoeConfig() with no max_num_tokens. Left unset, moe_max_num_tokens defaults to max_num_tokens * dp_size (8192 * 8 = 65536), which the DP-gathered row count can never exceed, so calculate_num_chunks returns 1 and MoE sizes one workspace for all 65536 rows. That workspace is allocated during warmup, after the KV cache has been sized from a profiling pass taken before it, so it has to fit in whatever the KV cache left behind. baseline is mtp_nextn=0, so it has the lowest profiling peak of the three parametrizations, hence the largest KV cache (13.55 GiB) and the least headroom — which is why it fails and its siblings do not.

Capping the MoE token count at 16384 gives 4 chunks and a ~3.5 GiB workspace. MoE chunking is mathematically equivalent, so accuracy is unaffected. The Blackwell branch immediately above has carried max_num_tokens=16384 since the test was introduced; this brings the SM90 branch in line.

The waive for this test is removed in the same change.

A separate PR fixes the kernel-side over-allocation that made this latent gap start failing (#16849 grew the SM90 fp8-block-scale MoE workspace from 11.0 to 14.0 GiB). The two are independent: this change alone restores margin for the test, the kernel change alone leaves it fitting by only 0.12 GiB.

Test Coverage

accuracy/test_llm_api_pytorch.py::TestDeepSeekV32::test_fp8_blockscale[baseline] on DGX_H200-8_GPUs-PyTorch-Post-Merge-1, un-waived by this PR and run via --stage-list.

PR Checklist

• Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

[pranav-nvidia]

/bot run --disable-fail-fast

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: f1f5a89f-0ff3-4d92-9b03-20fa336be64a

📥 Commits

Reviewing files that changed from the base of the PR and between 1c7d1c0 and b596f69.

📒 Files selected for processing (2)

• tests/integration/defs/accuracy/test_llm_api_pytorch.py
• tests/integration/test_lists/waives.txt

💤 Files with no reviewable changes (1)

• tests/integration/test_lists/waives.txt

🚧 Files skipped from review as they are similar to previous changes (1)

• tests/integration/defs/accuracy/test_llm_api_pytorch.py

Included review availability: Your plan provides up to 12 included reviews per hour; 11 remain after this review.

---

Walkthrough

The FP8 blockscale accuracy test now sets MoeConfig(max_num_tokens=16384) on the non-SM100/SM103 path. The DGX H200 baseline waiver for this test is removed.

Changes

FP8 blockscale validation

Layer / File(s)	Summary
Configure FP8 blockscale validation tests/integration/defs/accuracy/test_llm_api_pytorch.py	The non-SM100/SM103 test path sets the MoE token limit to 16,384 and documents the workspace and KV-cache sizing constraint.

Estimated code review effort: 1 (Trivial) | ~5 minutes

Merge Risk: ⚪ Minimal · up to b596f

This change limits workspace sizing for the SM90 test and removes its waiver, reducing intermittent test OOMs without changing product runtime behavior. No actionable merge-blocking risk remains beyond normal checks and review.

Suggested reviewers: bowenfu

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 1 functions across 1 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title follows the required NVBugs and fix format and clearly identifies the MoE workspace chunking change on the SM90 branch.
Description check	✅ Passed	The description explains the CUDA OOM cause, the configuration change, expected impact, waiver removal, and relevant test coverage. The required sections are present and sufficiently complete.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

• Fix all pre-merge checks with AI

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[tensorrt-cicd]

PR_Github #70405 [ run ] triggered by Bot. Commit: 0ec5127 Link to invocation

[tensorrt-cicd]

PR_Github #70405 [ run ] completed with state SUCCESS. Commit: 0ec5127
/LLM/main/L0_MergeRequest_PR pipeline #57644 completed with status: 'UNSTABLE'

CI Report

⚠️ Multi-GPU Label Required:
Multi-GPU tests require the ci: full pre-merge approved label on this PR. Ask a member of NVIDIA/trt-llm-ci-approvers to add the label, then re-trigger CI with the same bot command (no rebase needed).

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

[pranav-nvidia]

/bot run

[tensorrt-cicd]

PR_Github #70477 [ run ] triggered by Bot. Commit: 0ec5127 Link to invocation

[tensorrt-cicd]

PR_Github #70477 [ run ] completed with state FAILURE. Commit: 0ec5127
/LLM/main/L0_MergeRequest_PR pipeline #57702 completed with status: 'ABORTED'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

[pranav-nvidia]

/bot run

[tensorrt-cicd]

PR_Github #70489 [ run ] triggered by Bot. Commit: 0ec5127 Link to invocation

[tensorrt-cicd]

PR_Github #70489 [ run ] completed with state SUCCESS. Commit: 0ec5127
/LLM/main/L0_MergeRequest_PR pipeline #57711 completed with status: 'SUCCESS'

CI Report

Link to invocation

[coderabbitai]

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

[pranav-nvidia]

/bot run

[tensorrt-cicd]

PR_Github #70782 [ run ] triggered by Bot. Commit: b596f69 Link to invocation

[tensorrt-cicd]

PR_Github #70782 [ run ] completed with state SUCCESS. Commit: b596f69
/LLM/main/L0_MergeRequest_PR pipeline #57969 completed with status: 'SUCCESS'

CI Report

Link to invocation