[https://nvbugs/6652876][fix] Fix SM120 FP8 autotuner scale profiles

[yuxianq]

Dev Engineer Review

• Added an SM120-specific FP8 block-scaling GEMM constraint.
• Varies activation-scale dimension 0 and pads the M dimension to a multiple of 4.
• Preserved SM100 and SM90 layout behavior.
• Removed the LagunaXS NVFP4 and RTXPro6000D LagunaXS FP8 waiver entries.
• Pre-commit checks and Python compilation pass.
• The local autotuner unit test is blocked by an installed TensorRT-LLM binding mismatch.

QA Engineer Review

• Modified only tests/integration/test_lists/waives.txt.
• Removed the LagunaXS NVFP4 waiver.
• Removed the RTXPro6000D LagunaXS FP8 waiver.
• No test-db/ or qa/ files were modified.
• Verdict: needs follow-up because CBTS coverage data is unavailable.

Description

SM120 FP8 block-scaling GEMM activation scales store the token dimension on
dimension 0 and require it to be padded to four rows. The autotuner previously
reused the SM100 constraint, which varied dimension 1 instead. During the
generation warmup this produced undersized synthetic activation-scale buffers
and could make the SM120 TMA scale load raise a Warp MMU fault.

Specialize the SM120 constraint to vary activation-scale dimension 0 with the
required four-row padding while preserving the existing SM100 and SM90
layouts. The Laguna XS FP8 and NVFP4 waivers are removed so CI validates the
ordered suite again; the observed NVFP4 timeout followed the FP8 worker failure
and stale prefetched-session handoff.

Test Coverage

• pre-commit run --files tensorrt_llm/_torch/custom_ops/torch_custom_ops.py tests/integration/test_lists/waives.txt
• python3 -m py_compile tensorrt_llm/_torch/custom_ops/torch_custom_ops.py
• Restored CI coverage for:
  • accuracy/test_llm_api_pytorch.py::TestLagunaXS::test_fp8 on RTXPro6000D
  • accuracy/test_llm_api_pytorch.py::TestLagunaXS::test_nvfp4
• Local tests/unittest/_torch/misc/test_autotuner.py execution is blocked by
  an installed TensorRT-LLM binding mismatch with latest main
  (IKvCacheColdPageCodec is missing); the source compiles and all repository
  pre-commit checks pass.

PR Checklist

Please review the following before submitting your PR:

• PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

• PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

• Test cases are provided for new code paths (see test instructions)

• If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.

• Any new dependencies have been scanned for license and vulnerabilities

• CODEOWNERS updated if ownership changes

• Documentation updated as needed

• Update tava architecture diagram if there is a significant design change in PR.

• The reviewers assigned automatically/manually are appropriate for the PR.

• Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 2e7ecb5f-2ffa-4e41-a4a9-85a69f7f5174

📥 Commits

Reviewing files that changed from the base of the PR and between ba0b9b4 and 96942f4.

📒 Files selected for processing (2)

• tensorrt_llm/_torch/custom_ops/torch_custom_ops.py
• tests/integration/test_lists/waives.txt

💤 Files with no reviewable changes (1)

• tests/integration/test_lists/waives.txt

Included review availability: Your plan provides up to 12 included reviews per hour; 11 remain after this review.

---

Walkthrough

The change adds an SM120-specific FP8 block-scaling GEMM constraint that pads M to a multiple of 4. It also removes two waived integration tests.

Changes

SM120 FP8 GEMM support

Layer / File(s)	Summary
SM120 constraint selection tensorrt_llm/_torch/custom_ops/torch_custom_ops.py, tests/integration/test_lists/waives.txt	SM120 now uses an M-dimension constraint that rounds values up to a multiple of 4. Other SM100+ architectures retain the existing constraint. Two waived integration test entries were removed.

Estimated code review effort: 2 (Simple) | ~5 minutes

Merge Risk: ⚪ Minimal · up to 96942

The PR corrects SM120 FP8 activation-scale profiling and restores the affected CI coverage, addressing undersized buffers that could trigger runtime faults; no actionable merge-blocking risk remains beyond normal checks and review.

Suggested reviewers: bowenfu, cascade812, tongyuantongyu

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	Docstring check was indeterminate for this PR — some files could not be analyzed in time. Not blocking.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly identifies the SM120 FP8 autotuner fix and follows the required ticket and type format.
Description check	✅ Passed	The description explains the issue, solution, test coverage, known test limitation, and includes the required checklist.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[yuxianq]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #68745 [ run ] triggered by Bot. Commit: 96942f4 Link to invocation

[tensorrt-cicd]

PR_Github #68745 [ run ] completed with state FAILURE. Commit: 96942f4
/LLM/main/L0_MergeRequest_PR pipeline #56144 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[yuxianq]

It duplicates with #18140, close this one.