[Bugfix] Gate CUTLASS FP8 linear kernel on SM89+

[lesj0610]

Purpose

CutlassFP8ScaledMMLinearKernel.is_supported() only checks that the platform is CUDA. It never consults a capability probe, even though cutlass_scaled_mm_supports_fp8() rejects everything below SM89 (and also rejects SM89 without CUDA 12.4, or SM90+ without CUDA 12.0). Since _POSSIBLE_FP8_KERNELS[PlatformEnum.CUDA] places it ahead of MarlinFP8ScaledMMLinearKernel, it wins auto-selection on pre-Ada GPUs and nothing downstream can recover.

This became reachable at bca7bea2405127bd5291bb6fffa679bdcd8f6dd9 ("Remove VLLM_TEST_FORCE_FP8_MARLIN to replace with linear_backend/moe_backend"), which moved MarlinFP8ScaledMMLinearKernel from first to sixth in the CUDA FP8 priority list and dropped its compute_capability >= 89 self-exclusion. Before that commit the ordering plus the self-exclusion made Marlin the only candidate on SM75/80/86; after it, the ungated CUTLASS entry is chosen there instead.

Coverage does not catch it. The GSM8K eval configs that used to reach the Marlin path now pass --linear-backend marlin explicitly, so they no longer exercise auto-selection at all. tests/quantization/test_fp8.py still parametrizes force_marlin=False, which does go through auto-selection, but nothing guarantees that parametrization runs on pre-Ada hardware.

Symptoms on SM80 with a mixed-precision ModelOpt checkpoint whose attention projections are FP8:

• CompilationMode.VLLM_COMPILE: the FP8 activation-quant op keeps a float8_e4m3fn tensor in the Inductor graph. Triton has no fp8e4nv on SM80, so unsupported_input_tensor() makes fallback_node_due_to_unsupported_type() return True, PatternMatcherPass.apply() skips the auto_functionalized node, and decompose_auto_functionalized() ends with AssertionError: auto_functionalized was not removed.
• --enforce-eager: the CUTLASS FP8 GEMM has no kernel compiled for the arch.

Either way the affected ModelOpt FP8 linear layers are unservable unless the user knows to pass --linear-backend marlin by hand. MarlinFP8ScaledMMLinearKernel is documented as the "FP8 Marlin kernel for GPUs that lack FP8 hardware support", so making it unreachable on exactly those GPUs is the defect.

This PR reuses the probe that already backs cutlass_fp8_supported() so the CUTLASS entry declines the architecture/toolkit combinations it cannot serve and selection falls through to Marlin. On the tested CUDA 13.0 build, SM89+ behaviour is unchanged.

AI assistance: Claude Opus 5 was used to bisect the regression, implement the gate, and draft this description; the submitter reviewed the changes.

Changes

• CutlassFP8ScaledMMLinearKernel.is_supported() resolves the compute capability (argument first, device query as fallback) and returns unsupported when ops.cutlass_scaled_mm_supports_fp8() rejects it. The failure reason names the architecture and toolkit requirement the probe enforces, since a rejection on SM89/SM90 can come from the CUDA version rather than the capability.
• Add tests/model_executor/kernels/test_cutlass_fp8_linear.py: probe thresholds, the device-query fallback when a caller omits the capability, selector-level fall-through to Marlin on SM80 with a control that reproduces the regression, a check that SM90 still selects CUTLASS, and agreement with the unmocked runtime probe.

Test Plan

Run on an SM80 host.

.venv/bin/python -m ruff check \
  vllm/model_executor/kernels/linear/scaled_mm/cutlass.py \
  tests/model_executor/kernels/test_cutlass_fp8_linear.py

.venv/bin/python -m ruff format --check \
  vllm/model_executor/kernels/linear/scaled_mm/cutlass.py \
  tests/model_executor/kernels/test_cutlass_fp8_linear.py

.venv/bin/python -m pytest \
  tests/model_executor/kernels/test_cutlass_fp8_linear.py -q

.venv/bin/python -m pytest tests/model_executor/kernels/ -q

# Live gate on the host device.
.venv/bin/python -c "
from vllm.model_executor.kernels.linear import CutlassFP8ScaledMMLinearKernel as K
from vllm.platforms import current_platform
print(current_platform.get_device_capability().to_int(), K.is_supported(), K.is_supported(89))
"

# Capability boundary of the probe reused by the gate.
.venv/bin/python -c "
from vllm import _custom_ops as ops
print([(cc, ops.cutlass_scaled_mm_supports_fp8(cc)) for cc in (80, 86, 89, 90, 100, 120)])
"

Test Result

• ruff check: passed. ruff format --check: 2 files already formatted.
• pytest tests/model_executor/kernels/test_cutlass_fp8_linear.py -q: 11 passed, 0 skipped.
• pytest tests/model_executor/kernels/ -q: all 39 tests in that directory passed.
• Selector level, with one synthetic CUDA device pinned across every FP8 candidate: an SM80 per-tensor FP8 config selects MarlinFP8ScaledMMLinearKernel; the control that restores the CUDA-only check on the same config selects CutlassFP8ScaledMMLinearKernel, i.e. the regression is reproduced and the gate is what changes the outcome. An SM90 config still selects CutlassFP8ScaledMMLinearKernel.
• Live gate: capability 80, is_supported() returns False with the architecture/toolkit reason, is_supported(89) returns (True, None).
• Probe boundary on this CUDA 13.0 build: [(80, False), (86, False), (89, True), (90, True), (100, True), (120, True)] — the same threshold the pre-regression Marlin self-exclusion used.
• Pre-commit hooks passed, including mypy for Python 3.10.

The mocked tests pin the platform enum, the device-capability queries and the CUTLASS probe, so their outcome does not depend on the host or on the CUDA version the wheel was built against. test_cutlass_fp8_is_supported_matches_runtime_probe is the one unmocked case; it is skipif-guarded on current_platform.is_cuda(), which is a platform-enum check, and both sides of its assertion route through the same capability resolution, so it is consistent even where no device is visible.

Serving-level confirmation still needs a server restart with this head. The failing run was captured before the change and logs Selected CutlassFP8ScaledMMLinearKernel for ModelOptFp8LinearMethod immediately before the Inductor assertion; the expected post-fix log line is Selected MarlinFP8ScaledMMLinearKernel for ModelOptFp8LinearMethod.

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results.
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

Summary by CodeRabbit

• Bug Fixes

  • Improved FP8 linear operation compatibility checks by automatically detecting device capabilities.
  • Unsupported hardware now safely falls back to an alternative implementation instead of selecting an incompatible path.
  • Added clearer diagnostics when required CUDA capabilities are unavailable.

• Testing

  • Expanded coverage for FP8 kernel selection across supported and unsupported GPU architectures.

• Chores

  • Updated ARM64 CUDA 13.0 build and release environments for more consistent image generation.

[lesj0610]

@coderabbitai full review

[coderabbitai]

✅ Action performed

Full review finished.

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 16dd3bd5-cb00-441c-997f-7f05222d4169

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: ce633973-78fc-416f-885c-744330d06c64

📥 Commits

Reviewing files that changed from the base of the PR and between 040700a and ae490e7.

📒 Files selected for processing (5)

• .buildkite/image_build/image_build_arm64.sh
• .buildkite/release-pipeline.yaml
• .buildkite/scripts/hardware_ci/run-gh200-test.sh
• tests/model_executor/kernels/test_cutlass_fp8_linear.py
• vllm/model_executor/kernels/linear/scaled_mm/cutlass.py

Included review availability: Your plan provides up to 10 included reviews per hour; 8 remain after this review.

---

📝 Walkthrough

Walkthrough

The change gates CUTLASS FP8 kernel support by CUDA capability and adds selection tests. It also updates pinned CUDA 13.0 ARM64 builder image digests across image, release, and GH200 test pipelines.

Changes

CUTLASS FP8 capability gating

Layer / File(s)	Summary
Kernel capability gate and selector validation vllm/model_executor/kernels/linear/scaled_mm/cutlass.py, tests/model_executor/kernels/test_cutlass_fp8_linear.py	is_supported resolves device capability when omitted, checks CUDA and CUTLASS FP8 support, and returns rejection details. Tests cover unsupported capabilities, Marlin fallback, supported SM90 selection, and runtime probe alignment.

ARM64 CUDA builder image updates

Layer / File(s)	Summary
Pinned ARM64 builder references .buildkite/image_build/image_build_arm64.sh, .buildkite/release-pipeline.yaml, .buildkite/scripts/hardware_ci/run-gh200-test.sh	CUDA 13.0 ARM64 wheel, release image, Ubuntu 24.04 image, general image, and GH200 test builds now use the updated pinned builder digest.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Merge Risk: ⚪ Minimal · up to ae490

The change prevents unsupported GPUs and CUDA combinations from selecting the CUTLASS FP8 kernel and allows them to fall back to Marlin; the supplied checks pass, and no actionable merge-blocking risk remains beyond normal review.

Suggested reviewers: khluu

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 66.67% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 12 functions across 4 files. (1 skipped: 1 unsupported.)	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main change: gating the CUTLASS FP8 linear kernel for SM89 and newer architectures.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch lesj/cutlass-fp8-capability-gate-20260822

---

_{Comment @coderabbitai help to get the list of available commands.}