[siliangchen-amd]

Motivation

Follow-up to #40557: The reviewer asked why _has_vllm is still kept.

_has_vllm in fp8_kernel.py is only ever set inside if _is_hip:, and only in the arm where _use_aiter is false:

if _is_hip:
    _has_vllm = False
    if _use_aiter:
        from aiter import dynamic_per_tensor_quant, ...
    else:
        try:
            import vllm._C
            _has_vllm = True
        except ImportError:
            _has_vllm = False

So the vLLM arm is selected in exactly one configuration: ROCm, SGLANG_USE_AITER off, and vllm._C importable. Nothing shipped reaches it:

configuration	SGLANG_USE_AITER	vLLM present	vLLM arm
docker/rocm.Dockerfile	1	—	never selected
docker/rocm-gfx1151.Dockerfile	0	not installed	never selected (native path)
nightly-test-amd.yml	1	—	never selected
python/pyproject.toml	—	not a dependency	—

And the one configuration that does select it cannot complete a forward pass on current main. RMSNorm.forward_hip calls fused_add_rms_norm with AITER's 6-argument out-of-place signature; with AITER off, layernorm.py binds that name to vLLM's 4-argument in-place op instead. Replaying layer 0's calls in execution order in that configuration:

fp8_kernel: _use_aiter=False _has_vllm=True
layernorm.fused_add_rms_norm bound to vllm._custom_ops.fused_add_rms_norm

  [ok]    input_layernorm      rms_norm(out, x, w, eps)                        [4 args]
  [ok]    qkv_proj fp8 quant   scaled_fp8_quant -> vLLM arm
  [ok]    o_proj   fp8 quant   scaled_fp8_quant -> vLLM arm
  [FAIL]  post_attention_norm  fused_add_rms_norm(out, x, r_out, r, w, eps)    [6 args]
TypeError: fused_add_rms_norm() takes 4 positional arguments but 6 were given

A real RMSNorm module called with a residual fails the same way, from the dispatch it actually takes:

TypeError: fused_add_rms_norm() takes 4 positional arguments but 6 were given
  at sglang/kernels/fused_op.py:659 in forward
  at sglang/srt/layers/layernorm.py:766 in forward_hip

So the vLLM FP8 arm runs twice in layer 0 and then the first residual norm raises. The fused_add_rms_norm signature is the same in vLLM 0.26.0 and 0.27.1.

Modifications

• Remove _has_vllm, the vllm._C import probe, and the three elif _has_vllm: arms (dynamic per-token, dynamic per-tensor, static). The import guard becomes if _is_hip and _use_aiter:.
• With AITER on, nothing changes. With AITER off, the native path takes over — the path gfx1151 already runs.
• In the per-tensor branch both remaining backends take a torch.empty scale, so the allocation is hoisted. The comment added in [AMD] Drop the redundant scale zero-fill before AITER per-tensor FP8 quant #40557, which explained why only vLLM needed the pre-zero, goes with the vLLM arm.
• The native fallbacks' docstrings said "when vLLM is unavailable"; they now say "when AITER is disabled".

One file, +13/−36.

Accuracy Tests

The native fallback against the vLLM ops it replaces, on all three call sites: 144 cases over bf16/fp16, six shapes from 1x4096 to 4096x4096, and gaussian, heavy-tailed and zero-row inputs. Run against both vLLM 0.27.1 and 0.26.0 with identical results.

call site	byte agreement native/vLLM (worst)	scale ratio native/vLLM	worst reconstruction-error gap
static, calibrated scale	100%	1.0000	0
static, scale saturating 224–240	100%	1.0000	0
dynamic per-token	93.73%	0.9967 – 1.0033	+1.6e-4
dynamic per-tensor	95.01%	0.9971 – 1.0032	+3.4e-4

Static is bit-identical, including scales chosen to push values between 224 and 240 — so vLLM caps float8_e4m3fnuz at 224 exactly as sglang does.

The dynamic gap is a dtype effect, not an algorithmic one. The native path divides in the input dtype, and the scale spread tracks that dtype's epsilon:

input	native/vLLM max |ratio−1|	dtype epsilon	same formula in fp32
bf16	3.33e-3	3.91e-3	5.0e-8
fp16	4.18e-4	4.88e-4	5.1e-8

For scale: AITER, which every shipped ROCm image runs, divides by the type maximum 240 where vLLM and sglang divide by 224. Its scale is a constant 224/240 = 0.9333 of vLLM's and it agrees with vLLM on ~20% of dynamic-quant bytes. Bit-compatibility with vLLM is not something this code relies on today.

After the patch, the public scaled_fp8_quant is bit-identical to calling AITER directly with SGLANG_USE_AITER=1, and to calling the native fallback directly with SGLANG_USE_AITER=0, on all three call sites (18 cases per mode).

Speed Tests and Profiling

The native path is 2.4–11x slower per call than vLLM's fused ops. µs per call on MI325X, bf16, vLLM 0.26.0, median of 7×300:

call site	shape	vLLM (graph)	native (graph)	native/vLLM	AITER (graph)
dynamic per-token	1×4096	3.68	29.46	8.0x	2.40
dynamic per-token	64×4096	3.98	41.30	10.4x	2.59
dynamic per-token	4096×4096	18.69	115.84	6.2x	12.87
dynamic per-tensor	1×4096	10.92	26.52	2.4x	6.06
dynamic per-tensor	64×4096	11.55	35.14	3.0x	6.72
dynamic per-tensor	4096×4096	63.99	115.09	1.8x	54.96
static	1×4096	2.41	11.57	4.8x	2.31
static	64×4096	2.58	13.28	5.1x	2.50
static	4096×4096	14.40	88.94	6.2x	12.38

Eager timings show the same ordering (per-token 5.9–7.9x, per-tensor 1.7–3.0x, static 3.8–6.1x).

No working deployment pays this: the only configuration that reached the removed arm cannot serve a model (see Motivation), and gfx1151 was already on the native path. If the AITER-off + vLLM mode is ever repaired at layernorm.py:766, its FP8 activation quant should get a fused kernel rather than fall through to the native path.

Not in scope

Left alone on purpose, each for a reason:

• layernorm.py _has_vllm_rms_norm — also set True when AITER is on, where it means "a fast rms_norm is available", so it needs restructuring rather than deletion. It is also where the AITER-off + vLLM breakage actually lives.
• moe/.../fused_moe.py _has_vllm_ops — gated on not _is_cuda and not _is_hip and not _is_xpu, so it serves other platforms, not ROCm.
• rotary_embedding/base.py — a previous attempt to drop the vLLM dependency there was reverted in Revert "[ROCm] Remove vLLM rope dependency & use AITER impl" #12028 after it broke GPT-OSS.

The manual quant test under SGLANG_USE_AITER=0

test/manual/quant/test_custom_ops.py (not run in CI) passes 6/6 with AITER on, before and after this change. With AITER off it exercises whichever fallback is selected, so it is worth being explicit about:

host	before	after
AITER off, vLLM installed	6/6 via the vLLM arm	native path
AITER off, no vLLM (e.g. the gfx1151 image)	native path	native path, unchanged

On the native path it passed 11 of 60 across ten runs. That is the native fallback as it already is on any ROCm host without vLLM; this change does not touch its arithmetic. Two things fail:

• The per-tensor and per-token assertions require exact agreement with an fp32 reference, and the native path divides in the input dtype — the dtype effect quantified above. The test was adapted from vLLM's kernel test, where that bar fits a fused kernel checked against its own formula; for a fallback it is stricter than it needs to be, and AITER does not meet it either.
• test_scaled_fp8_quant_with_padding hits a genuine shape bug: with num_token_padding the native fallback copies an input-sized result into a taller output and raises. On ROCm, apply_fp8_linear pads compressed-tensors FP8 activations to at least 17 rows unless torch.compile is on, so a batch of fewer than 17 tokens reaches it on a host running without AITER or vLLM. It predates this change and is left for a separate one.

Checklist

• ruff format --check: clean. ruff check: 48 findings before and after the change on this file, none introduced.
• No new unit tests: this removes a branch no working configuration reaches, and nothing references _has_vllm. The existing manual test is covered in the section above.
• Accuracy and speed results above.

---

CI States

Latest PR Test (Base): ❌ Run #35836133330
Latest PR Test (Extra): ❌ Run #35836133089
Latest PR Test (AMD ROCm 10): ❌ Run #35836133443