[Manfredss]

Overview

Fixes #60162. With the Humming FP8 linear kernel, fp8 checkpoints with per-tensor weight scales and online fp8_per_tensor / fp8_per_channel quantization crash while loading weights. On Ampere with VLLM_BATCH_INVARIANT=1, Humming is the only FP8 linear kernel left (Marlin opts out of batch-invariant mode), so these models cannot start in that mode. Humming FP8 is W8A16, so fp8_per_channel (PTPC, which asks for per-token FP8 activations) now gets the same clear error as on Marlin instead of running.

Claims

• Fp8LinearMethod (non-block) and Fp8PerTensorOnlineLinearMethod load and run on Humming. GSM8K accuracy is within about one standard error of Marlin.
• On Ampere with VLLM_BATCH_INVARIANT=1, these models start and are batch-invariant under the default torch.compile + CUDA graphs.
• Fp8PtpcOnlineLinearMethod rejects Humming with the same error it already raises for Marlin.

Validation

Latest run, after merging main (Oct 26): one A40 (SM86) on Runpod, official image vllm/vllm-openai:nightly-00b6c4c565a8732c131e1f452d4a17803bb0842b, with this PR's diff applied to the installed package. Raw logs: PTPC, everything else.

Case	main	this PR
pytest tests/quantization/test_fp8.py::test_model_load_and_run	humming case fails (stride error)	3 passed, 3 skipped
Llama-3-8B-Instruct-FP8, VLLM_BATCH_INVARIANT=1	crash (stride error)*	starts; batch-invariant 3/3; GSM8K 0.7544
Qwen3-8B --quantization fp8_per_tensor, VLLM_BATCH_INVARIANT=1	crash (stride error)*	starts; batch-invariant 3/3; GSM8K 0.8673
Qwen3-8B --quantization fp8_per_channel, VLLM_BATCH_INVARIANT=1	crash (stride error)	ValueError: Humming is W8A16 (same check as Marlin)
Qwen3-8B --quantization fp8_per_channel, --linear-backend humming	—	same ValueError

* From the earlier runs below; main's code path for these is unchanged.

GSM8K is lm_eval gsm8k (5-shot, all 1319 questions), flexible-extract. For reference, on the Oct 20 RTX 3090 run, Llama-3-8B-Instruct-FP8 in default mode (Marlin on both sides) scored 0.7642 on main and with this PR.

Earlier runs (Oct 6, Oct 15, Oct 20)

Rows that show fp8_per_channel starting on Humming predate the PTPC check.

Oct 20, RTX 3090, nightly e68efc68: repro, lm_eval and bench with raw logs in this comment.

Re-validated after merging main (Oct 15). Head 671702b6e (main 984c242ede + this PR). Since main's latest wheel was not published yet, I used the precompiled extensions from 388a610dfb (the only build change in between is the DeepGEMM tag) with torch 2.14.0, humming-kernels 0.1.17 and flashinfer 0.7.1, on an A40 inside the official nightly image's CUDA environment. Toggling between main and this PR was checked by SHA-256 of the touched files.

Case	main	this PR
Llama-3-8B-Instruct-FP8, VLLM_BATCH_INVARIANT=1	crash (stride error)	starts; batch-invariant 3/3; GSM8K 0.780
Qwen3-8B fp8_per_tensor / fp8_per_channel, VLLM_BATCH_INVARIANT=1	—	start; batch-invariant 3/3 each
Llama-3-8B-Instruct-FP8, --linear-backend humming	—	GSM8K 0.784 (Marlin baseline on main: 0.772)
test_model_load_and_run with this PR's test	humming fails (stride error)	3 passed
tests/quantization/test_fp8.py	3 test_fp8_reloading failures	same 3 failures, 26 passed

GSM8K uses 500 questions, 5-shot.

The original validation below was done before the merge, on nightly 92044241.

Run on the official image vllm/vllm-openai:nightly-92044241a02f05de51420d654daf669de20b4691 on an A40 (SM86), with the container's environment. The installed source files were checked against commit 92044241 by SHA-256 before testing. "This PR" means this diff applied to the installed package; the touched files are unchanged between 92044241 and this PR's base. Batch invariance uses the reproducer from #59086: request A alone, alone again, then batched with a longer request B; 48 greedy tokens; 3 rounds.

Model / quantization	Flags	main	this PR
neuralmagic/Meta-Llama-3-8B-Instruct-FP8 (fp8, per-tensor)	VLLM_BATCH_INVARIANT=1	crash: self.stride(-1) must be 1 to view Float8_e4m3fn as Int ... but got 4096	starts; A batched vs alone identical 3/3; GSM8K 0.790
Qwen/Qwen3-8B --quantization fp8_per_tensor	VLLM_BATCH_INVARIANT=1	crash (same error)	starts; identical 3/3
Qwen/Qwen3-8B --quantization fp8_per_channel	VLLM_BATCH_INVARIANT=1	crash (same error)	starts; identical 3/3
Llama-3-8B-Instruct-FP8	--linear-backend humming	crash	starts; GSM8K 0.784
Qwen3-8B --quantization fp8_per_tensor	--linear-backend humming	crash	—
Llama-3-8B-Instruct-FP8 (control)	default (Marlin)	GSM8K 0.786	—
Qwen/Qwen3-8B-FP8 (block, control)	VLLM_BATCH_INVARIANT=1	works, identical 3/3	—

GSM8K: tests/evals/gsm8k/gsm8k_eval.py --num-questions 500 (5-shot).

test_model_load_and_run now also runs with linear_backend="humming" (pytest tests/quantization/test_fp8.py::test_model_load_and_run -v, same image):

	auto	marlin	humming
main	passed	passed	failed (stride error above)
this PR	passed	passed	passed

The rest of tests/quantization/test_fp8.py gives the same result with and without this PR on this GPU. Three test_fp8_reloading cases fail on the A40 either way with identical errors, and are unrelated to this change. pre-commit passes on the changed files.

Details

Missing layout tag. process_fp8_weight_tensor_strategy returns the weight as a (K, N) transposed view, and Fp8PerTensorOnlineLinearMethod transposes it itself. Humming's convert_linear_layer_to_humming_standard reads layer.processed_weight_dims (falling back to the parameter's checkpoint input_dim/output_dim) to detect the layout. Without it, Humming assumes (N, K) and calls .view(torch.int32) on the transposed view, which fails. Since #62326, CompressedTensors, fbgemm, ModelOpt and Quark FP8 set processed_weight_dims = (0, 1) after transposing. This PR does the same in the two methods that lacked it. Humming is the only FP8 scaled-mm kernel that reads it, so other kernels are unaffected.

PTPC. Fp8PtpcOnlineLinearMethod already refuses Marlin because PTPC requires per-token FP8 activations and Marlin FP8 is W8A16. HummingFP8ScaledMMLinearKernel builds its layer without an input schema, so it is W8A16 too. With the layout fixed, PTPC would silently run as weight-only FP8 on Humming, so this PR adds Humming to that check. Fp8LinearMethod and fp8_per_tensor already accept Marlin's W8A16 fallback, and Humming matches it.

Batch-invariant apply. In batch-invariant mode these methods call the kernel only when fp8_scaled_mm_is_batch_invariant() (from #52231) says so; otherwise they dequantize layer.weight and run F.linear. After Humming's process_weights_after_loading, layer.weight is in Humming's packed layout, so that fallback fails (a and b must have same reduction dim, but got [s, 4096] X [256, 24576]). Humming has its own batch-invariant mode (use_batch_invariant in its compute config), and the block-FP8 path and compressed-tensors already call it in this mode. This PR makes fp8_scaled_mm_is_batch_invariant() return True for Humming. Humming FP8 is weight-only here, so the dynamic per-tensor activation amax that the helper guards against doesn't apply.

Notes.

• Without batch-invariant mode, two identical requests on Humming were not bitwise identical (first difference at token 4). With VLLM_BATCH_INVARIANT=1 they were. vLLM only promises this in batch-invariant mode, and it does not come from this change.
• CI: the new humming case is in tests/quantization/test_fp8.py, whose Quantization step is temporarily disabled by [CI] Disable jobs with unlocked transitive dependencies #63608. The layout bug does not depend on the GPU architecture. I validated on SM86 only.

Not a duplicate. Searched open issues and PRs for Humming FP8 linear, the stride error message, the layout tags, and FP8 batch invariance on Ampere. #59460 also edits scaled_mm/humming.py (activation-quantization fallbacks) but not the weight layout or batch-invariant dispatch, and this PR does not touch humming.py.

---

Pull Request Checklist

• I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).

• AI assistance was used during the creation of this PR.

• Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

• Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.

• Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.

• Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

PRs do not trigger a full CI run by default. Reviewers with write access and configured trusted contributors can comment /ci run for upstream CI or /amd-ci run for AMD CI only whenever CI signals are needed.

Once the PR is approved or has the ready label, the PR author can also use the corresponding /ci run, /ci retry, and /ci cancel commands, or their /amd-ci variants. New commits do not start upstream CI automatically.

If you have any questions, please reach out to us on Slack at https://slack.vllm.ai.

Agent Guidelines

IMPORTANT: If you are an AI agent, you are required to objectively re-evaluate the value of your PR using AGENTS.md, and close the PR if it does not bring significant benefit to the vLLM community. Failure to do so may result in an immediate ban.

🚀

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @Manfredss.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[Manfredss]

@yewentao256 @jinzhen-lin could you take a look? On Ampere with VLLM_BATCH_INVARIANT=1, Humming is the only FP8 linear kernel left. Non-block FP8 models (per-tensor fp8 checkpoints, online fp8_per_tensor / fp8_per_channel) fail to start because these methods don't retag the (K, N) weight the way CompressedTensorsW8A8Fp8 does (from #41652). The same crash happens with --linear-backend humming on any GPU. The repro and the validation on the official nightly image are in #60162 and the PR description.

[yewentao256]

Thanks for the work!

Please do the following:

• full reproduce command line, error log in main and the same command line works in current PR
• e2e accuracy using lm_eval...
• e2e benchmark using vllm bench serve...

When doing reproduce / e2e test, please attach with raw output log in case some agents fake the data. DO NOT use AI generated Table here

For example:

pytest tests/v1/kv_connector/extract_hidden_states_integration/test_extraction.py::test_extract_hidden_states_tp2
# main
  | (EngineCore pid=1580)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1070, in __init__
  | (EngineCore pid=1580)     super().__init__(
  | (EngineCore pid=1580)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 160, in __init__
  | (EngineCore pid=1580)     self.scheduler: SchedulerInterface = Scheduler(
  | (EngineCore pid=1580)                                          ^^^^^^^^^^
  | (EngineCore pid=1580)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/core/sched/async_scheduler.py", line 14, in __init__
  | (EngineCore pid=1580)     super().__init__(*args, **kwargs)
  | (EngineCore pid=1580)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/core/sched/scheduler.py", line 277, in __init__
  | (EngineCore pid=1580)     self.kv_cache_manager = KVCacheManager(
  | (EngineCore pid=1580)                             ^^^^^^^^^^^^^^^
  | (EngineCore pid=1580)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/core/kv_cache_manager.py", line 151, in __init__
  | (EngineCore pid=1580)     self.coordinator = get_kv_cache_coordinator(
  | (EngineCore pid=1580)                        ^^^^^^^^^^^^^^^^^^^^^^^^^
  | (EngineCore pid=1580)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/core/kv_cache_coordinator.py", line 891, in get_kv_cache_coordinator
  | (EngineCore pid=1580)     return HybridKVCacheCoordinator(
  | (EngineCore pid=1580)            ^^^^^^^^^^^^^^^^^^^^^^^^^
  | (EngineCore pid=1580)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/core/kv_cache_coordinator.py", line 563, in __init__
  | (EngineCore pid=1580)     assert all(
  | (EngineCore pid=1580)            ^^^^
  | (EngineCore pid=1580) AssertionError: Each KV cache group's real block_size must be divisible by hash_block_size. block_sizes=[544, 544, 544, 544, 181], hash_block_size=98464
  | FAILED
  | v1/kv_connector/extract_hidden_states_integration/test_extraction.py::test_extract_hidden_states_tp2

# Now
================= 1 passed, 23 warnings in 174.57s (0:02:54) ==================

"lm_eval --model local-completions --model_args "base_url=http://127.0.0.1:9256/v1/completions,model=$MODEL,num_concurrent=1024" --tasks gsm8k --trust_remote_code"

# Now
|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.9212|±  |0.0074|
|     |       |strict-match    |     5|exact_match|↑  |0.9212|±  |0.0074|

# main
|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.9151|±  |0.0077|
|     |       |strict-match    |     5|exact_match|↑  |0.9158|±  |0.0076|

`lm_eval run --model local-chat-completions --model_args base_url=http://127.0.0.1:9256/v1/chat/completions,model=$MODEL,num_concurrent=1024,seed=256 --tasks aime25 --trust_remote_code --apply_chat_template`
# now
|Tasks |Version|Filter|n-shot|  Metric   |   |Value |   |Stderr|
|------|------:|------|-----:|-----------|---|-----:|---|-----:|
|aime25|      0|none  |     0|exact_match|↑  |0.7333|±  |0.0821|
# main
|Tasks |Version|Filter|n-shot|  Metric   |   |Value |   |Stderr|
|------|------:|------|-----:|-----------|---|-----:|---|-----:|
|aime25|      0|none  |     0|exact_match|↑  |0.7333|±  |0.0821|

vllm bench serve --model zai-org/GLM-5.2-FP8  --dataset-name random --host 127.0.0.1 --port 9256 --random-input-len 8192 --random-output-len 1 --request-rate inf --num-prompts 128 --num-warmups 16

# Now

============ Serving Benchmark Result ============
Successful requests:                     128       
Failed requests:                         0         
Benchmark duration (s):                  3.08      
Total input tokens:                      1048576   
Total generated tokens:                  128       
Request throughput (req/s):              41.50     
Output token throughput (tok/s):         41.50     
Peak output token throughput (tok/s):    76.00     
Peak concurrent requests:                128.00    
Total token throughput (tok/s):          340041.81 
---------------Time to First Token----------------
Mean TTFT (ms):                          1669.15   
Median TTFT (ms):                        1669.02   
P99 TTFT (ms):                           2724.38   
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          0.00      
Median TPOT (ms):                        0.00      
P99 TPOT (ms):                           0.00      
---------------Inter-token Latency----------------
Mean ITL (ms):                           0.00      
Median ITL (ms):                         0.00      
P99 ITL (ms):                            0.00      
==================================================

# main
============ Serving Benchmark Result ============
Successful requests:                     128       
Failed requests:                         0         
Benchmark duration (s):                  3.23      
Total input tokens:                      1048576   
Total generated tokens:                  128       
Request throughput (req/s):              39.63     
Output token throughput (tok/s):         39.63     
Peak output token throughput (tok/s):    50.00     
Peak concurrent requests:                128.00    
Total token throughput (tok/s):          324671.20 
---------------Time to First Token----------------
Mean TTFT (ms):                          1668.69   
Median TTFT (ms):                        1649.00   
P99 TTFT (ms):                           3126.43   
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          0.00      
Median TPOT (ms):                        0.00      
P99 TPOT (ms):                           0.00      
---------------Inter-token Latency----------------
Mean ITL (ms):                           0.00      
Median ITL (ms):                         0.00      
P99 ITL (ms):                            0.00      
==================================================

[Manfredss]

@yewentao256 thanks, here are the runs. One RTX 3090 (sm86, driver 580.65.06, CUDA 13.0) on Runpod, image vllm/vllm-openai:nightly-e68efc685ab1e6f1eda004ced7157e3ca411da2f. "main" is the image as shipped; "PR" is the same image with only this PR's changed files copied over the installed package (sha256 of every swapped file is logged per run).

Reproduce. On Ampere with VLLM_BATCH_INVARIANT=1, Humming is the only FP8 linear kernel left, so a per-tensor FP8 checkpoint can't start on main.

$ env VLLM_BATCH_INVARIANT=1 vllm serve neuralmagic/Meta-Llama-3-8B-Instruct-FP8 --port 8000 --max-model-len 4096 --gpu-memory-utilization 0.9 --no-enable-prefix-caching

# main
(EngineCore pid=9547) INFO 10-20 06:25:05 [__init__.py:808] Selected HummingFP8ScaledMMLinearKernel for Fp8LinearMethod
(EngineCore pid=9547) ERROR 10-20 06:25:09 [core.py:1543]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/kernels/linear/scaled_mm/humming.py", line 82, in process_weights_after_loading
(EngineCore pid=9547) ERROR 10-20 06:25:09 [core.py:1543]     convert_linear_layer_to_humming_standard(layer=layer, name_map=name_map)
(EngineCore pid=9547) ERROR 10-20 06:25:09 [core.py:1543]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/quantization/utils/humming/linear.py", line 39, in convert_linear_layer_to_humming_standard
(EngineCore pid=9547) ERROR 10-20 06:25:09 [core.py:1543] RuntimeError: self.stride(-1) must be 1 to view Float8_e4m3fn as Int (different element sizes), but got 4096

# PR
(EngineCore pid=12519) INFO 10-20 06:45:57 [__init__.py:808] Selected HummingFP8ScaledMMLinearKernel for Fp8LinearMethod
INFO:     Application startup complete.

Batch invariance on the PR server started with that command:

$ python3 repro_bi.py --model neuralmagic/Meta-Llama-3-8B-Instruct-FP8 --rounds 3
round 0: A=954 tokens, B=1915 tokens | A alone vs A alone: first token diff at None, max |dlogprob| on agreeing positions 0.0000 | A batched with B vs A alone: first token diff at None, max |dlogprob| on agreeing positions 0.0000
round 1: A=954 tokens, B=1915 tokens | A alone vs A alone: first token diff at None, max |dlogprob| on agreeing positions 0.0000 | A batched with B vs A alone: first token diff at None, max |dlogprob| on agreeing positions 0.0000
round 2: A=954 tokens, B=1915 tokens | A alone vs A alone: first token diff at None, max |dlogprob| on agreeing positions 0.0000 | A batched with B vs A alone: first token diff at None, max |dlogprob| on agreeing positions 0.0000

repro_bi.py (greedy; prompt A alone twice, then A in the same batch as a longer prompt B)

import argparse, random, threading, time
import httpx


def prompt(seed, n_words):
    rng = random.Random(seed)
    return " ".join(f"w{rng.randrange(10000)}" for _ in range(n_words))


def gen(base, model, text, n):
    r = httpx.post(base + "/v1/completions", json={"model": model, "prompt": text, "max_tokens": n,
                                                    "temperature": 0.0, "logprobs": 1}, timeout=600)
    r.raise_for_status()
    ch = r.json()["choices"][0]
    return ch["logprobs"]["tokens"], ch["logprobs"]["token_logprobs"], r.json()["usage"]["prompt_tokens"]


def diff(x, y):
    n = min(len(x[0]), len(y[0]))
    first = next((i for i in range(n) if x[0][i] != y[0][i]), None)
    agree = n if first is None else first
    gap = max((abs(x[1][i] - y[1][i]) for i in range(agree)), default=0.0)
    return f"first token diff at {first}, max |dlogprob| on agreeing positions {gap:.4f}"


ap = argparse.ArgumentParser()
ap.add_argument("--base-url", default="http://127.0.0.1:8000")
ap.add_argument("--model", default="Qwen/Qwen3-8B-AWQ")
ap.add_argument("--rounds", type=int, default=3)
a = ap.parse_args()
A, B = prompt(1, 330), prompt(2, 660)
for r in range(a.rounds):
    solo = gen(a.base_url, a.model, A, 48)
    solo2 = gen(a.base_url, a.model, A, 48)
    out = {}
    tb = threading.Thread(target=lambda: out.__setitem__("B", gen(a.base_url, a.model, B, 48)))
    ta = threading.Thread(target=lambda: out.__setitem__("A", gen(a.base_url, a.model, A, 48)))
    tb.start(); time.sleep(0.005); ta.start(); tb.join(); ta.join()
    print(f"round {r}: A={solo[2]} tokens, B={out['B'][2]} tokens | A alone vs A alone: {diff(solo2, solo)} | "
          f"A batched with B vs A alone: {diff(out['A'], solo)}")

Unit test.

$ cd tests && python3 -m pytest -v -p no:cacheprovider quantization/test_fp8.py::test_model_load_and_run

# main
quantization/test_fp8.py::test_model_load_and_run[False-auto-neuralmagic/Meta-Llama-3-8B-Instruct-FP8-KV] PASSED [ 16%]
quantization/test_fp8.py::test_model_load_and_run[False-auto-nm-testing/Qwen2-0.5B-Instruct-FP8-SkipQKV] SKIPPED [ 33%]
quantization/test_fp8.py::test_model_load_and_run[False-marlin-neuralmagic/Meta-Llama-3-8B-Instruct-FP8-KV] PASSED [ 50%]
quantization/test_fp8.py::test_model_load_and_run[False-marlin-nm-testing/Qwen2-0.5B-Instruct-FP8-SkipQKV] SKIPPED [ 66%]
quantization/test_fp8.py::test_model_load_and_run[False-humming-neuralmagic/Meta-Llama-3-8B-Instruct-FP8-KV] FAILED [ 83%]
quantization/test_fp8.py::test_model_load_and_run[False-humming-nm-testing/Qwen2-0.5B-Instruct-FP8-SkipQKV] SKIPPED [100%]
E               RuntimeError: self.stride(-1) must be 1 to view Float8_e4m3fn as Int (different element sizes), but got 4096
FAILED quantization/test_fp8.py::test_model_load_and_run[False-humming-neuralmagic/Meta-Llama-3-8B-Instruct-FP8-KV]
============= 1 failed, 2 passed, 3 skipped, 17 warnings in 36.97s =============

# PR
quantization/test_fp8.py::test_model_load_and_run[False-auto-neuralmagic/Meta-Llama-3-8B-Instruct-FP8-KV] PASSED [ 16%]
quantization/test_fp8.py::test_model_load_and_run[False-auto-nm-testing/Qwen2-0.5B-Instruct-FP8-SkipQKV] SKIPPED [ 33%]
quantization/test_fp8.py::test_model_load_and_run[False-marlin-neuralmagic/Meta-Llama-3-8B-Instruct-FP8-KV] PASSED [ 50%]
quantization/test_fp8.py::test_model_load_and_run[False-marlin-nm-testing/Qwen2-0.5B-Instruct-FP8-SkipQKV] SKIPPED [ 66%]
quantization/test_fp8.py::test_model_load_and_run[False-humming-neuralmagic/Meta-Llama-3-8B-Instruct-FP8-KV] PASSED [ 83%]
quantization/test_fp8.py::test_model_load_and_run[False-humming-nm-testing/Qwen2-0.5B-Instruct-FP8-SkipQKV] SKIPPED [100%]
================== 3 passed, 3 skipped, 17 warnings in 23.57s ==================

The two SKIPPED cases are the existing Checkpoint removed from HF. marker.

lm_eval (gsm8k). main can't start with VLLM_BATCH_INVARIANT=1, so there is no main number for that mode. Without it, both main and the PR pick Marlin on Ampere, which this PR doesn't touch:

(EngineCore pid=10240) INFO 10-20 06:25:54 [__init__.py:808] Selected MarlinFP8ScaledMMLinearKernel for Fp8LinearMethod
(EngineCore pid=11428) INFO 10-20 06:35:59 [__init__.py:808] Selected MarlinFP8ScaledMMLinearKernel for Fp8LinearMethod

$ lm_eval --model local-completions --model_args base_url=http://127.0.0.1:8000/v1/completions,model=neuralmagic/Meta-Llama-3-8B-Instruct-FP8,num_concurrent=64,max_retries=3,tokenized_requests=False --tasks gsm8k

# main, `vllm serve neuralmagic/Meta-Llama-3-8B-Instruct-FP8 --port 8000 --max-model-len 4096 --gpu-memory-utilization 0.9`
local-completions ({'base_url': 'http://127.0.0.1:8000/v1/completions', 'model': 'neuralmagic/Meta-Llama-3-8B-Instruct-FP8', 'num_concurrent': 64, 'max_retries': 3, 'tokenized_requests': False}), gen_kwargs: ({}), limit: None, num_fewshot: None, batch_size: 1
|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.7642|±  |0.0117|
|     |       |strict-match    |     5|exact_match|↑  |0.7657|±  |0.0117|

# PR, same command
local-completions ({'base_url': 'http://127.0.0.1:8000/v1/completions', 'model': 'neuralmagic/Meta-Llama-3-8B-Instruct-FP8', 'num_concurrent': 64, 'max_retries': 3, 'tokenized_requests': False}), gen_kwargs: ({}), limit: None, num_fewshot: None, batch_size: 1
|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.7642|±  |0.0117|
|     |       |strict-match    |     5|exact_match|↑  |0.7657|±  |0.0117|

# PR, `env VLLM_BATCH_INVARIANT=1 vllm serve neuralmagic/Meta-Llama-3-8B-Instruct-FP8 --port 8000 --max-model-len 4096 --gpu-memory-utilization 0.9 --no-enable-prefix-caching` (Humming)
local-completions ({'base_url': 'http://127.0.0.1:8000/v1/completions', 'model': 'neuralmagic/Meta-Llama-3-8B-Instruct-FP8', 'num_concurrent': 64, 'max_retries': 3, 'tokenized_requests': False}), gen_kwargs: ({}), limit: None, num_fewshot: None, batch_size: 1
|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.7544|±  |0.0119|
|     |       |strict-match    |     5|exact_match|↑  |0.7559|±  |0.0118|

vllm bench serve.

$ vllm bench serve --model neuralmagic/Meta-Llama-3-8B-Instruct-FP8 --host 127.0.0.1 --port 8000 --dataset-name random --random-input-len 1024 --random-output-len 128 --num-prompts 200 --request-rate inf --num-warmups 16

# main, default
============ Serving Benchmark Result ============
Successful requests:                     200       
Failed requests:                         0         
Benchmark duration (s):                  76.96     
Total input tokens:                      204800    
Total generated tokens:                  25600     
Request throughput (req/s):              2.60      
Output token throughput (tok/s):         332.62    
Peak output token throughput (tok/s):    1545.00   
Peak concurrent requests:                200.00    
Total token throughput (tok/s):          2993.58   
---------------Time to First Token----------------
Mean TTFT (ms):                          34992.83  
Median TTFT (ms):                        36523.35  
P99 TTFT (ms):                           73483.20  
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          195.12    
Median TPOT (ms):                        230.61    
P99 TPOT (ms):                           279.55    
---------------Inter-token Latency----------------
Mean ITL (ms):                           195.12    
Median ITL (ms):                         52.93     
P99 ITL (ms):                            612.44    
==================================================

# PR, default
============ Serving Benchmark Result ============
Successful requests:                     200       
Failed requests:                         0         
Benchmark duration (s):                  77.79     
Total input tokens:                      204800    
Total generated tokens:                  25600     
Request throughput (req/s):              2.57      
Output token throughput (tok/s):         329.11    
Peak output token throughput (tok/s):    1684.00   
Peak concurrent requests:                200.00    
Total token throughput (tok/s):          2962.00   
---------------Time to First Token----------------
Mean TTFT (ms):                          34893.11  
Median TTFT (ms):                        37025.00  
P99 TTFT (ms):                           74503.54  
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          208.50    
Median TPOT (ms):                        245.52    
P99 TPOT (ms):                           300.56    
---------------Inter-token Latency----------------
Mean ITL (ms):                           208.50    
Median ITL (ms):                         55.03     
P99 ITL (ms):                            623.68    
==================================================

# PR, VLLM_BATCH_INVARIANT=1
============ Serving Benchmark Result ============
Successful requests:                     200       
Failed requests:                         0         
Benchmark duration (s):                  75.96     
Total input tokens:                      204800    
Total generated tokens:                  25600     
Request throughput (req/s):              2.63      
Output token throughput (tok/s):         337.01    
Peak output token throughput (tok/s):    1447.00   
Peak concurrent requests:                200.00    
Total token throughput (tok/s):          3033.13   
---------------Time to First Token----------------
Mean TTFT (ms):                          33459.49  
Median TTFT (ms):                        34980.63  
P99 TTFT (ms):                           70776.05  
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          190.32    
Median TPOT (ms):                        219.35    
P99 TPOT (ms):                           271.91    
---------------Inter-token Latency----------------
Mean ITL (ms):                           190.32    
Median ITL (ms):                         58.37     
P99 ITL (ms):                            565.69    
==================================================

Full server logs for every run are kept and I can attach any of them.

[yewentao256]

Thanks for the work!

[yewentao256]

Please take a look at this AI generated comment:

One question about the PTPC path: `Fp8PtpcOnlineLinearMethod` requires per-token FP8 activation quantization, but Humming appears to use a weight-only path with FP16/BF16 activations.

With this layout fix, PTPC can now successfully run through Humming. Does Humming preserve the requested activation quantization semantics here, or should we reject this combination as we already do for Marlin?

[Manfredss]

Good catch, it doesn't. HummingFP8ScaledMMLinearKernel.process_weights_after_loading calls prepare_humming_linear_layer_config() without an input schema. Humming then uses its default HummingInputSchema() (a_dtype=None), which means 16-bit activations with input quantization disabled. That is W8A16, the same as Marlin FP8. With the layout fix alone (the earlier version of this PR), PTPC would silently run as weight-only FP8 whenever it lands on Humming. That happens in two cases:

• on Ampere with VLLM_BATCH_INVARIANT=1, where the other FP8 kernels need SM89+ and Marlin opts out;
• with --linear-backend humming.

970068e adds Humming to the existing Marlin check in Fp8PtpcOnlineLinearMethod.create_weights, so PTPC now raises the same ValueError, and the message names the selected kernel. The commit also drops the layout tag that the earlier version set in PTPC, since PTPC no longer reaches Humming. Fp8LinearMethod and fp8_per_tensor still run on Humming. They already fall back to W8A16 Marlin on GPUs without FP8 support, so Humming matches their existing behavior.

A40, nightly 00b6c4c5:

VLLM_BATCH_INVARIANT=1 vllm serve Qwen/Qwen3-8B --port 8000 --max-model-len 4096 --gpu-memory-utilization 0.9 --quantization fp8_per_channel

main

(EngineCore pid=5214) INFO 10-26 21:37:40 [__init__.py:817] Selected HummingFP8ScaledMMLinearKernel for Fp8PtpcOnlineLinearMethod
(EngineCore pid=5214) ERROR 10-26 21:37:42 [core.py:1574] RuntimeError: self.stride(-1) must be 1 to view Float8_e4m3fn as Int (different element sizes), but got 12288

layout fix applied to PTPC too, without the new check

(EngineCore pid=5749) INFO 10-26 21:38:22 [__init__.py:817] Selected HummingFP8ScaledMMLinearKernel for Fp8PtpcOnlineLinearMethod
(APIServer pid=5423) INFO 10-26 21:39:27 [entry.py:139] Starting vLLM server on http://0.0.0.0:8000

this PR

(EngineCore pid=6497) INFO 10-26 21:40:02 [__init__.py:817] Selected HummingFP8ScaledMMLinearKernel for Fp8PtpcOnlineLinearMethod
(EngineCore pid=6497) ERROR 10-26 21:40:03 [core.py:1574] ValueError: FP8 PTPC online quant requires a kernel that honors per-token activation quantization; HummingFP8ScaledMMLinearKernel is W8A16 weight-only. Requires SM89+ for Cutlass FP8 or ROCm MI3xx for rowwise scaled_mm.

this PR, without BI: vllm serve Qwen/Qwen3-8B --port 8000 --max-model-len 4096 --gpu-memory-utilization 0.9 --quantization fp8_per_channel --linear-backend humming

(EngineCore pid=7034) INFO 10-26 21:40:41 [__init__.py:817] Selected HummingFP8ScaledMMLinearKernel for Fp8PtpcOnlineLinearMethod
(EngineCore pid=7034) ERROR 10-26 21:40:42 [core.py:1574] ValueError: FP8 PTPC online quant requires a kernel that honors per-token activation quantization; HummingFP8ScaledMMLinearKernel is W8A16 weight-only. Requires SM89+ for Cutlass FP8 or ROCm MI3xx for rowwise scaled_mm.

main, same command with --linear-backend marlin, for comparison

(EngineCore pid=7536) INFO 10-26 21:41:17 [__init__.py:817] Selected MarlinFP8ScaledMMLinearKernel for Fp8PtpcOnlineLinearMethod
(EngineCore pid=7536) ERROR 10-26 21:41:18 [core.py:1574] ValueError: FP8 PTPC online quant requires a kernel that honors per-token activation quantization; MarlinFP8 is W8A16 weight-only. Requires SM89+ for Cutlass FP8 or ROCm MI3xx for rowwise scaled_mm.

[Manfredss]

Merged main and added two commits:

• 970068e: PTPC rejects Humming, see the inline thread.
• 606a1e6: since [Quantization] replace_parameter keeps the parameter object by default #62326, replace_parameter keeps the existing ModelWeightParameter, whose input_dim/output_dim are read-only, so the old tags failed on the new nightly with AttributeError: property 'input_dim' of 'ModelWeightParameter' object has no setter. The layout is now set with layer.processed_weight_dims = (0, 1), like the other FP8 methods since [Quantization] replace_parameter keeps the parameter object by default #62326.

Re-run on one A40 (sm86, driver 595.91.07, CUDA 13.2) on Runpod, image vllm/vllm-openai:nightly-00b6c4c565a8732c131e1f452d4a17803bb0842b (vllm 0.32.1rc1.dev505, torch 2.14.0+cu130, humming-kernels 0.1.18). "main" is the image as shipped; "PR" is this branch's diff applied to the installed package. repro_bi.py is the script from my earlier comment.

Unit test

pytest -v tests/quantization/test_fp8.py::test_model_load_and_run

main

quantization/test_fp8.py::test_model_load_and_run[False-auto-neuralmagic/Meta-Llama-3-8B-Instruct-FP8-KV] PASSED
quantization/test_fp8.py::test_model_load_and_run[False-auto-nm-testing/Qwen2-0.5B-Instruct-FP8-SkipQKV] SKIPPED
quantization/test_fp8.py::test_model_load_and_run[False-marlin-neuralmagic/Meta-Llama-3-8B-Instruct-FP8-KV] PASSED
quantization/test_fp8.py::test_model_load_and_run[False-marlin-nm-testing/Qwen2-0.5B-Instruct-FP8-SkipQKV] SKIPPED
quantization/test_fp8.py::test_model_load_and_run[False-humming-neuralmagic/Meta-Llama-3-8B-Instruct-FP8-KV] FAILED
quantization/test_fp8.py::test_model_load_and_run[False-humming-nm-testing/Qwen2-0.5B-Instruct-FP8-SkipQKV] SKIPPED
E               RuntimeError: self.stride(-1) must be 1 to view Float8_e4m3fn as Int (different element sizes), but got 4096
============= 1 failed, 2 passed, 3 skipped, 17 warnings in 12.33s =============

PR

quantization/test_fp8.py::test_model_load_and_run[False-auto-neuralmagic/Meta-Llama-3-8B-Instruct-FP8-KV] PASSED
quantization/test_fp8.py::test_model_load_and_run[False-auto-nm-testing/Qwen2-0.5B-Instruct-FP8-SkipQKV] SKIPPED
quantization/test_fp8.py::test_model_load_and_run[False-marlin-neuralmagic/Meta-Llama-3-8B-Instruct-FP8-KV] PASSED
quantization/test_fp8.py::test_model_load_and_run[False-marlin-nm-testing/Qwen2-0.5B-Instruct-FP8-SkipQKV] SKIPPED
quantization/test_fp8.py::test_model_load_and_run[False-humming-neuralmagic/Meta-Llama-3-8B-Instruct-FP8-KV] PASSED
quantization/test_fp8.py::test_model_load_and_run[False-humming-nm-testing/Qwen2-0.5B-Instruct-FP8-SkipQKV] SKIPPED
================== 3 passed, 3 skipped, 17 warnings in 34.15s ==================

Llama-3-8B-Instruct-FP8 (fp8 checkpoint), batch-invariant, PR

VLLM_BATCH_INVARIANT=1 vllm serve neuralmagic/Meta-Llama-3-8B-Instruct-FP8 --port 8000 --max-model-len 4096 --gpu-memory-utilization 0.9 --no-enable-prefix-caching

On main this command crashes at startup with the stride error (earlier comment).

$ python3 repro_bi.py --model neuralmagic/Meta-Llama-3-8B-Instruct-FP8 --rounds 3
round 0: A=954 tokens, B=1915 tokens | A alone vs A alone: first token diff at None, max |dlogprob| on agreeing positions 0.0000 | A batched with B vs A alone: first token diff at None, max |dlogprob| on agreeing positions 0.0000
round 1: A=954 tokens, B=1915 tokens | A alone vs A alone: first token diff at None, max |dlogprob| on agreeing positions 0.0000 | A batched with B vs A alone: first token diff at None, max |dlogprob| on agreeing positions 0.0000
round 2: A=954 tokens, B=1915 tokens | A alone vs A alone: first token diff at None, max |dlogprob| on agreeing positions 0.0000 | A batched with B vs A alone: first token diff at None, max |dlogprob| on agreeing positions 0.0000

$ lm_eval --model local-completions --model_args base_url=http://127.0.0.1:8000/v1/completions,model=neuralmagic/Meta-Llama-3-8B-Instruct-FP8,num_concurrent=64,max_retries=3,tokenized_requests=False --tasks gsm8k
local-completions ({'base_url': 'http://127.0.0.1:8000/v1/completions', 'model': 'neuralmagic/Meta-Llama-3-8B-Instruct-FP8', 'num_concurrent': 64, 'max_retries': 3, 'tokenized_requests': False}), gen_kwargs: ({}), limit: None, num_fewshot: None, batch_size: 1
|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.7544|±  |0.0119|
|     |       |strict-match    |     5|exact_match|↑  |0.7559|±  |0.0118|

vllm bench serve

$ vllm bench serve --model neuralmagic/Meta-Llama-3-8B-Instruct-FP8 --host 127.0.0.1 --port 8000 --dataset-name random --random-input-len 1024 --random-output-len 128 --num-prompts 200 --request-rate inf --num-warmups 16
============ Serving Benchmark Result ============
Successful requests:                     200       
Failed requests:                         0         
Benchmark duration (s):                  45.83     
Total input tokens:                      204800    
Total generated tokens:                  25600     
Request throughput (req/s):              4.36      
Output token throughput (tok/s):         558.60    
Peak output token throughput (tok/s):    2506.00   
Peak concurrent requests:                200.00    
Total token throughput (tok/s):          5027.38   
---------------Time to First Token----------------
Mean TTFT (ms):                          17868.71  
Median TTFT (ms):                        16956.46  
P99 TTFT (ms):                           38314.83  
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          203.59    
Median TPOT (ms):                        213.15    
P99 TPOT (ms):                           313.60    
---------------Inter-token Latency----------------
Mean ITL (ms):                           203.59    
Median ITL (ms):                         86.32     
P99 ITL (ms):                            449.88    
==================================================

Qwen3-8B fp8_per_tensor, batch-invariant, PR

VLLM_BATCH_INVARIANT=1 vllm serve Qwen/Qwen3-8B --port 8000 --max-model-len 4096 --gpu-memory-utilization 0.9 --quantization fp8_per_tensor --no-enable-prefix-caching

$ python3 repro_bi.py --model Qwen/Qwen3-8B --rounds 3
round 0: A=1610 tokens, B=3227 tokens | A alone vs A alone: first token diff at None, max |dlogprob| on agreeing positions 0.0000 | A batched with B vs A alone: first token diff at None, max |dlogprob| on agreeing positions 0.0000
round 1: A=1610 tokens, B=3227 tokens | A alone vs A alone: first token diff at None, max |dlogprob| on agreeing positions 0.0000 | A batched with B vs A alone: first token diff at None, max |dlogprob| on agreeing positions 0.0000
round 2: A=1610 tokens, B=3227 tokens | A alone vs A alone: first token diff at None, max |dlogprob| on agreeing positions 0.0000 | A batched with B vs A alone: first token diff at None, max |dlogprob| on agreeing positions 0.0000

$ lm_eval --model local-completions --model_args base_url=http://127.0.0.1:8000/v1/completions,model=Qwen/Qwen3-8B,num_concurrent=64,max_retries=3,tokenized_requests=False --tasks gsm8k
local-completions ({'base_url': 'http://127.0.0.1:8000/v1/completions', 'model': 'Qwen/Qwen3-8B', 'num_concurrent': 64, 'max_retries': 3, 'tokenized_requests': False}), gen_kwargs: ({}), limit: None, num_fewshot: None, batch_size: 1
|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.8673|±  |0.0093|
|     |       |strict-match    |     5|exact_match|↑  |0.8635|±  |0.0095|

vllm bench serve

$ vllm bench serve --model Qwen/Qwen3-8B --host 127.0.0.1 --port 8000 --dataset-name random --random-input-len 1024 --random-output-len 128 --num-prompts 200 --request-rate inf --num-warmups 16
============ Serving Benchmark Result ============
Successful requests:                     200       
Failed requests:                         0         
Benchmark duration (s):                  48.23     
Total input tokens:                      204800    
Total generated tokens:                  25600     
Request throughput (req/s):              4.15      
Output token throughput (tok/s):         530.77    
Peak output token throughput (tok/s):    2186.00   
Peak concurrent requests:                200.00    
Total token throughput (tok/s):          4776.89   
---------------Time to First Token----------------
Mean TTFT (ms):                          18377.23  
Median TTFT (ms):                        17339.22  
P99 TTFT (ms):                           39926.62  
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          216.93    
Median TPOT (ms):                        227.74    
P99 TPOT (ms):                           329.01    
---------------Inter-token Latency----------------
Mean ITL (ms):                           216.93    
Median ITL (ms):                         95.10     
P99 ITL (ms):                            476.07    
==================================================

[yewentao256]

LGTM, thanks for the work!

[github-actions]

✅ @Manfredss, CI is now available for this PR.

• /ci run starts upstream CI; /amd-ci run starts AMD CI only.
• Run commands test your branch with every commit currently on its upstream target branch. If it is behind, the bot merges the target branch into it first (pull before pushing again). Append --allow-stale to a run command to test an outdated branch as is, at your own risk.
• /ci retry retries failed jobs in the CI build for the current PR head. If the current head has no CI build, it starts a new CI build for the current head containing only jobs that failed in the latest earlier CI build for this PR.
• /amd-ci retry retries failed jobs in AMD CI for the current PR head. Use /amd-ci run when the current head has no AMD CI build.
• /ci cancel cancels scheduled or running CI builds for this PR branch; /amd-ci cancel does the same for AMD CI only.

[Manfredss]

/ci run

[github-actions]

✅ Triggered Buildkite CI #97784 for commit 95d2bdc3483f.

[yewentao256]

/ci run

[github-actions]

✅ Triggered Buildkite CI #98070 for commit 19a47745558f.

[Manfredss]

/ci run

[github-actions]

✅ Triggered Buildkite CI #98101 for commit 833c5f328d03.

ℹ️ This PR is 2 commits behind upstream main. Merged upstream main into this branch as 833c5f328d03 before starting CI. Pull the branch before pushing more changes.

[Manfredss]

/ci run

[github-actions]

✅ Triggered Buildkite CI #98201 for commit d88d84dfc888.