[devtyagi3909]

fixes #42369. drops fp8 kv scales at load time if runtime dtype is nvfp4. without this they get systematically mis-scaled (e.g. amax/448 vs amax/6). falling back to dynamic scales works correctly.

---

CI States

Latest PR Test (Base): ❌ Run #39578296118
Latest PR Test (Extra): ❌ Run #39578296451
Latest PR Test (AMD ROCm 10): ❌ Run #39578296354

[devtyagi3909]

fixed lint. humanized the log message.

[devtyagi3909]

generalised the check per your suggestion to catch both directions (fp4 runtime with fp8 checkpoint scales, and vice versa). also ensured the filtered scales degrade perfectly to dynamic. pushed the update.

would be awesome if you could run the full harness on the 2x5060 rig to validate this end-to-end!

[waynehacking8]

End to end on SM120 this fixes NVFP4 KV: the needle test at 8K to 96K goes from 6/36 to 36/36, level with the default FP8 launch.
Setup: RTX PRO 6000 Blackwell Max-Q, TP1, the report's checkpoint (unsloth/Qwen3.8-27B-NVFP4) and attention flags, main 4fc86f73c with and without this head; sglang-kernel 0.4.10, FlashInfer 0.7.2, torch 2.14.1.

KV dtype	Tree	Needle 8K to 96K	GSM8K strict	NLL, last 2K tokens of 32K windows
NVFP4	main	6/36	0.640	3.46
NVFP4	this PR	36/36	0.964	1.82
FP8 (default launch)	main	18/18	0.968	1.82
FP8 (default launch)	this PR	18/18	0.972	1.82
BF16	main	2/18	0.668	3.47
BF16	this PR	18/18	0.960	1.82

With this PR NVFP4 is level with the default FP8 launch on all three measures, and main's BF16 row shows the same breakage.

Two gaps against chengye75-star's invariant:

1. FP8 launch: the filter runs outside the mismatch check, so the FP8 launch also gets scales of 1.0 with no warning. That is harmless on this model (the FP8 rows above).
2. BF16 KV: main also breaks BF16 KV (needle 2/18) because the FlashInfer prefill backend passes k_scale/v_scale for any KV dtype (flashinfer_backend.py:1397, :1459, :1532 on 4fc86f73c). The PR fixes it (18/18) only through that unconditional filter, so BF16 should count as a mismatch once the filter moves inside the check.

Commands

python -m sglang.launch_server --model-path unsloth/Qwen3.8-27B-NVFP4 --kv-cache-dtype {nvfp4,auto,bf16} \
  --tp 1 --context-length 131072 --page-size 64 --prefill-attention-backend flashinfer \
  --decode-attention-backend trtllm_mha --disable-prefill-cuda-graph --disable-radix-cache \
  --mem-fraction-static 0.80 --max-running-requests 32
# needle: pass key in 8K/32K/96K WikiText-2 haystacks at depth 0.1/0.5/0.9, greedy chat, thinking off
python s42419_needle.py <port> <served> <model> out.json 8192,32768,98304 0.1,0.5,0.9 2 <rounds>
# prompt logprobs, 4 windows each of 32K and 64K, last 2048 positions
python s42419_logprob.py <port> <model> out.json 32768,65536 4 2048
# chat GSM8K, 250 questions, concurrency 32
python gsm8k_chat.py <port> <served> out.json 250 32

The job script, the client scripts, the per-arm outputs and the probe of the filter are in https://gist.github.com/waynehacking8/6081b45e2119aace7844e2d5988a491f.

[devtyagi3909]

thanks for verifying on sm120. rebased on latest main and addressed both gaps: moved the filter inside the mismatch block so fp8 launches preserve checkpoint scales, and added the bf16/unquantized mismatch check so static scales get dropped for bf16 runtimes as well. added unit test coverage in test_default_model_loader.py.

[waynehacking8]

With the default --kv-cache-dtype auto this drops the scales: expected_bits stays None for "auto", but configure_kv_cache_dtype later resolves auto to fp8_e4m3 for compressed-tensors checkpoints with a supported kv_cache_scheme, so the FP8 pool runs with scale 1.0. A probe of the filter shows the scales kept on base eb4b167d27, dropped on this head, and kept again with the change below (bfloat16 still drops them): https://gist.github.com/waynehacking8/0685c4403346f321f0896860aa882f42. A (8, "auto", False) case with kv_cache_quant_algo="FP8" in the new test would cover it.

Suggested change

	if runtime_kv_dtype:
	if runtime_kv_dtype == "auto":
	# configure_kv_cache_dtype resolves auto to an FP8 pool here.
	algo = getattr(quant_config, "kv_cache_quant_algo", None)
	if isinstance(algo, str) and algo.upper() == "FP8":
	expected_bits = 8
	elif runtime_kv_dtype:

[devtyagi3909]

i see the catch on the auto resolution for compressed-tensors. applied the suggestion and added the (8, "auto", False) test case with kv_cache_quant_algo="FP8". pushed.