Load exact GLM Spark projections and NVFP4 MTP from the QAD source

.lil/changes/vllm-841.json

@@ -0,0 +1,14 @@
+{
+  "schema": "local-inference-release-change/v1",
+  "id": "vllm-841",
+  "category": "fix",
+  "summary": "Bound InstantTensor staging and retain deferred checkpoint tensor ownership",
+  "models": ["Kimi-K3"],
+  "compatibility": "No user action required.",
+  "details": [
+    "Limits temporary checkpoint loading memory and preserves tensors retained by deferred layer processing.",
+    "CPU staging does not place inference weights on the CPU."
+  ],
+  "pull_requests": [841],
+  "requires": []
+}

.lil/changes/vllm-glm-spark-online-weights.json

@@ -0,0 +1,22 @@
+{
+  "schema": "local-inference-release-change/v1",
+  "id": "vllm-glm-spark-online-weights",
+  "category": "feature",
+  "summary": "Quantize the GLM Spark projection set and BF16 MTP experts during loading",
+  "models": [
+    "GLM-5.3-Flash"
+  ],
+  "compatibility": "Opt-in exact target configurations; existing online quantization defaults are unchanged. NVFP4 W4A16 requires a supported BF16-activation MoE backend.",
+  "details": [
+    "Strict targets reject missing projections, incomplete fused groups and independently enabled online quantizers outside the declared set.",
+    "The GLM example selects 531 MXFP8 projections and NVFP4 W4A16 MTP routed experts.",
+    "Layerwise loading releases replaced full-precision parameters before packed-weight preparation, and NVFP4 encoding writes directly into bounded output storage."
+  ],
+  "authors": [
+    "voipmonitor"
+  ],
+  "pull_requests": [
+    965
+  ],
+  "requires": []
+}

docs/features/quantization/glm53_spark_online.md

@@ -0,0 +1,66 @@
+# GLM-5.3-Flash online quantization with Spark projection selection
+
+Use two 96 GB RTX PRO 6000 Blackwell GPUs and a vLLM build containing online
+`nvfp4_a16` and strict quantization targets. Run the following from the vLLM
+repository inside that environment:
+
+```bash
+python examples/features/quantization/glm53_spark_online.py \
+  --output-dir glm-spark-config
+
+PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
+VLLM_USE_V2_MODEL_RUNNER=1 \
+VLLM_B12X_MOE_FP4_FORCE_A16=0 \
+VLLM_ENABLE_PCIE_ALLREDUCE=1 VLLM_PCIE_ALLREDUCE_BACKEND=b12x \
+VLLM_GLM53_SPLIT_TARGET_BLOCK_SIZE=2048 \
+VLLM_GLM53_SPLIT_MAMBA_BLOCK_SIZE=auto \
+VLLM_GLM53_KDA_GATE_SIDE_STREAM=1 VLLM_DISABLE_SHARED_EXPERTS_STREAM=0 \
+VLLM_GLM53_L2_PREFETCH=1 VLLM_LM_HEAD_A16=1 \
+VLLM_GLM53_MTP_DRAFT_HEAD=bf16 \
+VLLM_MTP_NVFP4_LM_HEAD=0 VLLM_MXFP8_LM_HEAD=0 \
+VLLM_GLM53_VISION_MXFP8=0 VLLM_GLM53_FP8_DENSE=0 \
+VLLM_GLM53_ONLINE_DENSE_MXFP8=0 \
+vllm serve local-inference-lab/GLM-5.3-Flash-NVFP4 \
+  --revision cfd47bd7680e68408924df09b179d5bed25b2ae9 \
+  --host 0.0.0.0 --port 8000 \
+  --tensor-parallel-size 2 --pipeline-parallel-size 1 \
+  --quantization modelopt_mixed --load-format safetensors --dtype bfloat16 \
+  --moe-backend b12x --linear-backend b12x --attention-backend B12X \
+  --additional-config '{"glm53_kda_decode_backend":"auto","kda_prefill_backend":"b12x"}' \
+  --quantization-config "$(cat glm-spark-config/main.json)" \
+  --speculative-config "$(cat glm-spark-config/mtp.json)" \
+  --kv-cache-dtype fp8 --kv-cache-memory-bytes 3221225472 \
+  --gpu-memory-utilization 0.93 \
+  --block-size 256 --mamba-cache-mode align \
+  --max-model-len 8192 --max-num-batched-tokens 1024 --max-num-seqs 8 \
+  --max-parallel-prefills 1 --enable-chunked-prefill --enable-prefix-caching \
+  --recurrent-checkpoint-policy request_boundaries \
+  --compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE"}' \
+  --cudagraph-capture-sizes 1 2 4 8 16 32 64 --max-cudagraph-capture-size 64 \
+  --no-enable-flashinfer-autotune --reasoning-parser glm45
+```
+
+The configuration generator creates a directory and refuses to overwrite one
+that already exists. The pinned source revision belongs to the `mtp-bf16`
+branch. vLLM reads the checkpoint and converts weights in memory; it does not
+write a quantized checkpoint.
+
+The adjacent projection manifest selects exactly 531 BF16 matrices from the
+Spark recipe: 520 in the main model and 11 in MTP. These use MXFP8 weights and
+the selected MXFP8 linear backend. MTP routed experts use NVFP4 weights with
+BF16 activations. The LM head, embeddings, vision tower, routers, norms,
+convolutions and the first three dense feed-forward layers retain their source
+representation. Existing main-model NVFP4 experts retain their activation
+recipe.
+
+`strict_targets` defaults to `false` in vLLM; both generated target maps set it
+to `true`. A missing projection, incomplete fused group, or independently
+enabled online quantizer causes startup to fail. Keep the head, vision and
+dense quantization flags shown above disabled when reproducing this selection.
+The recipe matches the Spark **layer selection**; it does not claim byte parity
+with Spark's serialized quantized weights.
+
+MTP uses three speculative tokens by default in the generator. Change this
+with `--num-speculative-tokens 1`, or omit `--speculative-config` from the serve
+command to disable MTP. The 3 GiB cache allocation is per GPU; it is a bounded
+8K-context example, not a million-token capacity configuration.

docs/features/quantization/online.md

@@ -51,6 +51,7 @@ vllm serve Qwen/Qwen3.5-35B-A3B --quantization mxfp4 \
 | `fp8_per_tensor` | fp8_e4m3 data, fp32 per-tensor scale | fp8_e4m3 data, fp32 per-tensor scale | On some GPUs (Ada, Hopper) linear activations use per-token scaling for better performance |
 | `fp8_per_block` | fp8_e4m3 data, fp32 per-128x128-block scale | fp8_e4m3 data, fp32 per-1x128-block scale | |
 | `mxfp8` | fp8_e4m3 data, e8m0 per-1x32-block scale | fp8_e4m3 data, e8m0 per-1x32-block scale | Requires SM 100+ (Blackwell or newer) for w8a8, other GPUs use a w8a16 fallback |
+| `nvfp4_a16` | fp4_e2m1 data, fp8_e4m3 per-1x16-block scale, FP32 global scale per expert | BF16 | MoE only; select a compatible W4A16 backend. B12X supports SM120. |
 | `mxfp4` | fp4_e2m1 data, e8m0 per-1x32-block scale ([OCP MX specs](https://www.opencompute.org/documents/ocp-microscaling-formats-mx-v1-0-spec-final-pdf)) | - linear: fp4_e2m1 data, e8m0 per-1x32-block scale in some backends, or BF16. <br> - MOE: fp4_e2m1 data, e8m0 per-1x32-block scale. | Linear MXFP4 backend is auto-selected per platform, not enforcing activation dtype. Some use BF16 activation. Use `--linear-backend` to pin one (e.g. `--linear-backend flashinfer`). |
 
 ## Advanced Configuration
@@ -177,6 +178,16 @@ llm = LLM(
 
 ### Fine-Grained Per-Layer Quantization Schemes
 
+For a fixed model recipe, set `"strict_targets": true` alongside `targets`.
+The default is `false`. Strict mode accepts exact names, requires every target
+to resolve before loading weights, and rejects independently enabled online
+quantizers outside that set. Fused linears can be named by all their source
+projections. Keep separate target maps for the main model and the draft;
+targets outside a local pipeline partition also cause an error.
+
+The [GLM Spark online-quantization guide](glm53_spark_online.md) includes an
+exact projection manifest and a complete TP2 launch example.
+
 Use the `targets` parameter to apply different online shorthands to different layers, instead of one scheme applied everywhere via `linear`/`moe`. Keys are exact layer names, regex patterns (prefixed with `re:`), or patterns understood by [`fnmatch.fnmatch`](https://docs.python.org/3/library/fnmatch.html#fnmatch.fnmatch); values are shorthand names (`fp8_per_tensor`, `fp8_per_block`, `fp8_per_channel`, `mxfp8`, `int8_per_channel_weight_only`, `nvfp4_per_token`).
 
 Example:

examples/features/quantization/glm53_spark_online.py

@@ -0,0 +1,66 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
+"""Write exact GLM Spark projection and MTP online-quantization configurations.
+
+The adjacent manifest records the selected projections in the pinned Spark
+checkpoint. This generates configuration only; vLLM quantizes BF16 weights
+during loading and does not rewrite checkpoint files.
+"""
+
+import argparse
+import json
+from pathlib import Path
+
+SOURCE_MODEL = "local-inference-lab/GLM-5.3-Flash-NVFP4"
+SOURCE_REVISION = "cfd47bd7680e68408924df09b179d5bed25b2ae9"
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--output-dir", type=Path, required=True)
+    parser.add_argument("--model", default=SOURCE_MODEL)
+    parser.add_argument("--revision", default=SOURCE_REVISION)
+    parser.add_argument("--num-speculative-tokens", type=int, default=3)
+    args = parser.parse_args()
+    manifest = json.loads(
+        Path(__file__).with_name("glm53_spark_targets.json").read_text()
+    )
+    main_targets, mtp_targets = {}, {}
+    source_prefix = "model.language_model."
+    for name in manifest["targets"]:
+        if not name.startswith(source_prefix):
+            raise ValueError(f"Unexpected source projection name: {name}")
+        if name.startswith(source_prefix + "layers.45."):
+            mtp_targets[name.replace(source_prefix, "model.", 1)] = "mxfp8"
+        else:
+            main_targets[name.replace(source_prefix, "language_model.model.", 1)] = (
+                "mxfp8"
+            )
+    if len(main_targets) != 520 or len(mtp_targets) != 11:
+        raise ValueError(
+            "The Spark manifest must define 520 main and 11 MTP projections"
+        )
+    for name in manifest["nvfp4_mtp_targets"]:
+        mtp_targets[name.replace(source_prefix, "model.", 1)] = "nvfp4_a16"
+    configurations = {
+        "main.json": {"targets": main_targets, "strict_targets": True},
+        "mtp.json": {
+            "method": "mtp",
+            "model": args.model,
+            "revision": args.revision,
+            "num_speculative_tokens": args.num_speculative_tokens,
+            "draft_sample_method": "probabilistic",
+            "rejection_sample_method": "standard",
+            "moe_backend": "b12x",
+            "attention_backend": "B12X",
+            "quantization_config": {"targets": mtp_targets, "strict_targets": True},
+        },
+    }
+    args.output_dir.mkdir(parents=True, exist_ok=False)
+    for filename, value in configurations.items():
+        (args.output_dir / filename).write_text(json.dumps(value, indent=2) + "\n")
+    print(f"Wrote {args.output_dir}: 531 MXFP8 projections and NVFP4 W4A16 MTP experts")
+
+
+if __name__ == "__main__":
+    main()