[liusy58]

Overview

Allow the weight-cache daemon to reuse online FP8/MXFP8 linear weights and ModelOpt NVFP4 CUTLASS MoE weights across engine restarts. Related to #56049; this covers a subset of its quantization roadmap.

Claims

• Restore online quantization runtime state without quantizing or transforming shared weights again. FP8 is limited to CUTLASS; MXFP8 requires the selected kernel's pre-processed-weight capability. Requantization from serialized checkpoints and online MoE remain excluded.
• Preserve the unpadded CUTLASS FP8 output width when IPC loading skips weight post-processing.
• Restore ModelOpt NVFP4 CUTLASS MoE kernels without multiplying activation scales into shared weight scales a second time.

Validation

Current head 35711168b0135573234bbaa4bd8f9e8fe32647e8 includes main 914d61fc538a84e8487cb16fe8241a70f6f4bf9e. October 29 agent-run validation after resolving the inherited online-FP8 import conflict with upstream #60163. The merge retains both CUTLASS and Humming imports, the upstream processed-dimension metadata, and upstream rejection of the unsupported Humming per-token/channel path. It adds no Humming IPC capability.

Check	Exact-head result
online-units	108 passed, 685 deselected, 14 warnings in 98.42s (0:01:38)
online-models	4 passed, 30 warnings in 305.90s (0:05:05)
nvfp4-model	1 passed, 18 warnings in 99.83s (0:01:39)
Full feature-file and commit hooks	Passed, including mypy and ruff

Runtime: NVIDIA B300, PyTorch 2.13.0+cu130, process-local CUDA 13.4 compatibility libraries. The two stable-libtorch extensions are from the official CUDA 13.0 wheel for main 914d61fc538a84e8487cb16fe8241a70f6f4bf9e, checked against wheel RECORD hashes and linked only into task worktrees. The shared venv and host driver were unchanged. This does not validate the full upstream PyTorch 2.14 dependency matrix.

Full-model checks use TP1, BF16, eager mode, CUTLASS and TRITON_ATTN, with the existing batch-invariant exact greedy-output comparisons. Each completed case checks ordinary load, missing-daemon fallback, IPC and a second IPC load; both IPC loads forbid fallback. These are output-equivalence smoke checks, not accuracy or performance benchmarks. No multi-GPU, ROCm or other-backend result is claimed.

The temporary local helper calls the existing run_cold_start_and_warm_restart with dataclasses.replace(case, model=local_checkpoint, llm_kwargs={**case.llm_kwargs, "gpu_memory_utilization": 0.12}) and VLLM_BATCH_INVARIANT=1. It changes only checkpoint location and the GPU memory budget; loading paths, prompts and assertions are unchanged. Online cases keep their checked-in 0.2 budget.

NVFP4 checkpoint: nvidia/Qwen3-30B-A3B-NVFP4, pinned 2538ded2a4edb247b4d2b4a8ba24e44bd4c017c3.

Commands for this revision

All commands set HF_HUB_OFFLINE=1, PYTHONPATH to the corresponding worktree and LD_LIBRARY_PATH=/home/liusy/codex-runtime/cuda-compat-13-4/usr/local/cuda-13.4/compat.

CUDA_VISIBLE_DEVICES=0  .venv/bin/python -m pytest tests/model_executor/model_loader/test_weight_cache.py tests/kernels/quantization/test_cutlass_scaled_mm.py -k 'test_cutlass_fp8_gemm_padded or (not cutlass and not cold_start and not d2d_warmup and not online)' -q
CUDA_VISIBLE_DEVICES=4  .venv/bin/python -m pytest tests/model_executor/model_loader/test_weight_cache.py::test_ipc_cache_online_fp8 tests/model_executor/model_loader/test_weight_cache.py::test_ipc_cache_online_blackwell -q
CUDA_VISIBLE_DEVICES=4 CODEX_CACHE_VALIDATION_CASE=nvfp4 CODEX_CACHE_MODEL_PATH=/home/liusy/codex-runtime/checkpoints/nvfp4-2538ded2a4 .venv/bin/python -m pytest tests/model_executor/model_loader/_codex_oct29_validation.py -q

GitHub checks for the new head are pending at publication. No Buildkite GPU-suite pass or completed human review is claimed. AI assistance and original feature baselines are recorded below.

Earlier validation and original feature baselines (previous heads)

Earlier head aa141888102c8431c5f25c53477cf18cb837fd5d includes main 36d4ef654a83e227216b0c0b55cf4041d96b1675. Agent-run B300 validation on October 29, after synchronizing the native extensions with that main commit:

Check	Result
Online FP8/Blackwell full-model cases	4 passed, 268.86s
Qwen3-30B-A3B NVFP4 full-model case	1 passed, 102.00s
Weight-cache and padded CUTLASS regressions	108 passed, 83.35s
Full feature-file and commit hooks	Passed, including mypy and ruff

The four online cases use their checked-in settings (gpu_memory_utilization=0.2). The NVFP4 case uses a local copy of nvidia/Qwen3-30B-A3B-NVFP4 pinned at 2538ded2a4edb247b4d2b4a8ba24e44bd4c017c3, with only the checkpoint path and GPU memory fraction (0.12) substituted for shared-machine coexistence.

B300, TP1, BF16, eager mode, CUTLASS linear kernels and TRITON_ATTN. Full-model cases exercise ordinary loading → missing-daemon fallback → IPC → second IPC, with fallback forbidden on both IPC loads and the existing exact batch-invariant greedy-output comparisons unchanged. These are output-equivalence smoke tests, not accuracy or performance benchmarks; multi-GPU and other hardware are not claimed.

Runtime: PyTorch 2.13.0+cu130 with process-local CUDA 13.4 compatibility libraries. The two stable-libtorch native extensions were taken from the official CUDA 13.0 wheel for exact main 36d4ef654a83e227216b0c0b55cf4041d96b1675, verified against wheel RECORD hashes, and linked only into the task worktrees. Their declared stable ABI supports PyTorch >=2.11. This is not validation of the complete upstream PyTorch 2.14 dependency matrix; the shared venv and host driver were unchanged.

An initial Llama run stalled reading the shared NFS checkpoint; it was stopped and rerun using a pinned local-disk checkpoint. An initial NVFP4 ordinary load failed before IPC because the old native extension exposed an 11-argument operator while current source required 12; the matching extensions above resolved that environment mismatch. Neither initial run is counted as a passing test or as a cache regression. All results in the table use the matching extensions. Existing torch/fork deprecation warnings are retained in the logs.

The exact heads above are MERGEABLE; GitHub pre-commit, DCO and Read the Docs passed on October 29. No Buildkite GPU-suite success or completed human review is claimed. Original feature baselines and earlier results remain below and are not reruns of the October 29 conflict resolution.

Original feature baselines, reproduction commands and earlier validation

B300, single GPU per scenario. Each end-to-end case compares normal loading, disk fallback without a daemon, first IPC attachment, and a second IPC attachment. Both IPC runs disable fallback. Quantization tests use greedy output comparison with VLLM_BATCH_INVARIANT=1 and TRITON_ATTN.

On main 0f6b261, branch 084cab1: online FP8 per-tensor/per-channel/per-block and MXFP8: 4 passed (272.15s); Qwen3-30B-A3B-NVFP4 with CUTLASS linear/MoE: 1 passed (102.83s); weight-cache unit and CUTLASS padding/scales/bias cases: 102 passed (79.85s).

After merging main 2fb320d (including #62734 and #63506), on then-then-current head d771fd8:

• All four online FP8/MXFP8 full-model cases passed again: 4 passed (299.37s), using the checked-in configuration.
• Qwen3-30B-A3B-NVFP4 passed again: 1 passed (122.73s). This local run substituted the same-model checkpoint directory and set gpu_memory_utilization=0.12 instead of the CI case's 0.3 to coexist with other B300 workloads. The four loading paths and output assertions were unchanged; the earlier 0.3 result above is from 084cab1.
• Weight-cache unit and padded CUTLASS cases: 108 passed (102.27s).
• Changed-file/commit hooks, GitHub pre-commit, DCO and Read the Docs passed. No Buildkite GPU-suite result is asserted.

The unit runs emitted a background-thread warning in unchanged upstream seed.py: self._server became None before accept during listener shutdown; the warning is retained. Model runs emitted torch/multiprocessing deprecation warnings.

Before the fix, the real online-FP8 daemon reported UnsupportedQuantForIPCError: ... Fp8PerTensorOnlineLinearMethod does not support loading from pre-processed weights. The padded CUTLASS regression failed at assert output_size is not None when IPC skipped post-processing; the same regression passes with the constructor restoring the logical width.

Reproduction from the branch using an idle GPU:

CUDA_VISIBLE_DEVICES=0 HF_HUB_OFFLINE=1 .venv/bin/python -m pytest tests/model_executor/model_loader/test_weight_cache.py -k 'online_fp8 or online_blackwell' -q
CUDA_VISIBLE_DEVICES=0 .venv/bin/python -m pytest tests/model_executor/model_loader/test_weight_cache.py -k nvfp4_cutlass -q
CUDA_VISIBLE_DEVICES=0 .venv/bin/python -m pytest tests/kernels/quantization/test_cutlass_scaled_mm.py -k test_cutlass_fp8_gemm_padded -q
CUDA_VISIBLE_DEVICES=0 .venv/bin/python -m pytest tests/model_executor/model_loader/test_weight_cache.py -k 'not cold_start and not d2d_warmup and not online and not nvfp4_cutlass' -q

Post-merge command (small GPU allocations only):

CUDA_VISIBLE_DEVICES=7 PYTHONPATH=. .venv/bin/python -m pytest tests/model_executor/model_loader/test_weight_cache.py tests/kernels/quantization/test_cutlass_scaled_mm.py -k 'test_cutlass_fp8_gemm_padded or (not cutlass and not cold_start and not d2d_warmup and not online)' -q

NVFP4 local validation substitutes the existing same-model checkpoint directory for its Hugging Face ID. This validates IPC equivalence to ordinary quantized inference, not quantization accuracy against BF16. No TP/PP/DP/EP, multi-node, or full hardware matrix claim. Earlier shared-GPU profiling failures are retained in the logs; the same online suite passed on an isolated GPU without changing assertions. Precompiled FlashAttention was incompatible with this host, hence the explicit Triton attention choice.

Details

Online methods previously rejected daemon caching despite successful ordinary loading. The restore paths opt in only when their kernel state can be reconstructed. NVFP4 CUTLASS otherwise performs an in-place scale fusion during kernel initialization, which must not run again on shared daemon tensors.

Adds focused loading regressions, extends the existing padding test to construct a real kernel, adds L4/B200 Buildkite selection, and documents supported paths in the preload guide. No model-runner/scheduler changes or new steady-state host synchronization.

Duplicate checks on October 26 examined issue #56049 and its comments and searched open PRs for the issue, preload quantization, and weight-cache FP8. #62734 covers daemon-loss detection and #63506 covers kernel cache keys; neither adds these restore paths. #60015 concerns compressed-tensors scale registration and is outside this change. No duplicate implementing this scope was found in those searches.

AI assistance: implementation, debugging, validation, and this description were prepared with Codex at the submitter's request. This is a draft for human review; no completed human approval or human-run validation is asserted.

---

Pull Request Checklist

• I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).
• AI assistance was used during the creation of this PR.
• Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.
• Testing and Validation: Agent-run validation and its limitations are recorded above; human confirmation remains pending.
• Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.
• Pull Request Contents: Includes motivation, duplicate checks, reproduction commands, limitations, and AI disclosure.

---

^{Need help on this PR? Tag @codesmith-bot with what you need.}

[mergify]

Documentation preview: https://vllm--63672.org.readthedocs.build/en/63672/

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @liusy58.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[Isotr0py]

/ci run

[github-actions]

✅ Triggered Buildkite CI #99491 for commit 4b0801a9bb10.

ℹ️ This PR is 324 commits behind upstream main. Merged upstream main into this branch as 4b0801a9bb10 before starting CI. Pull the branch before pushing more changes.