[liusy58]

Overview

Allow the weight-cache daemon to reuse AWQ/GPTQ linear weights already packed for Marlin across engine restarts. Related to #56049; based directly on main 6376fda7a9.

Claims

• Support AWQ INT4 and symmetric GPTQ INT4/INT8 linear weights with the actual Marlin kernel and FP16/BF16 activations.
• Preserve shared repacked weights, scales, zero points and padded bias without repeating checkpoint conversion or Marlin transformations.
• Continue rejecting other kernels, INT8/FP8 activation overrides and AWQ/GPTQ MoE.

Validation

B300, TP1. The two real-model cases use FP16, eager execution, TRITON_ATTN, explicitly selected Marlin, two prompts × 16 generated tokens, and prompt/sample top-5 logprobs. Each compares ordinary loading → missing-daemon disk fallback → IPC → second IPC; both IPC starts set fallback=False. Comparison uses the existing check_logprobs_close helper. VLLM_BATCH_INVARIANT=0 is intentional: AWQ switches away from Marlin when enabled, and Marlin does not guarantee batch invariance.

Check	Before implementation	After implementation
Pinned Qwen2.5-0.5B AWQ and GPTQ-INT4 checkpoints	2 failed, 118.22s; ordinary/fallback generation works, both daemons reject their quant method	2 passed, 144.56s, all four loading paths
Real fused INT4 layers, aligned/padded outputs and bias	4 failed, 32.41s: pre-processed weights rejected	All pass, with exact output/shape/shared-pointer/tensor-content checks across two attachments
Final layer/boundary suite, including GPTQ INT8/channelwise BF16 and actual Triton/A8 rejection	—	9 passed, 12.92s
Existing AWQ/GPTQ config and weight-cache regressions	—	76 passed, 18 deselected, 63.38s
Changed-file hooks	—	Passed, including mypy, ruff and CI test tethering

Checkpoints: Qwen/Qwen2.5-0.5B-Instruct-AWQ at db09cd27ead7fee40cdee309693cf83601b9c899; Qwen/Qwen2.5-0.5B-Instruct-GPTQ-Int4 at c34a4a91629f09f73a285f32dbd26106b033c654. INT8/BF16 and padded outputs are covered by real CUDA layers, not a complete INT8 checkpoint. This is cache-versus-ordinary output equivalence, not quantization accuracy against BF16 or a performance benchmark. Multi-GPU, other hardware and other kernels are not claimed.

Reproduction and baseline evidence

# Run each on an idle GPU; HF_HUB_OFFLINE assumes the pinned models are cached.
CUDA_VISIBLE_DEVICES=4 HF_HUB_OFFLINE=1 PYTHONPATH=. .venv/bin/python -m pytest tests/model_executor/model_loader/test_weight_cache.py -k marlin_preprocessed -q
CUDA_VISIBLE_DEVICES=5 HF_HUB_OFFLINE=1 PYTHONPATH=. .venv/bin/python -m pytest tests/model_executor/model_loader/test_weight_cache.py -k ipc_cache_marlin -q
CUDA_VISIBLE_DEVICES=4 HF_HUB_OFFLINE=1 PYTHONPATH=. .venv/bin/python -m pytest tests/quantization/test_auto_awq.py tests/quantization/test_auto_gptq.py tests/model_executor/model_loader/test_weight_cache.py -k 'not quantization_method and not cold_start and not d2d_warmup and not marlin' -q

Apply the new tests to main 6376fda7a9 without the two production changes to reproduce the failures. Before implementation, the real daemons report:

UnsupportedQuantForIPCError: layer model.layers.0.self_attn.qkv_proj:
AutoAWQMarlinLinearMethod does not support loading from pre-processed weights.
UnsupportedQuantForIPCError: layer model.layers.0.self_attn.qkv_proj:
AutoGPTQLinearMethod does not support loading from pre-processed weights.
2 failed, 74 deselected, 16 warnings in 118.22s

Host-specific environment: its precompiled Marlin path first failed with unsupported PTX during ordinary loading. Baseline and passing results above both use process-local LD_LIBRARY_PATH=/home/liusy/codex-runtime/cuda-compat-13-4/usr/local/cuda-13.4/compat, containing NVIDIA's checksum-verified cuda-compat-13-4 package 615.71.10-1ubuntu1, as supported by NVIDIA's compatibility matrix. It was extracted under the user's task directory; the host driver and shared Python environment were not modified. Initial environment failures and corrected negative-test setup errors are retained separately and are not represented as cache bug reproductions. Runs emit existing torch/fork deprecation warnings.

Details

Both methods previously rejected daemon caching even though ordinary Marlin loading worked. Their kernel configuration and parameter names are constructed before post-load processing; the remaining normal post-load work converts AWQ packing and/or repacks weights, permutes scales/zero points and pads/permutates bias. IPC already supplies those transformed tensors. Opt in only the actual MarlinLinearKernel with FP16/BF16 activations, and return before those transformations on the pre-processed path. No common loading interface, kernel implementation, scheduler/model-runner change or steady-state synchronization is added.

Reuse the existing daemon, generation and comparison helpers. The L4 Marlin Preload CI step selects the real-model and layer/boundary cases and tracks their source dependencies. Local validation is B300; L4 Buildkite execution is not claimed. The preload guide documents commands, defaults and exclusions, with links from the AWQ/GPTQ guides.

Duplicate checks covered current #56049/comments, issue-linked drafts and open Marlin-preload/AWQ/GPTQ-cache searches. #61908 introduces a different on-disk AWQ-v2 format, while #60479 changes GPTQ batch-invariant execution; neither restores daemon IPC weights. Existing FP8/FP4 cache drafts do not cover this path.

AI assistance: Codex implemented and ran the recorded validation at the submitter's request. This draft is for human review; no completed human approval or human-run tests are asserted.

---

Pull Request Checklist

• I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).

• AI assistance was used during the creation of this PR.

• Design Fit: Narrow capability opt-in using existing kernel construction and loading interfaces.

• Testing and Validation: Real-model and layer before/after checks, shared tensor preservation, regression and CI coverage recorded above.

• Code Quality and Style: Changed-file hooks passed; discoverable support documentation and CI selection included.

• Pull Request Contents: Root cause, duplicate checks, commands, limitations, environment and AI disclosure included.

[mergify]

Documentation preview: https://vllm--64182.org.readthedocs.build/en/64182/