[liusy58]

Overview

Allow serialized compressed-tensors W8A8 FP8 tensor/channel checkpoints to reuse Marlin-packed weights through the weight cache daemon. This is a stacked follow-up to #63673 for #56049; review the four-file incremental diff.

Claims

• Opt in the actual Marlin kernel for tensor/channel weights, preserving its packed weights, reordered scales and padded/permuted bias across repeated IPC attachments.
• Support FP16/BF16 activations; keep Marlin block FP8 and other compressed-tensors schemes outside this opt-in.
• Preserve the existing CUTLASS path and its exact-output regression assertions. Document both-process configuration and Marlin's weight-only activation semantics.

Validation

Validated on B300 with PyTorch 2.13/cu130, the official main 914d61fc538a stable extensions and process-local CUDAcompat13.4 libraries. Shared environment/driver and other users' processes were unchanged. This is model-output equivalence validation, not a quality/performance benchmark or the full upstream Torch2.14 matrix.

Case	Parent production + new tests	This change
Marlin real fused layer, two attachments	Fails at CompressedTensorsLinearMethod does not support pre-processed weights	16 Marlin combinations pass
Existing CUTLASS/attention + Marlin block rejection	Existing cases retained	All 27 layer/boundary cases pass, 38.31s
Llama3.1 8B FP8, Marlin	Ordinary/fallback generation succeeds; daemon raises UnsupportedQuantForIPCError, 76.72s	Complete ordinary/fallback/IPC/restart passes
Same Llama checkpoint, existing CUTLASS	Existing baseline behavior	Complete exact-output regression passes; both model cases total155.46s

Both IPC runs set fallback=False. Marlin is not batch invariant, so the model comparison uses existing check_logprobs_close defaults with prompt/sample top5 and two prompts x16 generated tokens. Layer comparisons require exact output equality and unchanged shared parameter pointers/content, including bias and scales. CUTLASS keeps its original exact model-output assertions.

CUDA_VISIBLE_DEVICES=7 HF_HUB_OFFLINE=1 \
LD_LIBRARY_PATH=/home/liusy/codex-runtime/cuda-compat-13-4/usr/local/cuda-13.4/compat \
.venv/bin/python -m pytest tests/quantization/test_compressed_tensors.py \
  -k compressed_fp8_preprocessed -q

.venv/bin/python -m pre_commit run --files \
  vllm/model_executor/layers/quantization/compressed_tensors/schemes/compressed_tensors_w8a8_fp8.py \
  tests/quantization/test_compressed_tensors.py \
  tests/model_executor/model_loader/test_weight_cache.py docs/features/preload.md

All changed-file hooks pass, including mypy and CI test tethering. The existing B200 Quantized Preload selectors (compressed_fp8 and compressed_fp8_preprocessed) select the new tests, and source dependencies already cover the changes. Hosted GPU CI has not been claimed.

Full-model reproduction and baseline evidence

The checked-in new test is test_ipc_cache_compressed_fp8_marlin. On the shared host the validation wrapper below runs the same helper against a local copy of RedHatAI/Meta-Llama-3.1-8B-FP8 revision 881d74b74ad8ebe8775fb178b1fd0c935477f1bc, avoiding shared-filesystem checkpoint I/O. It also runs the existing CUTLASS case at the same 0.12 memory budget.

from dataclasses import replace
import pytest
from tests.model_executor.model_loader import test_weight_cache as suite

@pytest.mark.parametrize("backend", ["marlin", "cutlass"])
def test_cache_restart(backend, vllm_runner, tmp_path, monkeypatch):
    original = (suite.COMPRESSED_FP8_MARLIN_CASE if backend == "marlin"
                else suite.COMPRESSED_FP8_CASE)
    case = replace(original,
        model="/home/liusy/codex-runtime/checkpoints/compressed-881d74b74a",
        llm_kwargs={**original.llm_kwargs, "gpu_memory_utilization": 0.12})
    monkeypatch.setenv("VLLM_BATCH_INVARIANT", "0" if backend == "marlin" else "1")
    suite.run_cold_start_and_warm_restart(vllm_runner, case, tmp_path,
        num_logprobs=5 if backend == "marlin" else None)

Actual wrapper command, with the same environment as the layer run:
.venv/bin/python -m pytest tests/model_executor/model_loader/_codex_oct30_marlin_validation.py -q
Result: 2 passed, 22 warnings in 155.46s (existing torch/fork deprecations).

Before changing production code from parent 03b39f9a8f87, the layer command with -k 'preprocessed_shared_weights and marlin' -x fails11.60s, and the wrapper with -k marlin -x fails76.72s:

RuntimeError: layer <root>: CompressedTensorsLinearMethod does not support pre-processed weights

UnsupportedQuantForIPCError: layer model.layers.0.self_attn.qkv_proj:
CompressedTensorsLinearMethod does not support loading from pre-processed weights.
Weight cache daemon rank(s) exited during startup (exitcodes=[1]); shutting down.

The before/after runs use the same tests and assertions. Full logs and JSON exit records are retained in the task's codex-oct30-compressed-marlin-evidence.tar.gz; no retries, skip additions or tolerance changes were used to obtain these results.

Details

The scheme previously advertised pre-processed support only for CUTLASS. Marlin's ordinary post-load hook repacks weights and transforms scales/bias; reapplying it to shared tensors would be incorrect. Its constructor already restores runtime state, so this extends the existing IPC early return by gating on the actual Marlin kernel and non-block strategy. No kernel or shared loader interfaces change.

Duplicate checks covered #56049-linked PRs, Marlin FP8 and compressed-tensors IPC. #63673 supplies the CUTLASS-based prerequisite; #64182 covers AWQ/GPTQ; #42657 concerns ordinary W8A16 processing. None enables this W8A8 Marlin cache path. The main-based PR includes its unmerged ancestors; the incremental comparison above isolates this change.

AI assistance was used. This remains a draft for human review and validation; no completed human approval or human-run testing is asserted. Hosted pre-commit, DCO and Read the Docs passed on babf3e883834. Fresh local-checkout, GitHub-source and rendered-HTML discoverability checks passed for the new feature. The preview section, examples and same-page anchors were verified in generated HTML; browser visual layout was not verified because navigation timed out. The pre-existing pipeline-parallelism prose contradiction elsewhere on this page is outside this change.

---

Pull Request Checklist

• I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).

• AI assistance was used during the creation of this PR.

• Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

• Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.

• Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.

• Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

[mergify]

Documentation preview: https://vllm--64738.org.readthedocs.build/en/64738/