[liusy58]

Overview

Enable weight-cache daemon restarts for serialized compressed-tensors W8A8 FP8 with CUTLASS 128×128 weight blocks. Related to #56049; depends on #63673 (and its prerequisite #63672).

Incremental review: liusy58/vllm@03b39f9...35e97cf (4 files; the diff against main also contains the prerequisites).

Claims

• Allow the actual CUTLASS block-FP8 kernel to attach already processed 128×128 block weights over CUDA IPC.
• Preserve the block weights' untransposed layout and shared weights/scales across repeated engine restarts.
• Retain the existing tensor/channel support and reject other block sizes and backends.

Validation

Current head 35e97cffd334c0732fed8ef0674138896faa373f includes main 914d61fc538a84e8487cb16fe8241a70f6f4bf9e. October 29 agent-run validation after resolving the inherited online-FP8 import conflict with upstream #60163. The merge retains both CUTLASS and Humming imports, the upstream processed-dimension metadata, and upstream rejection of the unsupported Humming per-token/channel path. It adds no Humming IPC capability.

Check	Exact-head result
ct-block-layers	37 passed, 33 deselected, 14 warnings in 31.61s
ct-block-model	1 passed, 18 warnings in 81.55s (0:01:21)
ct-block-llama	1 passed, 18 warnings in 85.80s (0:01:25)
Full feature-file and commit hooks	Passed, including mypy and ruff

Runtime: NVIDIA B300, PyTorch 2.13.0+cu130, process-local CUDA 13.4 compatibility libraries. The two stable-libtorch extensions are from the official CUDA 13.0 wheel for main 914d61fc538a84e8487cb16fe8241a70f6f4bf9e, checked against wheel RECORD hashes and linked only into task worktrees. The shared venv and host driver were unchanged. This does not validate the full upstream PyTorch 2.14 dependency matrix.

Full-model checks use TP1, BF16, eager mode, CUTLASS and TRITON_ATTN, with the existing batch-invariant exact greedy-output comparisons. Each completed case checks ordinary load, missing-daemon fallback, IPC and a second IPC load; both IPC loads forbid fallback. These are output-equivalence smoke checks, not accuracy or performance benchmarks. No multi-GPU, ROCm or other-backend result is claimed.

The temporary local helper calls the existing run_cold_start_and_warm_restart with dataclasses.replace(case, model=local_checkpoint, llm_kwargs={**case.llm_kwargs, "gpu_memory_utilization": 0.12}) and VLLM_BATCH_INVARIANT=1. It changes only checkpoint location and the GPU memory budget; loading paths, prompts and assertions are unchanged. Online cases keep their checked-in 0.2 budget.

Llama checkpoint: RedHatAI/Meta-Llama-3.1-8B-FP8 (redirected neuralmagic model), pinned 881d74b74ad8ebe8775fb178b1fd0c935477f1bc. Upstream #62407 retained-Parameter behavior and the existing attention identity/shared-storage assertions remain intact.

Block checkpoint: RedHatAI/Qwen3-0.6B-FP8-BLOCK, pinned 5def00c475640c0527e3a7067b1a26b94decefb5. Its model path is the existing cached model ID; the helper substitutes only the memory budget.

Commands for this revision

All commands set HF_HUB_OFFLINE=1, PYTHONPATH to the corresponding worktree and LD_LIBRARY_PATH=/home/liusy/codex-runtime/cuda-compat-13-4/usr/local/cuda-13.4/compat.

CUDA_VISIBLE_DEVICES=2  .venv/bin/python -m pytest tests/quantization/test_compressed_tensors.py -k 'compressed_fp8_preprocessed or scheme_selection or get_quant_method or find_matched_target or get_scheme_dict or runtime_activation_order or accepts_preordered_weights or fp8_moe_forwards' -q
CUDA_VISIBLE_DEVICES=2 CODEX_CACHE_VALIDATION_CASE=block .venv/bin/python -m pytest tests/model_executor/model_loader/_codex_oct29_validation.py -q
CUDA_VISIBLE_DEVICES=2 CODEX_CACHE_VALIDATION_CASE=compressed CODEX_CACHE_MODEL_PATH=/home/liusy/codex-runtime/checkpoints/compressed-881d74b74a .venv/bin/python -m pytest tests/model_executor/model_loader/_codex_oct29_validation.py -q

GitHub checks for the new head are pending at publication. No Buildkite GPU-suite pass or completed human review is claimed. AI assistance and original feature baselines are recorded below.

Earlier validation and original feature baselines (previous heads)

Earlier head fb04cad37dda54a727d03f021718a18f540a5d65 includes main 36d4ef654a83e227216b0c0b55cf4041d96b1675. Agent-run B300 validation on October 29, after synchronizing the native extensions with that main commit:

Check	Result
Configuration/selection/shared-weight/attention regressions	37 passed, 31.35s
Qwen3-0.6B block-FP8 full-model case	1 passed, 75.92s
Llama tensor/channel FP8 full-model regression	1 passed, 75.72s
Full feature-file and commit hooks	Passed, including mypy and ruff

Conflict resolution preserves upstream #62407 parameter registration while retaining IPC attention-scale restoration. The two existing attention tests now assert that post-load keeps the attached Parameter objects, shares the source storage, preserves exact scale values, and reconstructs CPU scale metadata from cached buffers.

The Llama case uses RedHatAI/Meta-Llama-3.1-8B-FP8 (the redirected neuralmagic model) pinned at 881d74b74ad8ebe8775fb178b1fd0c935477f1bc.

The temporary local wrapper calls the existing run_cold_start_and_warm_restart with only the checkpoint directory and gpu_memory_utilization=0.12 substituted; assertions and loading paths are unchanged. The block checkpoint remains RedHatAI/Qwen3-0.6B-FP8-BLOCK pinned at 5def00c475640c0527e3a7067b1a26b94decefb5, also using memory fraction 0.12.

B300, TP1, BF16, eager mode, CUTLASS linear kernels and TRITON_ATTN. Full-model cases exercise ordinary loading → missing-daemon fallback → IPC → second IPC, with fallback forbidden on both IPC loads and the existing exact batch-invariant greedy-output comparisons unchanged. These are output-equivalence smoke tests, not accuracy or performance benchmarks; multi-GPU and other hardware are not claimed.

Runtime: PyTorch 2.13.0+cu130 with process-local CUDA 13.4 compatibility libraries. The two stable-libtorch native extensions were taken from the official CUDA 13.0 wheel for exact main 36d4ef654a83e227216b0c0b55cf4041d96b1675, verified against wheel RECORD hashes, and linked only into the task worktrees. Their declared stable ABI supports PyTorch >=2.11. This is not validation of the complete upstream PyTorch 2.14 dependency matrix; the shared venv and host driver were unchanged.

An initial Llama run stalled reading the shared NFS checkpoint; it was stopped and rerun using a pinned local-disk checkpoint. An initial NVFP4 ordinary load failed before IPC because the old native extension exposed an 11-argument operator while current source required 12; the matching extensions above resolved that environment mismatch. Neither initial run is counted as a passing test or as a cache regression. All results in the table use the matching extensions. Existing torch/fork deprecation warnings are retained in the logs.

The exact heads above are MERGEABLE; GitHub pre-commit, DCO and Read the Docs passed on October 29. No Buildkite GPU-suite success or completed human review is claimed. Original feature baselines and earlier results remain below and are not reruns of the October 29 conflict resolution.

Original feature baselines, reproduction commands and earlier validation

NVIDIA B300, TP1, BF16, CUTLASS linear backend. Real-model checks use eager mode, TRITON_ATTN and VLLM_BATCH_INVARIANT=1, comparing ordinary loading → missing-daemon fallback → IPC → second IPC. Both IPC loads prohibit fallback. Each case compares exact greedy outputs for two prompts × 16 generated tokens.

Check	Base 4f9c3c8c06 plus new tests	With implementation
Real RedHatAI/Qwen3-0.6B-FP8-BLOCK, pinned 5def00c475640c0527e3a7067b1a26b94decefb5	1 failed, 103.87s: daemon rejects the compressed-tensors method	Passed all four loading paths with exact output equality
Real fused block-FP8 layer, two attachments	1 failed, 8.50s: pre-processed weights rejected	Exact outputs, layout metadata, shared pointers and tensor contents preserved
New block model plus existing Llama FP8 model regression	—	2 passed, 164.91s
Shared-weight/attention-scale and existing configuration/selection regressions	—	37 passed, 33 deselected, 30.20s; includes all 11 shared-weight/attention cases
Changed-file and commit hooks	—	Passed, including mypy and ruff

The Llama regression uses the existing local Meta-Llama-3.1-8B-FP8 checkpoint, with only its model path substituted; checked-in settings and assertions are unchanged. A first combined run passed the block case but failed to locate the Llama HF ID in the offline cache; the local-path rerun above passed both cases. These are smoke output-equivalence checks, not broad accuracy or performance benchmarks. Multi-GPU and other platforms are not claimed.

Reproduction commands and baseline output

CUDA_VISIBLE_DEVICES=0 HF_HUB_OFFLINE=1 PYTHONPATH=. .venv/bin/python -m pytest tests/model_executor/model_loader/test_weight_cache.py -k 'compressed_fp8 and block' -q
CUDA_VISIBLE_DEVICES=0 HF_HUB_OFFLINE=1 PYTHONPATH=. .venv/bin/python -m pytest tests/quantization/test_compressed_tensors.py -k 'preprocessed_shared_weights and block' -q
CUDA_VISIBLE_DEVICES=0 HF_HUB_OFFLINE=1 PYTHONPATH=. .venv/bin/python -m pytest tests/quantization/test_compressed_tensors.py -k 'compressed_fp8_preprocessed or scheme_selection or get_quant_method or find_matched_target or get_scheme_dict or runtime_activation_order or accepts_preordered_weights or fp8_moe_forwards' -q

HF_HUB_OFFLINE=1 assumes the pinned checkpoint and OPT-125m configuration are cached; omit it for the initial download. Before implementation, the first command failed after successful ordinary/fallback generation:

UnsupportedQuantForIPCError: layer model.layers.0.self_attn.qkv_proj:
CompressedTensorsLinearMethod does not support loading from pre-processed weights.
1 failed, 71 deselected, 16 warnings in 103.87s

The block layer baseline reported:

RuntimeError: layer <root>: CompressedTensorsLinearMethod does not support pre-processed weights
1 failed, 69 deselected, 14 warnings in 8.50s

For the two-model regression on this machine, a temporary test next to test_weight_cache.py calls the checked-in test with replace(suite.COMPRESSED_FP8_CASE, model="/mnt/minecraft/model/Meta-Llama-3.1-8B-FP8") and suite.COMPRESSED_BLOCK_FP8_CASE. It changes no production behavior or expectations. Run it with the same environment and -q; on a machine with both HF checkpoints cached, select -k compressed_fp8 directly. Runs emitted existing torch/fork deprecation warnings.

Details

The previous capability gate accepts only the tensor/channel CUTLASS kernel. Block FP8 uses a different kernel and keeps weights in [N, K] layout, so accepting it also requires avoiding the tensor/channel path's transposed-layout metadata. Opt in only the actual CutlassFp8BlockScaledMMKernel with [128, 128] blocks, retain the dynamic input_scale=None restoration, and return before any second weight transformation. The kernel constructor already initializes dynamic group-128 activation quantization and its scale layout. No new common loading interface, scheduler/model-runner path, or steady-state GPU synchronization.

The existing B200 Quantized Preload CI selection includes both model parameterizations and the extended real-layer test; its source dependencies cover the changed tests and quantization code. Local validation used B300; B200 Buildkite execution is not claimed. The preload feature guide documents the format, matching CUTLASS selection in daemon and engine, and exclusions. Other block sizes/backends, W8A16 fallback, transform wrappers and MoE remain outside this change.

Duplicate checks on October 27 covered #56049/comments, issue-linked PRs and open block-FP8/preload searches. #63673 adds tensor/channel FP8 only; #60015 fixes W8A16 checkpoint scale naming, and #62407 handles generic reload parameter retirement/aliases. Neither implements this W8A8 block-FP8 IPC path. The current branch includes main 914d61fc538a; its #62407 parameter-retention contract is preserved.

AI assistance: Codex implemented and ran the recorded validation at the submitter's request. This draft is for human review; no completed human approval or human-run tests are asserted.

---

Pull Request Checklist

• I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).

• AI assistance was used during the creation of this PR.

• Design Fit: Narrow capability opt-in and correct block layout metadata, reusing existing interfaces.

• Testing and Validation: Real checkpoint and layer before/after, regression results and CI resource limits recorded above.

• Code Quality and Style: Changed-file hooks passed; preload feature documentation updated.

• Pull Request Contents: Root cause, prerequisites, duplicate checks, commands, evidence, limitations and AI disclosure included.

[mergify]

Documentation preview: https://vllm--63976.org.readthedocs.build/en/63976/

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @liusy58.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @liusy58.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork