[Xunzhuo]

Overview

Make every Decision 3.0 model work in vLLM: vllm serve vllm-sr/d3-nano stopped while loading weights, and d3-flash, d3-lite, d3-mini and d3-edge had no published validation on current vLLM. Follow-up to #61565, which added Decision 3.0 and validated d3, d3-mini and d3-edge; its "Follow-ups" section promised this PR.

Claims

• d3-nano loads. Its checkpoint has one model.safetensors and no weight index, so the default loader also read readout.safetensors as a backbone file and stopped with ValueError: There is no module or parameter named 'weight' in D3ForSequenceClassification. D3ForSequenceClassification now names the readout's only tensor in is_unused_checkpoint_weight, so the backbone pass skips that file; the readout still loads as the FP32 score head through secondary_weights.
• All five sizes give the checkpoint runtime's decisions with vllm serve vllm-sr/<model> and no flags, on one AMD Instinct MI300X: 0 to 2 of 416 decisions changed per model, all near-ties (the reference's two most likely answers within 0.015 of each other), with a median probability difference of 0.003 (table below).
• Throughput. At 8 and 64 closed-loop clients vLLM serves 2.0x to 3.6x the requests per second of the checkpoint's own server on the mixed text / image / video load; one request at a time it is 7 % to 32 % slower (each question is its own engine request).
• The docs list all six models, the GPU test runs d3-nano next to d3-edge, and the test registry lists every size.

Validation

Setup: one AMD Instinct MI300X (ROCm 7.2), a new rocm/dev-ubuntu-24.04:7.2.3-complete container with empty caches. vLLM is the ROCm nightly wheel 0.32.0rc1.dev190+g3f1bcfd82 (it contains #61565, and every file on the Decision 3.0 path is identical to main) with this PR's d3.py copied over the installed one; no kernels were built. The reference is the checkpoint's own runtime, AutoModel.from_pretrained(..., trust_remote_code=True), with the versions its requirements.txt pins (torch 2.13.0+rocm7.1, transformers 5.17.0, flash-linear-attention 0.5.2), in its own venv on the same GPU. Checkpoints at their current main: vllm-sr/d3-edge @ cab71e6a, vllm-sr/d3-lite @ 9c107031, vllm-sr/d3-nano @ aeb70d4d, vllm-sr/d3-mini @ 97dd054d, vllm-sr/d3-flash @ 8b35c853.

Reproducer: in the same container, the unchanged nightly fails on d3-nano and this PR loads it:

vllm serve vllm-sr/d3-nano
# nightly 0.32.0rc1.dev190+g3f1bcfd82:
#   ValueError: There is no module or parameter named 'weight' in D3ForSequenceClassification
# with this PR: ready, and the agreement set below

Answers vs the reference: the same 416 questions as #61565 (3 states x 128 typed questions, 1-, 4- and 16-question text, a 4-question JSON state, a 40-option choice, an image with 1 and 4 questions, and a 10 s 1080p video). The decision is the top choice, noul >= 0.5, or a score's most likely level; |dp| is the largest absolute probability difference over a question's options.

Model	Decisions changed	Median abs dp	p90	Max	Image / video questions, max abs dp
d3-edge (0.59B)	0 / 416	0.0030	0.0122	0.034	0.0071
d3-lite (0.85B)	1 / 416 (reference margin 0.0149)	0.0031	0.0097	0.022	0.0182
d3-nano (2.21B)	1 / 416 (reference margin 0.0028)	0.0028	0.0094	0.030	0.0300
d3-mini (4.54B)	1 / 416 (reference margin 0.0042)	0.0028	0.0076	0.042	0.0192
d3-flash (8.39B)	2 / 416 (reference margins 0.0128, 0.0077)	0.0027	0.0077	0.020	0.0104

Throughput: closed loop, 20 s per level after an 8 s warm-up, over the mix of #61565 (40 % one-question text, 20 % four-question text, 10 % four-question JSON, 10 % 40-option choice, 15 % new 1.6 MP image, 5 % new 10 s 1080p video; every image and video carries new bytes). The checkpoint server is the package's d3_server.py (AutoModel behind HTTP). Each cell gives req/s and p50 / p90 latency in ms:

Model	Side	1 client	8 clients	64 clients
d3-edge	checkpoint server	18.7 (12 / 89)	36.5 (187 / 341)	26.9 (2,108 / 2,569)
d3-edge	vLLM	16.1 (17 / 85)	72.0 (34 / 302)	61.5 (36 / 4,233)
d3-lite	checkpoint server	15.9 (25 / 102)	24.6 (285 / 457)	19.2 (2,863 / 3,104)
d3-lite	vLLM	10.8 (36 / 107)	59.7 (81 / 229)	59.5 (94 / 4,213)
d3-nano	checkpoint server	14.2 (26 / 125)	20.7 (358 / 521)	16.5 (3,192 / 3,677)
d3-nano	vLLM	11.1 (32 / 113)	52.8 (94 / 254)	59.8 (235 / 3,699)
d3-mini	checkpoint server	11.4 (33 / 147)	16.2 (449 / 671)	12.7 (3,953 / 4,276)
d3-mini	vLLM	10.4 (32 / 135)	37.1 (167 / 360)	35.7 (1,585 / 2,031)
d3-flash	checkpoint server	10.7 (34 / 203)	11.2 (698 / 934)	9.3 (5,126 / 5,480)
d3-flash	vLLM	9.9 (32 / 169)	24.6 (297 / 485)	20.8 (2,706 / 3,130)

Start-up: vLLM's launch to the first answer from empty caches, the model download included: 132 to 172 s (d3-edge 140 s, d3-lite 136 s, d3-nano 132 s, d3-mini 163 s, d3-flash 172 s).

Tests

• pytest tests/models/multimodal/pooling/test_d3.py (MI300X, this PR): [vllm-sr/d3-edge] PASSED, [vllm-sr/d3-nano] PASSED (2 passed in 270 s). Without this PR, d3-nano fails at load as in the reproducer.
• pytest tests/config/test_model_arch_config.py tests/models/test_registry.py tests/entrypoints/generate/structured_decisions/test_strategies.py -k "decision_config or joint_head or D3 or d3 or strategy_selection" (MI300X, this PR): 13 passed.
• CPU check of DefaultModelLoader.get_all_weights on a d3-nano-like layout (model.safetensors with backbone tensors plus readout.safetensors, no index): on main the backbone pass yields the readout's root weight; with this PR it yields only the backbone tensors, and score.weight still comes from secondary_weights.
• vLLM CI on NVIDIA, Buildkite #95083 (H200, MIG 18 GB, the multimodal pooling step): test_d3.py [vllm-sr/d3-edge] and [vllm-sr/d3-nano] PASSED; the d3-nano run logs Skipped 1/2 safetensors shard(s) holding no weights the model loads.
• pre-commit run --from-ref origin/main --to-ref HEAD: all hooks pass (actionlint skipped locally; no workflow files changed).

Details

• Root cause: without model.safetensors.index.json, DefaultModelLoader._prepare_weights keeps every *.safetensors file of the repo (filter_duplicate_safetensors_files returns them all when the index is missing), so readout.safetensors ({weight: [255, hidden]}) joins the backbone pass. Its root key weight matches no module, and AutoWeightsLoader raises. The other five checkpoints ship an index that lists only the backbone shards, so the index filter already drops the readout there. d3-nano is the only Decision 3.0 repo without an index.
• Fix: is_unused_checkpoint_weight is the existing hook the default loader passes to filter_safetensors_files_by_weight_name for the primary weights only; it drops a safetensors file when every tensor in it is unused. The readout file holds a single tensor named weight, which no backbone checkpoint has at the root. Its own secondary_weights source sets no such rule and keeps loading it under the score. prefix. For indexed checkpoints nothing changes beyond reading the shard headers, which the loader already does for pooling models (their protocol default returns False).
• Alternatives considered: allow_patterns_overrides=["model*.safetensors"] would also skip the index filter and depend on file names; adding an index to the Hub repo fixes only that copy of the checkpoint.
• CI: test_d3.py runs in the optional "Multimodal Models (Extended Pooling)" step; d3-nano adds a 4.5 GB checkpoint there. d3-lite, d3-mini and d3-flash take the same code path as d3-edge and d3-nano and are covered by the measurements above rather than CI, to keep the step's downloads and run time small.
• Not measured: agreement and throughput on NVIDIA GPUs (only the CI test above ran there; no NVIDIA GPU was available to me), tensor parallelism, LoRA.
• Not a duplicate: no open or closed PR or issue covers d3-nano or the other sizes (searched for Decision 3.0, vllm-sr/d3, d3-nano, D3ForSequenceClassification, decision_config); [Model] Support Decision 3.0 decision models (vllm-sr/d3) #61565 is merged.
• AI assistance was used for the code, tests, measurements and this description.

---

Pull Request Checklist

• I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).

• AI assistance was used during the creation of this PR.

• Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

• Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.

• Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.

• Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

[mergify]

Documentation preview: https://vllm--61778.org.readthedocs.build/en/61778/

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

[Xunzhuo]

/ci run

[github-actions]

❌ @Xunzhuo, A reviewer with write access must run /ci run, approve the PR, or add the ready label first.

[Xunzhuo]

Thanks @noooop , anyone can take a look at vllm-project/recipes#1124?

[Xunzhuo]

/ci run --allow-stale

[github-actions]

✅ Triggered Buildkite CI #95083 for commit 304dc45c4cf0.

⚠️ This PR is 3 commits behind upstream main. Running CI at your own risk because --allow-stale was requested; outdated CI configuration may cause failures. Before merging, merge or rebase onto the latest main, then rerun /ci run on the latest PR commit.

[Xunzhuo]

looks like some flaky tests met again