[Xunzhuo]

Overview

Serve the Decision 3.0 decision models (vllm-sr/d3 and its smaller variants such as vllm-sr/d3-mini and vllm-sr/d3-edge) from their Hub checkpoints as they are: vllm serve vllm-sr/d3, then ask typed questions on /v1/systemone. Today these checkpoints do not load in vLLM (Qwen3_5Model is not a registered architecture and the readout is a separate file). This builds on vLLM's Qwen3.5 implementation, the structured decisions API (#60465 and its predecessors) and the Clef-Flash detection pattern (#61308).

Claims

• The published checkpoints load unchanged (measured here: d3, d3-mini and d3-edge; see Follow-ups). A bare Qwen3_5Model checkpoint that ships a decision_config.json is detected and served as D3ForSequenceClassification. The 255-row answer-code readout (readout.safetensors) becomes the FP32 score head. The declared noncausal_full_attention makes the full attention layers bidirectional, while the Gated DeltaNet layers stay causal. Tied embeddings need nothing, and per-model layer_types are honoured (d3-edge: llFllFllFFFF).
• /v1/systemone gives the checkpoint runtime's answers. For each question, D3Strategy writes the prompt the checkpoint's own d3_format.py writes (byte-identical on 416 prompts), reads the question's codes from the classify head, and applies the option mask and the checkpoint temperature. It supports choice (up to 255 options), yes / no and score questions, text and JSON states, and images and videos read with the runtime's budgets (1.6 MP images; 2 fps, at most 32 frames of 0.2 MP).
• Same decisions as the reference runtime on AMD Instinct MI325X. Against AutoModel.from_pretrained(..., trust_remote_code=True): 0 of 416 decisions changed on d3, 1 on d3-mini and 2 on d3-edge, all ties or near-ties. The median probability difference is 0.0024 to 0.0029.
• Throughput. On a mixed text / JSON / image / video load at 8 and 64 closed-loop clients, vLLM sustains 1.4x to 2.6x the rate of the checkpoint's own server (AutoModel behind HTTP, one request at a time).

Validation

Hardware and software: one AMD Instinct MI325X per side. vLLM used the official ROCm nightly image vllm/vllm-openai-rocm:nightly-5555a2d8f7116b2b4877f66516f4128176cf722a (PyTorch 2.13, ROCm 7.2), with this branch's Python tree (main f4b8d43a6 plus this PR) overlaid; no kernels were built. The reference used the same image with the versions the checkpoint pins (transformers 5.17.0, flash-linear-attention 0.5.2). vLLM ran with no flags: vllm serve vllm-sr/<model>. Models: vllm-sr/d3 @ fc2d74bd, vllm-sr/d3-mini @ 6d8971f4, vllm-sr/d3-edge @ fe71f8e1.

Answers vs the reference (model.system_one(...) in process): 416 questions per model, namely 3 states (a support ticket, release notes, a JSON account) x 128 typed questions (48 choice, 40 yes / no, 40 score), plus 1-, 4- and 16-question text, a 4-question JSON state, a 40-option choice, an image with 1 and 4 questions, and a 10 s 1080p video. The decision is the top choice, noul >= 0.5, or a score's most likely level; |dp| is the largest absolute probability difference over a question's options.

Model	Decisions changed	Median abs dp	p90	Max	Image / video questions, max abs dp
d3 (27B)	0 / 416	0.0024	0.0078	0.027	0.0049 / 0.021
d3-mini (4B)	1 / 416 (0.5003 vs 0.4996)	0.0026	0.0078	0.032	0.018 / 0.032
d3-edge (0.6B)	2 / 416 (margins 0.0019, 0.0012)	0.0029	0.012	0.043	0.0050 / 0.0055

For scale, two vLLM configurations of d3 (defaults vs --max-model-len 32768, so different batch shapes) differ from each other by up to 0.011 and change 2 decisions.

Throughput and latency: closed loop, 30 s per level after an 8 s warm-up, over a mix of 40 % one-question text, 20 % four-question text, 10 % four-question JSON state, 10 % 40-option choice, 15 % new 1.6 MP image and 5 % new 10 s 1080p video. Every image and video carries new bytes. Each cell gives req/s and p50 / p90 latency in ms:

Model	Side	1 client	8 clients	64 clients
d3	checkpoint server	6.9 (69 / 290)	6.1 (1,210 / 1,653)	4.6 (9,300 / 9,817)
d3	vLLM	6.7 (74 / 265)	8.6 (903 / 1,228)	8.5 (5,907 / 6,552)
d3-mini	checkpoint server	17.0 (23 / 123)	23.3 (339 / 499)	18.3 (3,157 / 3,427)
d3-mini	vLLM	15.8 (30 / 103)	43.0 (166 / 273)	45.0 (1,357 / 1,606)
d3-edge	checkpoint server	31.6 (7.4 / 66)	47.3 (141 / 306)	41.7 (1,463 / 1,724)
d3-edge	vLLM	28.6 (10 / 56)	102 (23 / 211)	110 (36 / 2,679)

One request at a time (p50 ms, AutoModel in process / vLLM):

Model	1 text question	16 text questions	new image, 1 question	new image, 4 questions	new 10 s video
d3	55.9 / 54.9	583 / 547	293 / 216	1,020 / 700	519 / 443
d3-mini	18.1 / 19.4	114 / 116	111 / 64	341 / 158	319 / 261
d3-edge	6.4 / 7.0	21.2 / 29.0	56 / 33	140 / 48	266 / 234

Start-up and memory (launch to the first answer; cold = empty compile caches, model files on local disk):

Model	AutoModel cold / warm	vLLM cold / warm	GPU memory above idle, steady: AutoModel / vLLM
d3	39.9 / 22.0 s	214.5 / 130.0 s (--max-model-len 32768: 139.0 / 67.8 s, same throughput)	52 / 199 GiB
d3-mini	28.6 / 10.5 s	136.2 / 61.7 s	11 / 216 GiB
d3-edge	27.0 / 8.2 s	129.2 / 37.1 s	3.5 / 227 GiB

Tests

• pytest tests/models/multimodal/pooling/test_d3.py: d3-edge vs the checkpoint runtime, with text prompts as token ids (including a 40-option choice) and one image prompt processed by vLLM. PASSED on MI325X (134 s).
• pytest tests/config/test_model_arch_config.py -k "decision_config or joint_head": 8 passed.
• pytest tests/entrypoints/generate/structured_decisions/: 115 passed, including the new d3 strategy tests.
• pytest tests/models/test_registry.py tests/models/test_initialization.py -k D3: 2 passed.
• pre-commit run on the diff: all hooks pass. actionlint was skipped locally because no workflow files changed.

Details

• Detection follows Clef-Flash: Qwen3_5ModelArchConfigConvertor maps architectures: ["Qwen3_5Model"] plus decision_config.json to D3ForSequenceClassification and keeps the decision config on hf_config. Bare Qwen3_5Model checkpoints without that file are unaffected (they stay unsupported, as today).
• Config hook D3ForSequenceClassificationConfig validates format_version 1, pooling: last and the attention mode. It sets is_causal=False for noncausal_full_attention, so chunked prefill and prefix caching turn off by themselves, sets id2label to the 255 codes, and makes the classify head return raw code logits by default. Probabilities only exist over a question's own options, so /classify and /pooling callers mask and normalize themselves.
• Model (d3.py): as_seq_cls_model(Qwen3_5ForConditionalGeneration), so no LM head is allocated and the score head is FP32. It adds a mapper for the bare-backbone keys, readout.safetensors as score.weight via secondary_weights, and a processing info that applies the runtime's media budgets. The checkpoint's processor config allows 16.7 MP images, while the runtime reads at most 1.6 MP; video frames use transformers' cap_pixels_per_frame with the runtime's 200,704-pixel cap.
• Strategy: one classify read per question. The media come first, so they are processed once per request and shared by its questions, and the encoder cache deduplicates the vision tower. Videos go through the existing qwen3_vl loader, whose sampling rule already matches the runtime's, with max_frames=32. A small change to the question builder, DecisionLimits.labeled, lets label-free strategies accept more options than the A to Z label set; the defaults of other strategies are unchanged.
• Why not an example client: the System One API already has question types on main, so the option mask and temperature belong in the strategy rather than in client code.
• Tradeoffs and limits
  • Start-up: with chunked prefill off, the profiling pass runs one max_model_len-token sequence. At the config's 262,144 tokens this dominates start-up (71 s of d3's 130 s warm start). --max-model-len 32768 halves it, with no throughput change on this mix.
  • Memory: vLLM reserves --gpu-memory-utilization (0.92) of HBM. The Gated DeltaNet layers still keep per-request state, and the attention layers keep no KV.
  • Media under load: at 64 clients, text stays fast (d3-edge text p90 about 80 ms) while image and video requests queue (image p50 2.65 s), because decoding a new 300-frame 1080p video per request is CPU-bound. --renderer-num-workers 4 did not change this.
  • Small-model latency: one request at a time, multi-question requests on the small models are slower than in process (d3-edge, 16 questions: 29.0 vs 21.2 ms), since each question is its own engine request.
  • API differences from the runtime: a request with an invalid or too-long question returns 400 instead of per-question errors; at most 64 questions per request (the API's limit); confidence and usage follow /v1/systemone's definitions; option descriptions, instructions and score levels are rendered as text (non-string values go through str, where the runtime writes JSON); the runtime's optional permutation_average is not available.
  • Not measured: d3-flash, d3-nano and d3-lite (same layout and code path), CUDA GPUs, tensor parallelism, LoRA.
• Not a duplicate: no open or closed PR or issue covers these checkpoints, decision_config.json readout checkpoints, or bare Qwen3_5Model loading (searched for d3, Decision 3.0, Qwen3_5Model, decision_config.json, decider, readout). The Laya PR ([Model] Add Laya typed-decision models (Jev /v1/systemone via structured_server) #58429) adds an unrelated ModernBERT-based model.
• AI assistance was used for the code, tests, measurements and this description. The PR stays a draft until the submitter has reviewed every line.

Follow-ups

• The other five Decision 3.0 models (vllm-sr/d3-flash, vllm-sr/d3-mini, vllm-sr/d3-nano, vllm-sr/d3-lite, vllm-sr/d3-edge) share d3's architecture and take the same code path: each Hub checkpoint is a bare Qwen3_5Model with a d3-code-readout-v1 decision_config.json (noncausal_full_attention, 255 codes) and a readout.safetensors. The variations between them, tied embeddings and d3-edge's layer layout, are exercised by d3-mini and d3-edge.
• d3-mini and d3-edge already match the reference in this PR's measurements (1 and 2 of 416 decisions changed, all ties or near-ties; see the agreement and throughput tables under Validation).
• A follow-up PR will validate d3-flash, d3-nano and d3-lite, publish agreement and throughput for all five, and add them explicitly to the docs examples and tests where it makes sense.

---

Pull Request Checklist

• I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).

• AI assistance was used during the creation of this PR.

• Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

• Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.

• Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.

• Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

[mergify]

Documentation preview: https://vllm--61565.org.readthedocs.build/en/61565/

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[noooop]

following #61588

[Xunzhuo]

@noooop PTAL again, Thanks. cc @simon-mo

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @Xunzhuo.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[simon-mo]

/ci run

[github-actions]

✅ Triggered Buildkite CI #94819 for commit 07423a0de482.

ℹ️ This PR is 1 commit behind upstream main. Merged upstream main into this branch as 07423a0de482 before starting CI. Pull the branch before pushing more changes.

[Xunzhuo]

I think CI failure is not related to the changes, looks like a flaky test?

[Xunzhuo]

@khluu PTAL