[Frontend] Serve LiquidAI/d1-3B on /v1/systemone

[mgoin]

Overview

Serve LiquidAI/d1-3B on /v1/systemone with answers that match the checkpoint's own model.system_one(), for text and image states.

The question types (#60696) and the images request field (#61064) have since landed on main, so this PR is now only the Liquid read strategy and the small changes to the next-token read that it needs.

vllm serve LiquidAI/d1-3B

Demo

d1-3B playing Snake through /v1/systemone, recorded in real time on one B300. Each move is one HTTP request with a single choice question (up / down / left / right), and the snake takes the model's top choice with no fallback logic.

In this 15 second run it made 666 moves (44 a second, 23 ms median per request), ate 46 apples and died once. The state is a short text: where the apple is relative to the head, and what is in each of the four neighboring cells. Given raw coordinates or an ASCII grid instead, the model played poorly (14 and 4 apples in 400 moves, against 32). The recording used the default prefix caching; see Details for why turning it off is faster here.

Claims

• d1-3B picks and probabilities on /v1/systemone match the reference implementation shipped with the checkpoint, for text and image states and all three question types.
• Images are fetched once per request and shrunk to at most 1024 x 1024 pixels, as the reference does.
• A read that fails after rendering an image no longer breaks the next request that uses the same image.

Validation

Prompt parity (CPU). For 15 questions over 11 cases, the rendered prompt and the label token ids are identical to what prompt.py in the d1-3B repo produces (render and readout_ids).

Answers against the reference (one B300, bf16). Reference: AutoModel.from_pretrained("LiquidAI/d1-3B", trust_remote_code=True).system_one(...) on transformers 5.18.0, each question asked alone. "Gap" is the largest absolute difference in any option's probability.

Case	vLLM	Reference	Gap
Ticket, 3 questions (noul / choice / score)	0.990 / billing / 0.757	0.990 / billing / 0.757	0.0007
Single choice	technical	technical	0.0008
Single noul	0.018	0.020	0.0023
Single score, 3 levels	1.817	1.823	0.0032
JSON state, 2 questions	billing / 0.349	billing / 0.349	< 0.0001
Japanese text in a JSON state	Japanese	Japanese	0.031
Noul with criteria	0.982	0.982	< 0.0001
Options named a to d	b	b	0.0002
Score, 10 levels	7.373	7.373	< 0.0001
Image only (COCO 000000039769), 1 question	two	two	0.0005
Image and text, 2 questions	two / 0.984	two / 0.986	0.0019

Every pick matches. The Japanese case has the same prompt tokens and the same pick; its distribution is spread over three options (0.33 / 0.51 / 0.16). I did not track down the source of that 0.031 gap.

Reviewer findings (thanks @simon-veitner-redhat). All five are addressed, re-implemented on the current main and re-checked on a B300:

Finding	Now
A decision that fails after rendering an image breaks the next request with that image	Renders skip the sender cache. A failing two-question image request returns 400, and a chat request and a decision with the same image then return 200.
Each question fetches every image again	Images are fetched once and passed to every question.
Images above 1 MP differ from system_one()	Shrunk as the reference does. A 1280x960 image gives the same 1814 input tokens and probabilities within 0.0019.
A one-letter option name that is several tokens fails with a max() error	Returns 400 naming the label.
confidence differs from system_one()	Left as vLLM's definition. The docs now claim only picks and probabilities.

Tests.

pytest tests/entrypoints/generate/structured_decisions tests/entrypoints/generate/generative_scoring

72 passed on a B300, on this branch as pushed.

Not covered in CI. There is no d1-3B end-to-end test, to avoid pulling a 3B checkpoint. The d1 prompt wording is pinned by a CPU-only unit test; the answer parity above was run by hand.

Details

• Why a strategy. d1 is a post-trained LFM2.5-VL-3B that expects a fixed prompt (QUESTION: / Options: / Reply with the option code only.) and pools label variants (yes/Yes/YES, A/ A). LiquidStrategy subclasses NextTokenStrategy and overrides only the user turn and the label tokens.
• Label groups. A read now takes a group of tokens per label and scores the best one. The generic strategy uses groups of one, so its behavior is unchanged.
• Selected by architecture. d1's config.json is the same as the base model's, so every Lfm2VlForConditionalGeneration gets this prompt. Liquid's runner does the same.
• Usage. input_tokens sums the prompt of every question, so a multi-question request reports more than the reference, which counts the state once.
• Limitations. A choice is still capped at 26 options (the reference switches to two-digit codes beyond that). An empty-string state is treated like a null state.
• Latency. On one B300 a single-question request takes about 14 ms: roughly 9 ms in the engine step, 3 ms between the API server and the engine, and 1 ms each for HTTP and for rendering the prompt. With prefix caching on (the default) and a state that changes every request, the median rises to 22 ms and is bimodal (14 / 22 / 31 ms); --no-enable-prefix-caching brings it back to a flat 14 ms. I did not trace the cause in the scheduler.
• Overlap. Not a duplicate: no other open PR serves d1.
• Shared changes. The media connector moves from JointSchemaStrategy to the ReadStrategy base so both strategies use it, and every next-token render now skips the multimodal sender cache.

---

Pull Request Checklist

• I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).

• AI assistance was used during the creation of this PR.

• Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

• Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.

• Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.

• Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

🤖 Generated with Claude Code

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[mergify]

Documentation preview: https://vllm--60676.org.readthedocs.build/en/60676/

[LucasWilkinson]

Awesome work thanks! overall LGTM, i would like to figure out if theres a better home for model specific strategies (maybe in the model folders) but for now this seems fine since i assume in the short to medium term there will be churn on the strategy interface so maybe best to keep it all together for now

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @mgoin.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @mgoin.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[simon-veitner-redhat]

Hi @mgoin, I ran an adversarial pass on the d1 strategy on an H200. The prompt and readout hold up: every pick matched model.system_one() on 27 text and image cases, and with both sides in fp32 the probabilities agree within 0.0003 (the bf16 gaps, including the Japanese 0.031, are single bf16 logit steps). Tests pass (56/56).

The image path has three bugs, plus two smaller items. The fixes are four commits on pr60676-review-fixes, based on this PR's head, each cherry-pickable on its own. All repros were re-run on that branch and now behave as expected.

1. A failed or overlapping decision breaks the next request with the same image (chat included)

read() renders all questions before sending any. The first render marks the image as sent in the mm sender cache (P0). If a later question fails (e.g. an 11-level score, L285), nothing reaches the engine, and the next request with those image bytes sends only the hash and hits a P1 cache miss. A request that arrives while a long decision is still rendering fails the same way. Fix: render with skip_mm_cache=True, as the render-only endpoints do: simon-veitner-redhat@d11e50c

Repro (failed request), vllm serve LiquidAI/d1-3B --port 18676, same on any Lfm2Vl model

# Server: vllm serve LiquidAI/d1-3B --port 18676   (also any Lfm2Vl model, e.g. LFM2-VL-450M)
# A decision request that fails after rendering question 1 poisons the image cache;
# the next chat completion with the same image then fails with 500.
import base64, json, os, struct, urllib.request, zlib
URL, M = f"http://127.0.0.1:{os.environ.get('PORT', '18676')}", os.environ.get("MODEL", "LiquidAI/d1-3B")
raw = b"".join(b"\0" + os.urandom(3 * 32) for _ in range(32))  # fresh 32x32 noise PNG each run
ch = lambda t, d: struct.pack(">I", len(d)) + t + d + struct.pack(">I", zlib.crc32(t + d))
img = "data:image/png;base64," + base64.b64encode(b"\x89PNG\r\n\x1a\n" + ch(b"IHDR", struct.pack(">IIBBBBB", 32, 32, 8, 2, 0, 0, 0)) + ch(b"IDAT", zlib.compress(raw)) + ch(b"IEND", b"")).decode()
def post(path, body):
    try: r = urllib.request.urlopen(urllib.request.Request(URL + path, json.dumps(body).encode(), {"Content-Type": "application/json"})); return r.status, r.read()[:160]
    except urllib.error.HTTPError as e: return e.code, e.read()[:160]
if not os.environ.get("CONTROL"): print("decision:", post("/v1/systemone", {"model": M, "state": "x", "images": [img], "questions": {
    "a": {"type": "noul", "instructions": "Red?"}, "b": {"type": "score", "instructions": "Rate.", "criteria": list("0123456789X")}}}))
print("chat:    ", post("/v1/chat/completions", {"model": M, "max_tokens": 1, "messages": [{"role": "user", "content": [
    {"type": "image_url", "image_url": {"url": img}}, {"type": "text", "text": "Color?"}]}]}))

Observed (d1-3B): decision: 400 (at most 10 levels), then chat: 500 Internal server error (log: "Multi-modal cache miss ... P0/P1 cache drift").
Expected: chat 200, as it is without the failed decision first (CONTROL=1).

Repro (overlapping request, no failure)

# Server: vllm serve LiquidAI/d1-3B --port 18676   (also LFM2-VL-450M)
# Request A: 64 questions over one image (64 renders before anything reaches the engine).
# Request B: one question over the same image, sent 0.3 s later -> fails (no failed request involved).
import base64, io, json, os, threading, time, urllib.request
from PIL import Image
URL, M = f"http://127.0.0.1:{os.environ.get('PORT', '18676')}", os.environ.get("MODEL", "LiquidAI/d1-3B")
buf = io.BytesIO(); Image.frombytes("RGB", (1024, 1024), os.urandom(3 << 20)).save(buf, "PNG")
img = "data:image/png;base64," + base64.b64encode(buf.getvalue()).decode()
def post(tag, n):
    body = {"model": M, "state": "x", "images": [img], "questions": {f"q{i}": {"type": "noul", "instructions": f"Q{i}?"} for i in range(n)}}
    t = time.time()
    try: r = urllib.request.urlopen(urllib.request.Request(URL + "/v1/systemone", json.dumps(body).encode(), {"Content-Type": "application/json"})); s, b = r.status, r.read()
    except urllib.error.HTTPError as e: s, b = e.code, e.read()
    print(tag, s, f"{time.time() - t:.2f}s", b[:110])
a = threading.Thread(target=post, args=("A (64 q):", 64)); a.start(); time.sleep(0.3); post("B (1 q): ", 1); a.join()

Observed: B (1 q): 400 Generation error for item 0 in 4 of 4 runs (1 on d1-3B, 3 on LFM2-VL-450M); A 200.
Expected: both 200.

2. Each question fetches every image again

Every render builds its own media connector, so N questions fetch, decode and hash each image N times, one after another, and a URL with changing content gives the questions different pictures. Fix: fetch once with the chat path's media rules and pass the decoded images to every question: simon-veitner-redhat@8ca8987

Repro, vllm serve LiquidAI/d1-3B --port 18676, same on any Lfm2Vl model

# Server: vllm serve LiquidAI/d1-3B --port 18676   (also LFM2-VL-450M)
# One request, 2 identical questions, 1 image URL. The URL serves red then blue (a live snapshot).
# Each question fetches the image again, so the two identical questions see different pictures.
import io, json, os, threading, urllib.request
from http.server import BaseHTTPRequestHandler, HTTPServer
from PIL import Image
URL, M = f"http://127.0.0.1:{os.environ.get('PORT', '18676')}", os.environ.get("MODEL", "LiquidAI/d1-3B")
def png(c): b = io.BytesIO(); Image.new("RGB", (64, 64), c).save(b, "PNG"); return b.getvalue()
imgs, gets = [png((230, 20, 20)), png((20, 20, 230))], []
class H(BaseHTTPRequestHandler):
    def do_GET(self):
        body = imgs[len(gets) % 2]; gets.append(1)
        self.send_response(200); self.send_header("Content-Type", "image/png"); self.send_header("Content-Length", str(len(body))); self.end_headers(); self.wfile.write(body)
    def log_message(self, *a): pass
srv = HTTPServer(("127.0.0.1", 0), H); threading.Thread(target=srv.serve_forever, daemon=True).start()
q = {"type": "noul", "instructions": "Is the image red?"}
body = {"model": M, "state": None, "images": [f"http://127.0.0.1:{srv.server_port}/x.png"], "questions": {"q1": q, "q2": q}}
r = json.load(urllib.request.urlopen(urllib.request.Request(URL + "/v1/systemone", json.dumps(body).encode(), {"Content-Type": "application/json"})))
print("image GETs:", len(gets), "| P(red) q1 =", round(r["answers"]["q1"]["noul"], 3), "q2 =", round(r["answers"]["q2"]["noul"], 3))

Observed (d1-3B): image GETs: 2 | P(red) q1 = 0.986 q2 = 0.02.
Expected: image GETs: 1 and the same answer for both questions.

3. Images above 1 MP give different answers from system_one()

d1's runner.py shrinks every picture to at most 1024×1024 px (bicubic) before the processor; here it goes in as sent, so the tile grid and the pixels differ. Same commit as 2 (LiquidStrategy.max_image_pixels).

Repro, vllm serve LiquidAI/d1-3B --port 18676, img/cat03.jpg = Wikimedia Commons Cat03.jpg

# Server: vllm serve LiquidAI/d1-3B --port 18676
# The same photo sent as-is (1600x1598) and downscaled to 1 MP the way d1's runner.cap_pixels does.
# model.system_one() always sees the 1 MP version, so only the second row matches the reference.
import base64, io, json, math, os, sys, urllib.request
from PIL import Image
URL = f"http://127.0.0.1:{os.environ.get('PORT', '18676')}/v1/systemone"
im = Image.open(sys.argv[1] if len(sys.argv) > 1 else "img/cat03.jpg").convert("RGB")
s = math.sqrt(1024 * 1024 / (im.width * im.height))
for tag, pic in [("as sent ", im), ("1MP-capped", im.resize((int(im.width * s), int(im.height * s)), Image.Resampling.BICUBIC))]:
    b = io.BytesIO(); pic.save(b, "PNG")
    body = {"model": "LiquidAI/d1-3B", "state": None, "images": ["data:image/png;base64," + base64.b64encode(b.getvalue()).decode()],
            "questions": {"indoor": {"type": "noul", "instructions": "Is the photo taken indoors?"}}}
    r = json.load(urllib.request.urlopen(urllib.request.Request(URL, json.dumps(body).encode(), {"Content-Type": "application/json"})))
    print(tag, pic.size, "P(indoor) =", round(r["answers"]["indoor"]["noul"], 4), "input_tokens =", r["usage"]["input_tokens"])

Observed: as sent P(indoor) = 0.1824, 2578 tokens; capped 0.0953, 1293 tokens.
Expected: system_one() on the same file gives 0.0953 with 1293 tokens. In fp32 a 3840×2880 photo flips a yes/no answer (0.454 vs 0.551).

4. A one-letter option name that is several tokens fails with a max() error

"丙".isalpha() is True, so 丙 becomes a code; 丙 and " 丙" are both multi-token, the group is [], and the check (L305) compares [] with ids[:1] == [] and passes. Fix (one line, not ids or): simon-veitner-redhat@fee6ea2

Repro, vllm serve LiquidAI/d1-3B --port 18676

# 丙 is one letter (isalpha) but two d1 tokens -> empty label group.
curl -s -w '\nHTTP %{http_code}\n' http://127.0.0.1:${PORT:-18676}/v1/systemone -H 'Content-Type: application/json' \
  -d '{"model":"LiquidAI/d1-3B","state":"Average quality.","questions":{"g":{"type":"choice","instructions":"Grade?","criteria":{"甲":"good","丙":"average"}}}}'

Observed: 400 "max() iterable argument is empty".
Expected: the intended 400 "label '丙' is not one distinct token for this model" (system_one() answers, using fallback code A).

5. confidence does not match system_one()

The reference returns probs[best]; here it is probs[top] * label_mass, with label_mass summing every pooled variant. Picks and probabilities match; confidence is 0.004–0.020 lower on d1. Keeping one definition across models seems fine; the commit narrows the docs claim to picks and probabilities: simon-veitner-redhat@3760918

Repro, vllm serve LiquidAI/d1-3B --port 18676

# The reference's choice confidence is max(probabilities); vLLM returns max(probabilities) * label_mass.
curl -s http://127.0.0.1:${PORT:-18676}/v1/systemone -H 'Content-Type: application/json' -d '{"model":"'${MODEL:-LiquidAI/d1-3B}'","state":"I was charged twice this month, please refund one of them.","questions":{"team":{"type":"choice","instructions":"Which team should handle this?","criteria":{"billing":"Charges, refunds, invoices","technical":"App or site faults","fraud":"Suspected unauthorised use"}}}}' \
 | jq -c '{confidence: .answers.team.confidence, max_prob: ([.answers.team.probabilities[]]|max), label_mass: .diagnostics.team.label_mass, max_prob_x_mass: (([.answers.team.probabilities[]]|max) * .diagnostics.team.label_mass)}'

Observed: {"confidence":0.97587,"max_prob":0.98423,"label_mass":0.99150}.
Expected (system_one(), same input): confidence 0.98286, its max probability.

Run-to-run variance (fixed separately)

Under concurrent load a label's bf16 logit moves by one step, so near-ties flip: the Japanese case picks English in about half of loaded runs. Computing the lm_head in fp32 removes this at no measured cost (322 vs 324 req/s at 32 in flight), but --hf-overrides '{"head_dtype": "float32"}' is ignored for multimodal models today; I will send that as a separate [Bugfix] PR. Until then --hf-overrides '{"text_config": {"head_dtype": "float32"}}' works.

Repro, vllm serve LiquidAI/d1-3B --port 18676

# Server: vllm serve LiquidAI/d1-3B --port 18676   (bf16, default prefix caching)
# The PR's Japanese case asked alone, then 32 times mixed with 32 other requests, 32 in flight.
# Batch shape moves one label logit by one bf16 step: Japanese 0.3985 / English 0.3744 becomes
# a 0.3837 tie, and argmax picks English.
import collections, json, os, random, urllib.request
from concurrent.futures import ThreadPoolExecutor
URL = f"http://127.0.0.1:{os.environ.get('PORT', '18676')}/v1/systemone"
LANG = {"type": "choice", "instructions": "Which language is the message in?", "criteria": {"English": None, "Japanese": None, "Chinese": None}}
JA = {"model": "LiquidAI/d1-3B", "state": {"message": "注文した商品がまだ届きません。返金してください。"}, "questions": {"lang": LANG}}
def other(i):  # a different request with a random-length state
    return {"model": "LiquidAI/d1-3B", "state": "refund order crash " * random.Random(i).choice([5, 50, 300]), "questions": {"d": {"type": "noul", "instructions": "Angry?"}}}
def ask(body):
    r = json.load(urllib.request.urlopen(urllib.request.Request(URL, json.dumps(body).encode(), {"Content-Type": "application/json"})))
    a = r["answers"].get("lang")
    return a and (a["choice"], round(a["probabilities"]["Japanese"], 4), round(a["probabilities"]["English"], 4))
print("alone:", ask(JA))
jobs = [JA] * 32 + [other(i) for i in range(32)]
random.Random(0).shuffle(jobs)
with ThreadPoolExecutor(32) as ex:
    print("mixed:", collections.Counter(a for a in ex.map(ask, jobs) if a))

Observed: alone: ('Japanese', 0.3985, 0.3744), mixed: Counter({('Japanese', 0.3985, 0.3744): 17, ('English', 0.3837, 0.3837): 15}).
Expected: all 32 equal to the answer asked alone (with the fp32 head, all 32 pick Japanese).

This review was prepared with AI assistance (Claude). I reviewed Claude's findings, and all of them were reproduced on GPU.

[mgoin]

/ci run

[github-actions]

✅ Triggered Buildkite CI #94240 for commit d74817609455.

[vllm-agent]

CI selector (shadow): 1 test steps (1 jobs) instead of 34 (47 jobs)

Shadow mode: this changes nothing about what CI runs. It shows what the evidence-based selector would pick for this PR, next to today's rules. How it works.

Feedback welcome: reply here if it would skip a step this change needs, or runs something unrelated.

steps (jobs)	Today's rules	Selector	Would skip	Would add
NVIDIA, CPU and others	34 (47)	1 (1)	33 (46)	0 (0)
AMD mirrors	33 (41)	1 (1)	32 (40)	0 (0)

Selector would run (1)

• entrypoints-integration-api-server-generate

Would skip (today's rules run them) (33)

• basic-correctness ×2
• basic-correctness-cpu-offload
• basic-correctness-cumem
• basic-correctness-prefetch-offload
• basic-correctness-sleep-mode
• basic-models-test-other-cpu
• basic-models-tests-initialization
• basic-models-tests-other
• benchmarks-cli-test
• cpu-language-generation-and-pooling-model-tests ×3
• entrypoints-integration-api-server ×4
• entrypoints-integration-api-server-openai-chat_completion
• entrypoints-integration-api-server-openai-completion
• entrypoints-integration-llm
• entrypoints-integration-multimodal
• entrypoints-integration-pooling
• entrypoints-integration-responses-api
• entrypoints-integration-speech_to_text
• entrypoints-unit-tests
• examples
• kernels-fla-ops-test-b200
• kernels-mhc-test-b200
• kernels-root-misc-test-b200
• language-models-tests-granite-l4-compatibility
• language-models-tests-hybrid ×2
• language-models-tests-standard
• multi-modal-models-standard-1-qwen2
• multi-modal-models-standard-2-qwen3-gemma
• multi-modal-models-standard-3-llava-qwen2-vl
• multi-modal-models-standard-4-other-whisper
• multi-modal-processor ×4
• multi-modal-processor-cpu ×4
• rl-entrypoints-tests

Would add (today's rules do not run them) (0)

none

AMD mirrors: would skip (32)

• basic-correctness ×2
• basic-correctness-cpu-offload
• basic-correctness-cumem
• basic-correctness-prefetch-offload
• basic-correctness-sleep-mode
• basic-models-tests-initialization
• basic-models-tests-other
• benchmarks-cli-test
• entrypoints-integration-api-server ×4
• entrypoints-integration-api-server-openai-chat_completion
• entrypoints-integration-api-server-openai-completion
• entrypoints-integration-llm
• entrypoints-integration-multimodal
• entrypoints-integration-pooling
• entrypoints-integration-responses-api
• entrypoints-integration-speech_to_text
• entrypoints-unit-tests
• examples
• kernels-fla-ops-test-b200
• kernels-mhc-test-b200
• kernels-root-misc-test-b200
• language-models-tests-granite-l4-compatibility
• language-models-tests-hybrid ×2
• language-models-tests-standard
• multi-modal-models-standard-1-qwen2
• multi-modal-models-standard-2-qwen3-gemma
• multi-modal-models-standard-3-llava-qwen2-vl
• multi-modal-models-standard-4-other-whisper
• multi-modal-processor ×4
• platform-tests
• pytorch-compilation-passes-unit-tests
• rl-entrypoints-tests

AMD mirrors: would add (0)

none

_{3 changed files · base a686c8e1b7 · head d748176094 · Python record: build 93039 at 1e5d0ea888 · kernel record: table 74f4404 (build 94125), map 74f4404 · not counted: 11 build steps, 5 A100 steps the generator no longer emits, 0 optional steps the selector would also run}