serve: structured decisions, one scored forward per request

[mmastrac]

Note: this is for diffgemma only

Matching vLLM PR: vllm-project/vllm#57250

---

Overview

A system message that is a JSON question schema turns a /v1/chat/completions request into one scored denoise forward: the user message is the state, the reply is a probability distribution per question, and no text is generated. The answer template (id: label per question) is seeded into the canvas with each label slot as noise, and the label distribution is read from the logits at that slot, so every question in a request costs the same forward and each slot conditions on the whole template bidirectionally. Modeled on TypeSafe's Jev API (choice / score / yes-no questions with probabilities and confidence).

What is in it

• structured.rs: schema parsing and detection, prompt and template rendering, tokenizer-verified single-token labels, reply JSON.
• PipelineOp::Score / score_canvas: prefill, seed, run N forwards, read logits rows. Commits nothing; the conversation is finalized with the schema prefix alone so every later state on the schema reuses it.
• Request knobs for the experiments below: samples, steps, active, hole, climb, climb_mode.
• Prefill: a short tail chunk runs at a 64-row multiple instead of the full canvas. Bit-identical KV (new narrow_prefill_chunk_bit_identity test, golden 8/8); a 51-token delta drops from 1.3 s to 0.67 s. DGQ_PREFILL_NARROW=0 restores full width.
• Rebased on perf(serve): build the empty-reply predicate once per worker #20 (the empty-reply predicate built once per worker); the scoring path uses the shared per-request config.
• ARCHITECTURE correction: a cross-turn delta always takes the fast quantized resume; the force flag never reaches it.

Numbers (M3 Pro, q4 pack, three questions, ~190-token prompt, one server process)

Case	Wall
First request on a schema (f32 engine prefill, once)	8 to 15 s
Next state, default (4 reads averaged, 64 rows)	3.1 s (0.7 prefill + 4 × 0.6 forward)
Next state, 4 reads, active: 256	5.8 s
Next state, samples: 1	1.3 s
Next state, samples: 1, active: 256	2.0 s
Generating the same three answers as JSON (19 tokens, 3 to 5 full-canvas steps)	9.6 to 13.1 s

Same labels from both paths on all three test tickets.

Findings recorded in PLAN.md

• A joint hill climb (write the top labels back, re-read from step 0) converges in one round and only ratifies the first read to 1.00. A leave-one-out climb avoids that but adds nothing a seed sweep does not show.
• The hole noise alone flips a borderline answer: 16 seeds split 10/6 on the outage ticket's "urgent", each read at 0.8 or better. A read is the denoiser's posterior given one noise draw, so the reply now averages 4 reads by default (the Monte Carlo estimate of the marginal) and reports stderr and agreement; on that ticket "urgent" is yes 0.60 ± 0.16 at agreement 0.75, and the easy tickets stay unanimous at 1.00 ± 0.00.
• Prefill precision matters for the schema prefix, not the state: engine-prefilled schema gives "urgent" yes 0.56 to 0.68 across arms, fast-prefilled schema gives no 0.80, with the state's own precision within noise. The mixed default (engine schema once, fast state deltas) keeps the reference precision on the question wording.
• Canvas width: at 32 reads per width the unambiguous answers are identical at both 64 and 256 rows and the borderline one is within 1.3 standard errors (yes 0.56 ± 0.07 vs 0.68 ± 0.06), so the width now defaults to the smallest 64-row multiple that holds the template (active: 256 opts back).
• Profiled under samply: the pipeline thread's CPU-active time over a session is under a second; the levers are GPU dispatch shape.

Gates

• diffgemma golden: 8/8 byte-identical on the rebased tree.
• narrow_prefill_chunk_bit_identity: four fingerprints identical.
• Unit tests pass, except ten pre-existing Tier-1 half-kernel oracle failures that also fail on origin/main (one bf16 ulp, from the RNE rounding change; CPU oracles still truncate). Tracked separately.

🤖 Generated with Claude Code