Repository navigation
[Feat] Structured decisions endpoint (/v1/systemone) - #59299
Conversation
POST /v1/systemone answers typed questions about a state with a probability for every allowed answer, in the request and answer shapes of Jev's decision API. Until now this lived only in the DiffusionGemma example server, which calls vLLM over HTTP. Each question is one read on the engine client: the chat prompt with the assistant reply prefilled up to "id:", one generated token, and logprob_token_ids set to the question's label tokens. Softmax over the labels gives the answer. The reads share the system prompt and the state, so prefix caching prefills the state once per request. Question types are plugins: a QuestionType subclass registered with register_question_type parses its criteria, names its labels and shapes its answer, and the prompt, slot resolution and reads are shared. This PR registers "choice" only. depends_on, ask_if and alone are refused with a 400 until they are implemented. The route is registered with the other generate routes and served only when the model supports "generate". Tests: unit tests for the registry, the choice type and label resolution on the Qwen3 tokenizer, and an end-to-end test on Qwen/Qwen3-0.6B (13 and 7 passing on a GB10 against the nightly-ddd6fbca image with this package overlaid). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Matt Mastracci <matthew@mastracci.com>
A question with a field other than type, instructions and criteria now gets a 400 naming the fields, replacing the fixed list of refused fields (depends_on, ask_if, alone). An ignored field could be one that changes the answer, so unknown fields are errors whatever they are. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Matt Mastracci <matthew@mastracci.com>
…rategies Decision templates. The system prompt of a decision now comes from a Jinja template rendered in the same sandboxed environment as chat templates. The built-in template keeps the previous wording. --decision-template takes a file path or inline text, loaded like --chat-template, and a request may carry its own decision_template when the server runs with --trust-request-chat-template. A template may define an answer_prefix(question) macro, which the server calls to prefill each read, so a template that asks for a different reply format reads at the matching place. Without the macro the prefix is "<id>:". Per-type option wording moved from QuestionType.describe into the template. Read strategies. A registry picks a ReadStrategy per model at startup: the first whose supports() accepts the model serves every decision. The existing next-token read moved into NextTokenStrategy, which declines diffusion models, so DiffusionGemma gets no decision route until a canvas strategy exists. Each strategy reports DecisionLimits (questions per request, choices per question) for its model, and the server enforces them. For next-token reads that is 64 questions and the engine's 128 logprob_token_ids. The 26-alternative bound is now a property of letter labels in QuestionType.labels, not a global. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Matt Mastracci <matthew@mastracci.com>
The request-template test asserted which option a 0.6B model picks under a terse custom prompt, which is the model's call, not the server's. It now checks that the macro's prefix lands the read on a label (label mass above 0.5) and that the probabilities differ from the default template's on the same input. Adds a 400 case for a template that does not compile. Docs: decision templates, the answer_prefix macro, the limits, and that diffusion models are not served yet. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Matt Mastracci <matthew@mastracci.com>
Answer macro. A decision template now says how an answer is written with
answer(question, label), "<id>: <label>" by default, in place of
answer_prefix(question). The server renders the answer once per label and
finds the one token where the labels differ: the tokens before it are the
read's prefill, and the differing token is each label's id. A template can
therefore use any answer format ("(A)", "id -> A") without the server
assuming a space before the label, and the system prompt can call the same
macro to show the reply format. The default template does.
Templates render once per request with the real variables, so a template
body that uses questions at the top level works, and its macros see them.
Loading only compiles the template.
Review cleanups: "alternatives" is now "options" everywhere, to match the
template variables, the docs and the Jev criteria. The strategy docstring
says an unsupported model gets a 501 from the route, which is what happens.
The serving class catches only StructuredDecisionError, so other errors
reach the server's handler with their own status. The router documents the
501. The strategy resolves every prompt before it submits any request.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Matt Mastracci <matthew@mastracci.com>
Drops comments that describe a piece by comparing it with another (decision templates against chat templates, the next-token strategy against diffusion) and docstrings that restate their declaration. Drops tests that another test already covers or that check trivial math: label softmax, the template without instructions, the default answer, a label changing two tokens, answer repeatability, the unknown-model 404 and the HTTP form of a template compile error. The HTTP validation test keeps one case each for unknown fields and the question limit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Matt Mastracci <matthew@mastracci.com>
A model class now names the structured-decision read strategy it needs with a decision_read_strategy class variable, "next_token" by default, recorded in _ModelInfo beside the other capability flags. The server registers strategies by name and picks the one the model names. A name with no registered strategy gets the 501. DiffusionGemma names "canvas", so it keeps the 501 until a canvas strategy is registered. This replaces supports(), which tested ModelConfig.is_diffusion and relied on registration order. limits() is now an instance method, so a strategy can size its limits from the served config (a canvas from the diffusion config) through its context. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Matt Mastracci <matthew@mastracci.com>
…zer, shuffle per question
Letter labels were assumed to be one token after "id: ". That holds for
Qwen3 and Gemma with the default template, but not for every template:
without the space, Qwen3 fuses the colon into 15 of the 26 letters (":A"),
so a template like "id:label" would read a mix of fused and unfused slots.
The server now tries A..Z then AA..ZZ in the template's answer the first
time it uses a template with a tokenizer. It groups the labels that land in
one token by the tokens around them and by what else that token holds
(a fused space, a fused colon, nothing), and keeps the largest group. On
Qwen3-0.6B the default format keeps 552 labels, and "q:L" keeps the 528
unfused ones over the fused colon group. Digits give only 10, and
letter-digit or two-digit labels are never one token on Qwen3 or
DiffusionGemma, so letters stay.
Each question takes single letters first, shuffled with a seed from a
SHA-256 of its id, type, instructions and options, and falls back to
two-letter labels only past 26 options. Shuffling the whole group made
things worse: two-letter labels carry meaning (the language question gave
English "TM" and French "LT", and the model answered French at 0.999 on
an English ticket). Seeding from the question keeps the prompt stable for
prefix caching. The default prompt now says labels are chosen randomly.
A question may have at most min(labels kept, strategy limit) options, and
the error names that number. The next-token strategy's limit is still the
128 logprob_token_ids cap.
The e2e tests used to pass because both right answers had label A. With
shuffled labels Qwen3-0.6B gets the second of two semantic questions right
0 to 2 times in 10 random label draws (Qwen3-4B: 9 to 10), so the tests
now ask copy questions whose answer is in the state (0.94 and 0.997 on the
real labels), and the request-template test no longer asserts label mass,
which depends on how the model reads a custom format.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Matt Mastracci <matthew@mastracci.com>
A request may set `seed`, which is mixed into each question's label shuffle. Reading a question under several seeds and averaging the answers cancels the model's preference for particular labels. Without a seed the labels still come from the question's content alone, so repeated requests keep one prompt for prefix caching. Measured on the original semantic e2e questions (a double-charge ticket: team billing/shipping/security, language English/French), 320 seeds: - Qwen3-4B is right on 310/320 single reads and on every average of 2 or more seeds. - Qwen3-0.6B gets worse with averaging (team 65/320 single, 0/20 at 16 seeds; language 28/320, 0/20). Its bias is option position, not labels: it picks the last-listed option. With the options reversed it is right on 240/320 and 309/320; with the order shuffled per seed its answers sit near chance (billing 0.38, English 0.53). The e2e tests keep the copy questions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Matt Mastracci <matthew@mastracci.com>
The e2e test asserted one read's answers, which only proves the one label draw its question hash picks. A new parametrized test averages each question's probabilities over seeds 0 to 15 and asserts the argmax, on three question sets Qwen3-0.6B gets right in every 16-seed group (10 of 10 groups over 160 label draws each): - copy team + copy language from a JSON state (150 and 148 of 160 single reads right) - "mentions a card?" yes + sentiment angry (147 and 134 of 160) - sky color blue (159 of 160) The same battery found what 0.6B cannot do under shuffled labels: questions whose answer is "no" (5 of 160, a yes bias) and naming a message's language (near chance), so neither is in the test. test_choice_decision keeps the response-shape checks without asserting answers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Matt Mastracci <matthew@mastracci.com>
|
Documentation preview: https://vllm--59299.org.readthedocs.build/en/59299/ |
Drops the references to Jev's decision API, which vLLM readers have no
context for. Rewrites comments and docs that used nouns as verbs ("names
the strategy", "shapes the answer", "the group wins", "every label reads
the same way") into plain statements, and cuts docstrings that restated
a field or return type (AnswerSlot fields, RenderedDecision.slot). Adds
one comment on why the label sweep skips text that does not round-trip
through the tokenizer.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Matt Mastracci <matthew@mastracci.com>
Request-supplied Jinja is the part of the first PR a security review would slow down on, and nothing needs it yet. The server now compiles its one decision template (--decision-template or the built-in) at startup, which also makes a broken template stop startup. The decision_template request field, the --trust-request-chat-template plumbing, select_template/compile_template and their tests go. The startup check for fewer than two labels goes too: every question needs at least two options, so a template that leaves fewer labels already fails each request with "at most N options for this model". Also cuts two comments that argued instead of stating a condition. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Matt Mastracci <matthew@mastracci.com>
…docs
Replaces figures of speech and wording that speaks of the model as if it
wanted things ("wanted to say something else", "deserves less trust",
"sit inside one token", "left open", "line up") with the concrete
condition. Claims about seeds now say averaging reduces label bias
rather than cancels it. The choice-criteria error now says a value may
be null, which the parser already accepted. The docs no longer mention
request templates in the error sentence, and the cost line says what
each further question prefills.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Matt Mastracci <matthew@mastracci.com>
pre-commit's mypy hook failed on four files: - The read loop indexed output.logprobs after a check mypy could not narrow through result.outputs[0]. The check now binds output first and tests result, output and output.logprobs directly. - BaseServing._check_model and _maybe_get_adapters take a closed union of request types. The decision request is passed with type: ignore[arg-type], as generative_scoring already does. - The strategy test's SimpleNamespace stand-in for ModelConfig is typed Any. - max(t, key=t.get) has an overloaded key type; it is a lambda now. Checked with tools/pre_commit/mypy.py and mypy 1.20.2 for Python 3.10 and 3.13. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Matt Mastracci <matthew@mastracci.com>
A vLLM developer reported accuracy dropping with every question after the first. Every read rendered all of the request's questions into the system prompt, which asks for "one line per question, in this order", then prefilled the reply at the read's own line. For any question after the first, the reply started at a line the prompt said comes later. Measured with four questions (copy a team name, "mentions a card?" yes, sentiment, sky color), each rotated through all four positions, 48 label seeds each, 192 reads per position: layout Qwen3-0.6B pos 1-4 Qwen3-1.7B pos 1-4 all questions (before) 156 129 117 115 159 114 163 171 all questions, "in any order" 157 134 128 121 170 114 164 169 prefill earlier answers 156 158 151 171 159 137 181 188 own question, after the state 165 165 165 165 185 185 185 185 Each read now puts the state first, then the decision template rendered for its question alone, in the user turn. Accuracy no longer depends on position, and the total is the highest of the four (0.6B 86% against 67%, 1.7B 96% against 79%). Reads still run in parallel. The shared prefix is now the state rather than the system prompt plus the state, so each read prefills its own question text, a few dozen tokens. The default template text is unchanged, since it is what was measured. RenderedDecision.system_text becomes text, and the docs describe the template as the text of one question's read. Tested on a GB10 (gx10) with this branch's files over a vllm-openai nightly image: 26 passed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Matt Mastracci <matthew@mastracci.com>
…usion, confidence) From Lucas Wilkinson's review: - The one-token label read moves to vllm/entrypoints/generate/label_reads.py as next_token_label_reads(). /generative_scoring and the decision endpoint both call it. Each caller still builds its own SamplingParams, so /generative_scoring keeps its sampling settings, request ids, input logging and error messages. - The read strategy comes from model_config.is_diffusion. The model registry hook (decision_read_strategy on the model class, _ModelInfo, interfaces.py, DiffusionGemma's "canvas") and the strategy name registry go until the diffusion PR has a second strategy to register. - --decision-template goes, so the template text stays internal. The docs drop the decision template section. - confidence was the top label's share among the labels, which can read about 1.0 when label_mass is 2e-14. It is now that share times label_mass: the chosen label's probability over the whole vocabulary. Tested on a GB10 (gx10) with this branch's files over a vllm-openai nightly image: structured_decisions and generative_scoring, 42 passed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Matt Mastracci <matthew@mastracci.com>
IRONICBo
left a comment
There was a problem hiding this comment.
One compatibility edge case in the request validator:
|
✅ Triggered Buildkite CI #93493 for commit |
chaunceyjiang
left a comment
There was a problem hiding this comment.
Great work!
I tested it locally, and it works well.
I have two minor, non-blocking suggestions that we can address in a follow-up PR:
- I suggest adding a
cache_saltparameter and passing it to the core engine. - I suggest adding
get_trace_headershandling, consistent with the other endpoints.
|
/ci run |
|
❌ @chaunceyjiang, Only reviewers with write access can use CI commands before CI is delegated to the PR author. |
Co-authored-by: Chauncey <chaunceyjiang@gmail.com> Signed-off-by: Matt Mastracci <matthew@mastracci.com>
|
/ci run |
|
✅ Triggered Buildkite CI #93598 for commit |
mgoin
left a comment
There was a problem hiding this comment.
LGTM, just one small issue we can deal with in a followup
|
|
||
|
|
||
| def state_text(state: Any) -> str: | ||
| return state if isinstance(state, str) else json.dumps(state) |
There was a problem hiding this comment.
should use ensure_ascii=False in dumps to not scramble non-ASCII text. we can do this as a quick followup
…state prompts (#60651) Signed-off-by: Francisco Javier Arceo <4163062+franciscojavierarceo@users.noreply.github.com> Co-authored-by: Misha Goin <mgoin64@gmail.com>
* feat(decisions): add hosted_vllm provider Adds HostedVLLMDecisionsConfig so hosted_vllm/<served model> works on POST /v1/systemone and POST /v1/decisions against a vLLM server that serves the System One body (vllm-project/vllm#59299, after v0.31.0). vLLM only supports choice questions and rejects others, so the base decisions config gains a per-provider health_check_questions attribute and the evaluation-mode health probe sends a choice question for hosted_vllm while every other provider keeps the noul probe. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(decisions): move the hosted_vllm placeholder key to constants Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: kerry <kerry@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Align answers, confidence, and error bodies with llama.cpp and SGLang, and step aside when vLLM vllm-project#59299 already owns the route. Refuse a request whose label token ids do not match the loaded decision model. Co-authored-by: Grok 4.7 <cursoragent@cursor.com>
* feat(decisions): add hosted_vllm provider Adds HostedVLLMDecisionsConfig so hosted_vllm/<served model> works on POST /v1/systemone and POST /v1/decisions against a vLLM server that serves the System One body (vllm-project/vllm#59299, after v0.31.0). vLLM only supports choice questions and rejects others, so the base decisions config gains a per-provider health_check_questions attribute and the evaluation-mode health probe sends a choice question for hosted_vllm while every other provider keeps the noul probe. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(decisions): move the hosted_vllm placeholder key to constants Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: kerry <kerry@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> (cherry picked from commit 61e5f2d)
Purpose
This PR was prepared with the help of AI, but reviewed by myself. This PR description is fully human-written with the exception of the
curlcommand below and the test result one-liner.The current structured decisions server lives as an example. This example is untuned, requires additional setup and could run more effectively inside of vLLM itself.
This implements a new endpoint, with a single question type (
choice), for AR models only:POST /v1/systemoneExample:
{ "id": "decision-80bd0a693f255e1e", "object": "structured_decision", "created": 1790714266, "model": "Qwen/Qwen3-0.6B", "answers": { "team": { "type": "choice", "choice": "billing", "probabilities": {"billing": 0.955, "shipping": 0.008, "security": 0.037}, "confidence": 0.955 }, "lang": { "type": "choice", "choice": "French", "probabilities": {"English": 0.0006, "French": 0.9994}, "confidence": 0.9994 } }, "usage": {"input_tokens": 254, "output_tokens": 2}, "diagnostics": { "team": {"label_mass": 0.998, "argmax_is_label": true}, "lang": {"label_mass": 0.9999, "argmax_is_label": true} } }Limits: 64 questions (only
choice), and options capped at the smaller of the tokenizer's supported label set (see below) and 128.For a generic endpoint, we need to consider a few things:
Tokenization
Each model tokenizes differently, so we must ensure that all choice-type questions provide the model a choice between N single-token options in a way that supports the answer format. The server tries labels A to ZZ in the template's answer format and keeps the largest group
Templates
Decision templates are fixed - we may decide to offer a template in the future if someone needs it.
Strategies
Decision strategies vary by model:
next_tokencovers AR, an upcoming diffusion strategy will cover diffusion and other decision-specific models may wish to ship their own. The model usesdecision_read_strategyto map the strategy.Models without a registered read strategy return 501.
What's upcoming in future patches in the series:
Test Plan
Tests included, against Qwen, proving the request type works.
pytest tests/entrypoints/generate/structured_decisionsTest Result
28 passed on a GB10 (DGX Spark), with this branch's files mounted over a
vllm-openainightly image.Essential Elements of an Effective PR Description Checklist
supported_models.mdandexamplesfor a new model.