[Frontend] Tokenize the /v1/systemone state once per decision and let big decisions take turns by simon-veitner-redhat · Pull Request #61251 · vllm-project/vllm

simon-veitner-redhat · 2026-10-12T16:57:57Z

Overview

Fixes #61249. The decision endpoints, /v1/systemone and /v1/decisions, now tokenize the shared state once per decision instead of once per question. A 64-question decision on a 27k-token state now runs at about the speed of its engine work. Big decisions also take turns in the engine, so a burst of them does not make small decisions wait for the whole burst. This applies to next-token models and to Decision 3.0 (D3) models.

The branch is on main 7020600.

Claims

On Qwen3-4B, a 64-question decision on a 27k-token state takes 0.91 s instead of 2.91 s on its own. At 16 concurrent decisions it takes 8.24 s instead of 45.6 s (p50). /v1/decisions shows the same gains: 0.87 s instead of 2.74 s, and 7.65 s instead of 43.8 s.
During a burst of 64 big decisions, a 1-question decision waits 1.99 / 3.49 s p50 / p95 (2.02 / 60.5 s on main). Throughput is 65 questions/s instead of 21.
Answers are unchanged. With VLLM_BATCH_INVARIANT=1, all 634 Qwen3-4B answers on both endpoints are bit-identical to main.
DiffusionGemma gets the same kind of win: 3.49 s → 1.59 s on its own, and 49.5 s → 12.7 s p50 at 16 concurrent decisions.
On D3 (vllm-sr/d3-mini), a bystander waits 5.53 / 7.17 s instead of 58.7 / 109.7 s during a burst of 32 big decisions, at the same throughput.

Validation

Every run compares this branch with main 7020600. Each tree runs on its own H200. The server flags are --max-model-len 32768 and --gpu-memory-utilization 0.90, with the default --renderer-num-workers 1. For Qwen3-4B this gives a max_num_batched_tokens of 8192.

Each concurrent client has its own state, and every state is warmed once before the timed rounds. Every decision has 64 questions unless a row says otherwise.

Bystander: a 1-question decision on a short state, sent every second during the burst.
Floor: /v1/completions with max_tokens=1 on the same 64 prompts as token ids, so it measures the engine work alone.

No run had errors or preemptions.

Qwen3-4B, 27k-token state

endpoint	concurrent decisions	tree	decision p50 / p95	questions/s	floor p50	bystander p50 / p95
`/v1/systemone`	1	main	2.91 / 2.93 s	22	1.11 s
`/v1/systemone`	1	this branch	0.91 / 0.91 s	71	1.07 s
`/v1/systemone`	16	main	45.6 / 46.1 s	22	15.6 s
`/v1/systemone`	16	this branch	8.24 / 14.3 s	72	15.9 s
`/v1/systemone`	64	main	168.6 / 193.6 s	21		2.02 / 60.5 s
`/v1/systemone`	64	this branch	32.6 / 60.1 s	65		1.99 / 3.49 s
`/v1/systemone`	128	main	337.5 / 387.7 s	21		3.99 / 125.4 s
`/v1/systemone`	128	this branch	63.4 / 117.5 s	66		1.83 / 7.21 s
`/v1/decisions`	1	main	2.74 / 2.78 s	23
`/v1/decisions`	1	this branch	0.87 / 0.89 s	73
`/v1/decisions`	16	main	43.8 / 44.4 s	23
`/v1/decisions`	16	this branch	7.65 / 13.2 s	78
`/v1/decisions`	64	main	163.1 / 188.1 s	21		1.96 / 58.2 s
`/v1/decisions`	64	this branch	32.9 / 59.9 s	65		1.85 / 3.53 s

Cells at 1 and 16 decisions ran 2 rounds; cells at 64 and 128 ran 1 round. The prefix cache hit rate was 0.98 or higher everywhere.

A decision can beat the floor because the floor sends 64 separate completion requests per decision.

Qwen3-4B, other shapes

These are ungated decisions and 1-question decisions, which keep the per-question render.

endpoint	questions × state	concurrent decisions	main p50 / p95	this branch p50 / p95	questions/s main → branch
`/v1/systemone`	64 × 175 tokens	64	2.84 / 3.11 s	1.46 / 2.23 s	1,338 → 1,814
`/v1/decisions`	64 × 175 tokens	64	3.08 / 3.34 s	1.48 / 2.27 s	1,238 → 1,759
`/v1/systemone`	8 × 4k	1	0.07 / 0.08 s	0.04 / 0.04 s	109 → 201
`/v1/systemone`	8 × 4k	16	0.86 / 0.87 s	0.35 / 0.35 s	149 → 368
`/v1/systemone`	8 × 4k	64	3.34 / 3.40 s	1.19 / 1.21 s	153 → 424
`/v1/systemone`	1 × 27k	1	0.06 / 0.06 s	0.05 / 0.06 s	16 → 18
`/v1/systemone`	1 × 27k	16	0.29 / 0.50 s	0.28 / 0.48 s	30 → 32
`/v1/systemone`	1 × 27k	64	17.6 / 37.9 s	17.4 / 37.3 s	2 → 2
`/v1/decisions`	1 × 27k	16	0.30 / 0.52 s	0.28 / 0.48 s	30 → 32
`/v1/systemone`	1 × 175 tokens	64	0.07 / 0.11 s	0.07 / 0.08 s	731 → 842

1-question decisions are unchanged within noise. At 64 concurrent 1 × 27k decisions, the 64 distinct states do not fit in the KV cache, on both trees.

DiffusionGemma

google/diffusiongemma-26B-A4B-it on one H200 is served with --diffusion-config '{"canvas_length": 64}' --max-logprobs 32 --enable-prefix-caching and the flags above. It uses the 27k-token state, one round per cell, on /v1/systemone.

questions	concurrent decisions	tree	decision p50 / p95	questions/s
64	1	main	3.49 / 3.49 s	18
64	1	this branch	1.59 / 1.59 s	40
64	16	main	49.5 / 51.1 s	20
64	16	this branch	12.7 / 20.8 s	48
1	16	main	0.39 / 0.78 s	20
1	16	this branch	0.44 / 0.63 s	25

Decision 3.0

vllm-sr/d3-mini is served with the flags above. D3 runs without the prefix cache and without chunked prefill, so max_num_batched_tokens is 32768 and every question prefills the whole state.

questions × state	concurrent decisions	tree	decision p50 / p95	questions/s	bystander p50 / p95
64 × 4k	1	main	4.04 / 4.05 s	16
64 × 4k	1	this branch	3.76 / 3.77 s	17
64 × 4k	8	main	17.1 / 29.7 s	17
64 × 4k	8	this branch	16.8 / 29.3 s	17
64 × 4k	32	main	61.2 / 112.5 s	17	58.7 / 109.7 s
64 × 4k	32	this branch	62.1 / 113.7 s	17	5.53 / 7.17 s
64 × 27k	1	main	34.0 / 34.0 s	2
64 × 27k	1	this branch	31.7 / 31.7 s	2
64 × 177 tokens	16	main	2.33 / 3.95 s	254
64 × 177 tokens	16	this branch	2.32 / 3.95 s	254
1 × 4k	16	main	0.65 / 0.94 s	17
1 × 4k	16	this branch	0.66 / 0.95 s	17

D3's engine work dominates its latency, so tokenizing once saves only the tokenization: 0.3 s at 4k and 2.2 s at 27k. The gate leaves D3 decision latency and throughput unchanged, and it keeps the bystander fast.

Answers

The fixed set has 28 requests per endpoint and 317 answers per endpoint:

20 support tickets, half of them with instructions;
5 and 64 questions on 200-, 4k- and 27k-token states;
a JSON state;
a state that ends in whitespace.

Requests are sent one at a time, and the set is dumped twice per server.

Qwen3-4B with VLLM_BATCH_INVARIANT=1. On both endpoints, 634 of 634 answers are bit-identical between main and this branch, and between two runs of each tree.
Qwen3-4B, default mode. input_tokens is identical on every request. Main against this branch agrees on the argmax in 316 of 317 answers per endpoint. Main against itself agrees on 317 and 315. The probabilities differ only through batching: with one decision's reads arriving together, they share engine steps differently than on main.
DiffusionGemma. Argmax agreement between the trees is 316 / 317 and 315 / 317. Main against itself gives 313 / 317 and 310 / 317. input_tokens is identical everywhere.
D3 (/v1/systemone only; batch-invariant mode does not support its GDN layers). Argmax agreement is 248 / 248, with max |Δp| 6.2e-3, the same as main against itself. input_tokens is identical everywhere.

Test plan

pytest tests/entrypoints/generate/structured_decisions/test_strategies.py \
       tests/entrypoints/generate/structured_decisions/test_decisions.py \
       tests/entrypoints/generate/structured_decisions/test_question_types.py
# need a GPU
pytest tests/entrypoints/generate/structured_decisions/test_structured_decisions_e2e.py \
       tests/entrypoints/openai/decisions/test_decisions_e2e.py

CPU tests: 122 passed.
e2e tests: 15 passed.
ruff is clean. mypy reports nothing new.

New tests in test_strategies.py:

test_spliced_prompts_match_full_render checks that spliced prompts equal the full render for Qwen3, Qwen3.5, Qwen3.5-MoE, DiffusionGemma and LiquidAI d1. It covers several states, including ones with paragraphs, with and without instructions.
test_token_id_renders_skip_the_splice checks that a render returning token ids instead of text falls back.
test_splice_falls_back_when_tokens_join_across_the_split covers a tokenizer that merges characters across the split point.
test_splice_matches_full_render_when_newlines_cross_the_split covers a run of newlines that crosses the split point and is longer than the head window.
test_oversized_state_fails_like_full_render checks that an oversized state gives the same error as the full render.
test_images_skip_the_splice checks that a request with images renders each question in full.
test_big_decisions_take_the_gate checks which decisions hold the gate during their reads. A decision over the token budget does. A decision under the budget, a 1-question decision and any decision under the priority policy do not. It also checks that the slot is free again afterwards.
test_d3_spliced_prompts_match_full_encode checks that D3 prompts equal the full encode for the Qwen3 and Qwen3.5 tokenizers, and that a long state is tokenized once per decision.
test_big_d3_decisions_take_the_gate checks that without the prefix cache a D3 decision counts the state once per question, and that with it the state counts once.

The decision-server tests now run on the real Qwen3 renderer, so multi-question reads on both endpoints take the spliced path.

Details

Root cause. NextTokenStrategy.read rendered each question's user turn through preprocess_chat, once per question. That is two renderer pool jobs per question: the chat template and a full tokenization. The pool has one worker by default.

A 64-question decision on a 27k-token state spent about 2 s there, and every other request queued behind it. D3Strategy.read encoded each question's full prompt with the tokenizer on the event loop.

Tokenize once. For more than one question, the next-token read renders every question's prompt with the chat template, as the per-question path does. It then splits the rendered texts after the last \n\n they all share. That point falls right after the state in each strategy's own prompt format, before LiquidAI d1's QUESTION: header.

The read then tokenizes the text before the split once. For each question it tokenizes a short window: the last 64 characters before the split plus the question's own text. A decision costs N template renders, one tokenization of the shared prefix and N + 1 short ones, instead of a full tokenization per question.

Each prompt is the prefix ids plus the ids of its window after the head. The joined prompts still go through preprocess_cmpl, so the token length check and cache_salt apply as before. The prefix goes through the renderer's tokenize path with the request's tokenize params, so an oversized state still fails the character check before tokenization. A single question keeps the per-question render, which is cheaper there.

D3 builds its own prompts, so it applies the same split and head check to its prompt texts, after the media prefix when the request has media.

The read falls back to the full per-question render in these cases:

the request has images (next-token models);
the renderer returns token ids instead of text (Mistral tokenizers, prompt embeds);
the prompts share no \n\n;
a question's text after the split starts with whitespace, which could join a newline run longer than the head window;
the last 64 characters before the split tokenize differently with a question after them.

The last check is a heuristic for tokenizers whose split depends on the next characters.

OnlineRenderer.build_chat_params is now shared with preprocess_chat.

Why the gate. With tokenization fast, every decision in a burst sends all its reads to the engine within a second or two. The engine serves them in arrival order. So a 1-question decision that arrives during a burst of long decisions waits for the whole burst.

The gate. It lives in the frontend.

A decision with more than one question counts as big when its prompts hold more tokens than max_num_batched_tokens.
With the prefix cache on, the shared state counts once, as the cache computes it once. Without it, as for D3, every prompt counts in full.
At most two big decisions have reads in the engine at a time, and the rest wait in FIFO order.
An admitted decision still sends all its reads at once, so its state stays in the prefix cache.
Short decisions and 1-question decisions skip the gate.
Under --scheduling-policy priority the gate is off, because the engine already orders requests by priority.
The gate reads the scheduler and cache config when a decision arrives.

One gate for both endpoints. /v1/systemone and /v1/decisions share one read strategy, so they share one gate. A big decision on either endpoint takes a slot from the other.

Coverage. Qwen3, Qwen3.5, Qwen3.5-MoE, DiffusionGemma, LiquidAI d1 and D3 splice and are gated. DiffusionGemma and d1 inherit read() from the next-token strategy. Clef-Omni and Clef-Flash encode their own prompts and read a whole decision as one pooling request, so they never splice and are not gated.

Limits.

The gate is per API server process, so N API servers admit up to 2N big decisions. api_server_count defaults to the data-parallel size.
--max-num-queued-reqs and --max-num-queued-tokens do not count decisions waiting at the gate, so a waiting decision can still get a 503 when its reads are added.
Under --scheduling-policy priority the gate is off. A bystander then needs a lower priority value than the burst to get through quickly.
When every decision in a burst shares one cached state, the gate trades some throughput for the bystander's latency.
1-question decisions on a long state are not gated. A bystander behind 64 of them waits as it would behind 64 long completions.
Generative scoring (generative_scoring/serving.py) fans out its reads the same way. I left that for a separate change.

Size. +193 / −48 non-test lines, in strategies.py and online_renderer.py.

Pull Request Checklist

I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).
AI assistance was used during the creation of this PR.
Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.
Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.
Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.
Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

AI assistance (Claude Code) was used to investigate, write and benchmark this change. I reviewed the code and the results.

BEFORE SUBMITTING, PLEASE READ https://docs.vllm.ai/en/latest/contributing (anything written below this line will be removed by GitHub Actions)

claude

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

mergify · 2026-10-12T17:03:01Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @simon-veitner-redhat.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

mergify · 2026-10-13T06:18:15Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @simon-veitner-redhat.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

mergify · 2026-10-14T00:05:59Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @simon-veitner-redhat.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

/v1/systemone rendered the chat template and tokenized the whole state once per question, in two renderer pool jobs each. A 64-question decision on a 27k-token state spent about 2 s of CPU there and queued every other request behind it on the single-worker pool. For more than one question, NextTokenStrategy now renders a probe with a marker after the last "\n\n" that every question's user turn shares, which ends the state. It tokenizes the text before the marker once and builds each prompt as those ids plus the ids of the rest of its prompt. That takes three render pool jobs per decision instead of two per question: the probe render, the prefix tokenization, and one job that renders every question to check it and tokenizes the short question windows. The two template jobs run under the chat template render timeout; the prefix tokenization is not timed, as in preprocess_chat. A single question keeps the per-question render, which is cheaper there. The prefix goes through the renderer's tokenize path with the request's tokenize params, so an oversized state fails the character check before tokenization, as before. The joined prompts then go through preprocess_cmpl for the token length check and cache_salt. When the prefix fits but the whole prompt does not, the token length error reports the exact count instead of "at least". The read falls back to the per-question render when the request has images, the renderer is not HfRenderer, the user turns share no "\n\n", the marker is not found exactly once, the text before it does not end with the "\n\n" separator, the template changes any question's text, or the last 64 characters of the prefix tokenize differently with a question after them. That window check is a heuristic backstop for tokenizers whose split depends on the next characters, such as GPT-2. OnlineRenderer.build_chat_params is shared with preprocess_chat, and HfRenderer.run_chat_template_job_async runs one template job on the render pool. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Simon Veitner <sveitner@redhat.com>

With the state tokenized once, every decision in a burst sends all its reads to the engine within a second or two. The engine admits them all and serves them in arrival order, so a 1-question decision that arrives during a burst of long decisions waits for the whole burst. A decision with more than one question whose prompts hold more tokens than max_num_batched_tokens, counting the shared state once as the prefix cache does, needs more than one engine step of prefill. Such decisions now take turns: at most two have reads in the engine at a time, and the others wait in FIFO order. Each admitted decision still sends all its reads at once, so its state stays in the prefix cache. Shorter decisions and 1-question decisions, which are a single engine request, skip the wait. Under --scheduling-policy priority the gate is off, since the engine orders requests by priority itself. Qwen3-4B on one H200, 64 clients, each with its own state, 2 bursts per cell (128 decisions), a 1-question bystander sent every second. Parent vs this change, run side by side: | workload | bystander p50 / p95 | decision p50 | questions/s | |-----------|------------------------------|----------------|--------------| | 64q x 27k | 34.7 / 65.6 -> 2.0 / 5.4 s | 42.2 -> 33.1 s | 58 -> 65 | | 8q x 27k | 21.3 / 40.8 -> 1.1 / 1.5 s | 20.3 -> 19.5 s | 12 -> 12 | | 16q x 175 | n=2, burst under 1 s | 0.43 -> 0.41 s | 1448 -> 1516 | | 1q x 27k | 20.0 / 37.2 -> 19.3 / 37.0 s | 17.0 -> 17.6 s | 2 -> 2 | Throughput holds for these workloads. 16q x 175 stays under the budget, and so does 64q x 175 (14.7k prompt tokens, 3.3k with the state counted once): about 1,850-1,900 questions/s either way, run side by side. The rows above were measured with the state counted per question, which gates each of them the same way; one burst each of 64q x 27k and 8q x 27k on this revision gave a bystander 2.1 / 6.2 s and 1.1 / 1.6 s. 1q x 27k is not gated either, and a bystander there waits as it would behind 64 long completions. With --scheduling-policy priority and the bystander sent at priority -1 (gate off), 64q x 27k gives a bystander 1.9 / 16.5 s and 60 questions/s. Limits: - The gate is per API server process, so N API servers admit up to 2N long decisions; api_server_count defaults to the data-parallel size. - --max-num-queued-reqs/tokens do not count decisions waiting at the gate, so a waiting decision can still get a 503 when its reads are added. - DiffusionGemma inherits read() and is gated the same way, on its prompt tokens. Clef-Omni reads a whole decision as one pooling request, so it is not gated. Generative scoring (generative_scoring/serving.py) fans out its reads the same way; that is left for a separate change. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Simon Veitner <sveitner@redhat.com>

A question whose text after the split point (the last blank line the questions share) begins with whitespace can extend a newline run that starts before the 64-character head window. The head check then passes while the run tokenizes differently in the full prompt, so the spliced ids differ from the per-question render. This showed up with byte-level BPE tokenizers. Render such decisions per question instead. Signed-off-by: Simon Veitner <sveitner@redhat.com> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

Render each question's prompt with render_messages_async, one render pool job at a time as the per-question path does, and split the rendered texts after the last blank line they all share. This drops the marker probe, the checks of the probe against the full renders, the HfRenderer check and the render pool helper in hf.py. A template that edits the question no longer forces a fallback. The splice gives up at the first render that returns token ids instead of text. The tests of the removed guards go too. Signed-off-by: Simon Veitner <sveitner@redhat.com> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

The decision server fixture and the state init test now build their engines with a vllm_config, which every strategy reads for the big decision gate. The server fixture renders through the real Qwen3 OnlineRenderer instead of a stub, so multi-question reads on both /v1/systemone and /v1/decisions take the spliced path. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Simon Veitner <sveitner@redhat.com>

D3 reads now wait at the same big-decision gate as next-token reads, and tokenize the start their prompts share once. The gate counts a shared state once only when the prefix cache is on, since D3 runs without it and prefills the state for every question. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Simon Veitner <sveitner@redhat.com>

The strategy constructor no longer reads the engine client's config, so a strategy can be built from a context that has no engine yet. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Simon Veitner <sveitner@redhat.com>

mergify · 2026-10-19T03:20:00Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @simon-veitner-redhat.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

chaunceyjiang

@simon-veitner-redhat Thanks for your work! I think this is a great idea.

However, I think we could put this PR on hold for now and stick with the current simpler implementation to keep development moving quickly.

We can revisit this PR once things are more mature. What do you think?

simon-veitner-redhat · 2026-10-22T12:33:19Z

@simon-veitner-redhat Thanks for your work! I think this is a great idea.

However, I think we could put this PR on hold for now and stick with the current simpler implementation to keep development moving quickly.

We can revisit this PR once things are more mature. What do you think?

agree. will convert to draft

simon-veitner-redhat requested review from AndreasKaratzas, DarkLight1337, NickLucche, aarnphm, njhill and robertgshaw2-redhat as code owners October 12, 2026 16:57

claude Bot reviewed Oct 12, 2026

View reviewed changes

mergify Bot added the frontend label Oct 12, 2026

mergify Bot added the needs-rebase label Oct 12, 2026

simon-veitner-redhat force-pushed the systemone-state-tokenize-once-fairness-draft branch 3 times, most recently from 4440158 to a2de129 Compare October 12, 2026 18:18

mergify Bot removed the needs-rebase label Oct 12, 2026

SCcagg5 mentioned this pull request Oct 12, 2026

[Frontend] systemone: prefill the shared prefix once before the reads on hybrid models #61192

Merged

6 tasks done

mergify Bot added the needs-rebase label Oct 13, 2026

simon-veitner-redhat force-pushed the systemone-state-tokenize-once-fairness-draft branch from a2de129 to 807d872 Compare October 13, 2026 08:52

mergify Bot removed the needs-rebase label Oct 13, 2026

chaunceyjiang self-assigned this Oct 13, 2026

simon-veitner-redhat force-pushed the systemone-state-tokenize-once-fairness-draft branch from 807d872 to d5aa557 Compare October 13, 2026 10:26

mergify Bot added the needs-rebase label Oct 14, 2026

gty111 mentioned this pull request Oct 15, 2026

[Model] Add Clef 27B regression coverage and serving docs #61729

Closed

6 tasks done

simon-veitner-redhat and others added 6 commits October 15, 2026 09:35

simon-veitner-redhat force-pushed the systemone-state-tokenize-once-fairness-draft branch from d5aa557 to 811d101 Compare October 15, 2026 11:20

mergify Bot removed the needs-rebase label Oct 15, 2026

Vigilans mentioned this pull request Oct 17, 2026

[Frontend] Accept images on /v1/systemone for multimodal next-token models #62181

Merged

6 tasks done

cipheraxat mentioned this pull request Oct 18, 2026

[Frontend] Serve Gemma 4 on /v1/systemone and /v1/decisions #62303

Merged

6 tasks done

webcodes-cz mentioned this pull request Oct 18, 2026

[Bugfix][0.32] Keep Clef Omni media safe after schema refusal #62316

Draft

5 of 6 tasks

mergify Bot added the needs-rebase label Oct 19, 2026

twu3202 mentioned this pull request Oct 19, 2026

[RFC]: Next-phase roadmap: prefill-only execution, reuse and scheduling InferMatrixLab/system1-omni#150

Open

1 task done

chaunceyjiang reviewed Oct 19, 2026

View reviewed changes

Xunzhuo mentioned this pull request Oct 20, 2026

[Model] Support Decision 3.0 prefix_bidir attention (vllm-sr/d3-turbo) #62584

Open

6 tasks done

simon-veitner-redhat marked this pull request as draft October 22, 2026 12:33

waizuichougou mentioned this pull request Oct 25, 2026

[Bugfix][Frontend] Preserve multimodal inputs in /v1/systemone prefix warmup #63543

Open

5 of 6 tasks

taneem-ibrahim mentioned this pull request Oct 25, 2026

[Pooling] Reduce Clef metadata overhead #63495

Merged

jperezdealgaba mentioned this pull request Oct 29, 2026

[Security] Enforce request template trust for structured decisions #64503

Merged

5 of 6 tasks

Navigation Menu

Repository navigation

[Frontend] Tokenize the /v1/systemone state once per decision and let big decisions take turns - #61251

[Frontend] Tokenize the /v1/systemone state once per decision and let big decisions take turns#61251
simon-veitner-redhat wants to merge 7 commits into
vllm-project:mainfrom
simon-veitner-redhat:systemone-state-tokenize-once-fairness-draft

simon-veitner-redhat commented Oct 12, 2026 •

edited

Loading

Uh oh!

claude Bot left a comment

Uh oh!

mergify Bot commented Oct 12, 2026

Uh oh!

mergify Bot commented Oct 13, 2026

Uh oh!

mergify Bot commented Oct 14, 2026

Uh oh!

mergify Bot commented Oct 19, 2026

Uh oh!

chaunceyjiang left a comment

Uh oh!

simon-veitner-redhat commented Oct 22, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

simon-veitner-redhat commented Oct 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Claims

Validation

Qwen3-4B, 27k-token state

Qwen3-4B, other shapes

DiffusionGemma

Decision 3.0

Answers

Test plan

Details

Uh oh!

claude Bot left a comment

Choose a reason for hiding this comment

Claude Code Review

Uh oh!

mergify Bot commented Oct 12, 2026

Uh oh!

mergify Bot commented Oct 13, 2026

Uh oh!

mergify Bot commented Oct 14, 2026

Uh oh!

mergify Bot commented Oct 19, 2026

Uh oh!

chaunceyjiang left a comment

Choose a reason for hiding this comment

Uh oh!

simon-veitner-redhat commented Oct 22, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

simon-veitner-redhat commented Oct 12, 2026 •

edited

Loading