Repository navigation
Conversation
…arams vLLM decode reads kv_transfer_params.prompt_token_ids, builds the engine input from them, and skips chat templating and tokenization (vllm-project/vllm#48145). Pass the ids render produced on the chat completions, Responses and Messages paths so decode does not re-render a prompt the coordinator already tokenized. Multimodal requests keep re-rendering messages: engines without vllm-project/vllm#55771 drop multi_modal_data on this path. Signed-off-by: roytman <roytman@il.ibm.com>
github-actions
Bot
added
size/M
shimib
added a commit
to shimib/llm-d-inference-scheduler
that referenced
this pull request
Nov 3, 2026
…code hits and stop buffering when nothing is rewritten Follow-ups to llm-d#3751 from the post-merge review on that PR. A served conditional-decode hit is a KV-reuse case like any other, but the step streamed the worker's response straight to the client, so a renderer drift there went unseen. The parity setup moves out of the decode step into a shared helper, and conditional-decode runs it on its probe: the check is installed on the writer, a streamed probe is asked for include_usage on a cloned body so a 412 continues the pipeline with the client's body untouched, and the step takes the same prompt_token_parity parameter. With warn as the default, every decode response with rendered tokens went through the usage writer, which dropped Content-Length unconditionally and held a non-streaming body until the end before forwarding it, purely to read usage. The writer now has a passthrough mode for the observe-only case: it forwards every write as it arrives, keeps Content-Length, and reads usage on the side (complete SSE lines as they come, a bounded copy of a JSON body at the flush). The cached-token rewrite and the injected-frame drop are unchanged. ids_forwarded now reads the leg's body instead of the route: the generate route, completions with the IDs as the prompt, or kv_transfer_params .prompt_token_ids on the chat-shaped routes. A step that starts forwarding IDs (llm-d#3741) needs no change here. A stream_options that is present but not an object is forwarded as the client sent it and the check is skipped, instead of being replaced by a valid object. Tests: missing usage is a no-op, the completions route carries its own label values, a malformed stream_options is left alone, observe-only keeps Content-Length, and the conditional-decode hit, miss, streaming and mode cases. Docs: the api_type values are APIType.String() (vllm_generate, not generate), the check needs a render step, and conditional-decode gained the parameter in the architecture table and the sample config. The parameter key moves to the const block in params.go. The field-name constants stay local until llm-d#3756 lands, which moves neighbouring names into pkg/common/request. Part of llm-d#3748. Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: Shimi Bandiel <shimib@google.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What type of PR is this?
/kind feature
/kind performance
What this PR does / why we need it:
The coordinator's
renderstep tokenizes every chat completions, Responses and Messages request, but thedecodestep forwards the client's original body, so the decode worker applies the chat template and tokenizes the same prompt again.vLLM decode reads
kv_transfer_params.prompt_token_ids, builds the engine input from those ids, and skips chat templating and tokenization (vllm-project/vllm#48145, in v0.30.0). This PR sets that field on the decode leg fromreqCtx.TokenIDs, next to the transfer params the KV connector builds. The request stays on the client's route, so streaming, tool and reasoning parsing, and the response envelope are vLLM's own.The field is not set when:
multi_modal_dataon this path, so decode keeps renderingmessagesrenderstep in the pipeline)/v1/completionsor/inference/v1/generate, which already carry token ids inpromptandtoken_idsconditional-decodeis unchanged; it sends nokv_transfer_params.Verified on CoreWeave H200 with Qwen3-VL-32B-Instruct,
kv-nixl, one encode, prefill and decode pod, vLLM nightly5d35ce39:prompt_token_ids(17 ids, matching prefill'sremote_num_tokens); an image request's did not; both answered correctlymessages: a request to the decode worker whosemessagesasked for the capital of France, withprompt_token_idsrendered from the same question about Germany, answered "Berlin"Which issue(s) this PR fixes:
Part of #3656 (path A, "coordinator sets the field on the decode leg for chat, messages, responses").
Release note (write
NONEif no user-facing change):Test plan:
prompt_token_idsfor chat completions, Responses and Messages, and omits it for multimodal requests and requests render did not tokenizemake presubmitgo test ./pkg/coordinator/...)