Conversation
Signed-off-by: vx120 <893600387@qq.com>
Signed-off-by: vx120 <893600387@qq.com>
Signed-off-by: vx120 <893600387@qq.com>
Signed-off-by: vx120 <893600387@qq.com>
Signed-off-by: vx120 <893600387@qq.com>
Remove unnecessary blank line before the if statement. Signed-off-by: vx120 <57470515+vx120@users.noreply.github.com>
Signed-off-by: vx120 <893600387@qq.com>
Clarify error message for top_k parameter in sampling distribution replay. Signed-off-by: vx120 <57470515+vx120@users.noreply.github.com>
Signed-off-by: vx120 <893600387@qq.com>
Sampling masks are emitted only for FINAL_ONLY outputs, yet default cumulative requests were silently accepted. Validate the supported output mode when the request is admitted and keep the temperature check alongside it. Co-authored-by: Codex <noreply@openai.com> Signed-off-by: aoshen02 <aoshen@inferact.ai>
Avoid accessing and slicing sampling masks in the scheduler update-from-output path when the feature is disabled. Co-authored-by: Codex <noreply@openai.com> Signed-off-by: aoshen02 <aoshen@inferact.ai>
Validate sampling replay request output mode
Co-authored-by: Codex <noreply@openai.com> Signed-off-by: aoshen02 <aoshen@inferact.ai>
…tput Compact sampling mask before async output
Gate sampling masks in scheduler
Co-authored-by: Codex <noreply@openai.com> Signed-off-by: aoshen02 <aoshen@inferact.ai>
…-mask-log Remove FlashInfer sampling mask log
Co-authored-by: Codex <noreply@openai.com> Signed-off-by: aoshen02 <aoshen@inferact.ai>
…g-mask Disable FlashInfer sampling for sampling masks
Co-authored-by: Codex <noreply@openai.com> Signed-off-by: aoshen02 <aoshen@inferact.ai>
Keep sampling mask tensors local to sampler
Co-authored-by: Codex <noreply@openai.com> Signed-off-by: aoshen02 <aoshen@inferact.ai>
Extract sampling mask output payload
Co-authored-by: OpenAI Codex <noreply@openai.com> Signed-off-by: aoshen02 <aoshen@inferact.ai>
Restore sampler spacing
Co-authored-by: OpenAI Codex <noreply@openai.com> Signed-off-by: aoshen02 <aoshen@inferact.ai>
- Change SamplingMask from CSR (flat token_ids + offsets) to list[list[int]] for a user-friendly API surface. - Add SamplingMaskLists.to_nested_list() for internal CSR → nested list conversion at the output boundary. - Expose sampling_mask in chat completion, completion, and token-in-token-out endpoint response protocols. - Add unit test for the conversion path. Depends on vllm-project#49577 (mask replay). Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
aoshen02
requested review from
ApostaC,
BugenZhao,
DarkLight1337,
ProExpertProg,
WoosukKwon,
aarnphm,
alexm-redhat,
chaunceyjiang,
heheda12345,
hmellor,
houseroad,
ivanium,
mgoin,
njhill,
orozery,
robertgshaw2-redhat,
russellb,
tlrmchlsmth,
yewentao256,
youkaichao and
ywang96
as code owners
July 31, 2026 04:20
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
Only the token-in-token-out /inference/v1/generate endpoint needs this; the OpenAI /v1/completions and /v1/chat/completions endpoints should not carry vLLM-specific RL training fields. Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-up to #49577 (mask replay). This PR:
Changes
SamplingMaskfrom CSR tolist[list[int]]— the CSR format (flattoken_ids+offsets) is efficient for internal GPU→CPU transfer but unnecessarily complex for the user-facing API. The internal pipeline (SamplingMaskTensors→SamplingMaskLists) still uses CSR; conversion to nested lists happens once at the output boundary viaSamplingMaskLists.to_nested_list().Exposes
sampling_maskin HTTP endpoint responses — adds the field to chat completion, completion, and token-in-token-out (generate) endpoint protocols. The mask isnullwhenenable_return_sampling_maskis off or the request was aborted.Response format
{ "choices": [{ "sampling_mask": [ [101, 55, 77, 88], [202, 33, 44], [303, 11, 22] ] }] }Each inner list contains the vocabulary token IDs that survived top-k/top-p/min-p filtering for the corresponding generated token.
Motivation
Without endpoint support, the sampling mask data stops at
CompletionOutputand is only accessible via the offlineLLM.generate()API. RL training frameworks (verl, slime/miles) that call vLLM over HTTP cannot consume it. This format (list[list[int]]) is also consistent with SGLang'soutput_token_sampling_mask.Test plan
test_sampling_mask_lists_to_nested_list— verifies CSR → nested list conversiontest_sampling_mask_tensors_tolist,test_sampling_mask_tensors_from_logits,test_sampling_mask_import)--enable-return-sampling-mask --logprobs-mode processed_logprobs(requires GPU, depends on [Feature] Mask Replay #49577)🤖 Generated with Claude Code
Co-authored-by: Claude Opus 4.6 (1M context) noreply@anthropic.com