Conversation
Contributor
|
This pull request has merge conflicts that must be resolved before it can be |
Contributor
|
This pull request has merge conflicts that must be resolved before it can be |
justtestingthingsx
pushed a commit
to meandmyboiclaude/vllm
that referenced
this pull request
Aug 7, 2026
…g (+DP support) Skips draft proposal/sampling for requests whose drafts can never be consumed (non-final prefill chunks, guaranteed-final decode steps), propagated via SchedulerOutput.no_draft_req_ids, plus DP draft-skip sync. Includes DP block-table/dp_utils/speculator changes and tests (merged clean). BEHAVIOR NOTE (rule 3): the skip is NOT behind a user opt-in flag. It is gated on `vllm_config.use_v2_model_runner` (report_unusable_drafts / can_skip_lookahead are False unless the V2 model runner is active, and can_skip_lookahead additionally excludes dflash/dspark). The legacy V1 runner path is unaffected; on the V2 runner + spec decode this changes default behavior by design (that is the PR's purpose). Merge notes: preserved our vllm-project#40757 zero-collapse floor guard verbatim (untouched, outside conflict region) and our vllm-project#48944 DynamicSDLookup type for self.dynamic_sd_lookup + dynamic_sd_ctx_agg; added PR's report_unusable_drafts/can_skip_lookahead flags and no_draft_req_ids SchedulerOutput kwarg alongside our ec_manager_metadata. Co-authored-by: vLLM contributors (cherry picked from commit 467b5e9) (cherry picked from commit 29994b7)
justtestingthingsx
pushed a commit
to meandmyboiclaude/vllm
that referenced
this pull request
Aug 7, 2026
justtestingthingsx
pushed a commit
to meandmyboiclaude/vllm
that referenced
this pull request
Aug 11, 2026
…g (+DP support) Skips draft proposal/sampling for requests whose drafts can never be consumed (non-final prefill chunks, guaranteed-final decode steps), propagated via SchedulerOutput.no_draft_req_ids, plus DP draft-skip sync. Includes DP block-table/dp_utils/speculator changes and tests (merged clean). BEHAVIOR NOTE (rule 3): the skip is NOT behind a user opt-in flag. It is gated on `vllm_config.use_v2_model_runner` (report_unusable_drafts / can_skip_lookahead are False unless the V2 model runner is active, and can_skip_lookahead additionally excludes dflash/dspark). The legacy V1 runner path is unaffected; on the V2 runner + spec decode this changes default behavior by design (that is the PR's purpose). Merge notes: preserved our vllm-project#40757 zero-collapse floor guard verbatim (untouched, outside conflict region) and our vllm-project#48944 DynamicSDLookup type for self.dynamic_sd_lookup + dynamic_sd_ctx_agg; added PR's report_unusable_drafts/can_skip_lookahead flags and no_draft_req_ids SchedulerOutput kwarg alongside our ec_manager_metadata. Co-authored-by: vLLM contributors (cherry picked from commit 467b5e9) (cherry picked from commit 29994b7)
justtestingthingsx
pushed a commit
to meandmyboiclaude/vllm
that referenced
this pull request
Aug 22, 2026
…re-pads an already-padded layer to the shared page (boot AssertionError class); commensurate per-token-bytes zero guard; AR speculator draft-prefill dispatches on num_tokens_padded (restores upstream vllm-project#47352, reverted by carried vllm-project#48244 pick); gemma4_dspark + laguna_dflash fused-KV dtype derived from norm weight + bias cast (fp16 drafters); unpadded() propagates mm_req_doc_ranges; V1 runner zeroes padded-row num_prompt_tokens_cpu (stale-length dummy-row class); fused_recurrent + fused_sigmoid_gating INPLACE_FINAL_STATE load masked to the row (SM-fault class, mirrors vllm-project#50021's sibling bound); profile_cudagraph_memory empty-sample guard; TQ spec head_size_v; packed-codec (kvarn_/turboquant_) branch in _validate_cache_dtype logging; max_page_block_lcm includes padded-to-max layers (PR vllm-project#52804 intent) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Member
Author
|
/ci run |
|
✅ Triggered Buildkite CI #87016 for commit |
Contributor
|
This pull request has merge conflicts that must be resolved before it can be |
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Claude Signed-off-by: Nick Hill <nickhill123@gmail.com>
njhill
force-pushed
the
skip-drafting
branch
from
September 3, 2026 19:44
21f9f5d to
ce14a64
Compare
Contributor
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: trueThanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
|
This pull request has merge conflicts that must be resolved before it can be |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Avoid spec decode drafting and lookahead allocation for requests/steps where it isn't necessary (non-final prefill chunks or based on
max_tokens).This is a similar optimization to those from @NickLucche in #45280 and #45283, with a benefit that it doesn't rely on config and can apply in some more cases (e.g. aggregated scenarios with chunked prefills).
However the config-based approach should still be worthwhile/complimentary since it means we can avoid capturing cuda graphs for the extra eagle steps and draft sampling (for full CG case) altogether. It also works naturally with DP.
I did not apply this to MRV1 since we are in the process of deprecating it.
Performance comparison
Setup
Qwen/Qwen3-8B(fp16) + eagle3 drafterAngelSlim/Qwen3-8B_eagle3,num_speculative_tokens=3vllm bench throughput --dataset-name random --random-input-len 16384 --random-output-len 1 --num-prompts 64 --max-model-len 20480 --seed 42, i.e. 1,048,576 prompt tokens total, 1 output token per requestResults (total tokens/s, higher is better)