Skip to content

[CI][Spec Decode] Add MTP placeholder-token regression coverage - #54893

Merged
AndreasKaratzas merged 2 commits into
mainfrom
akaratza_cohere_spec_mtp
Sep 2, 2026
Merged

AndreasKaratzas merged 2 commits into
mainfrom
akaratza_cohere_spec_mtp

Conversation

@AndreasKaratzas

@AndreasKaratzas AndreasKaratzas commented Sep 2, 2026

Copy link
Copy Markdown
Member
  • Add a focused MTP regression test that compares emitted token IDs with a non-speculative greedy baseline.
  • Exercise both PIECEWISE and FULL_AND_PIECEWISE CUDA graph modes and verify that the requested mode is not silently downgraded.
  • Force deterministic draft rejection with a nonzero allowed token ID so any leaked placeholder slots are immediately observable as an output mismatch.
  • Assert that the MTP drafter produced drafts and accepted none, while also exercising the logprobs output-assembly path.

Speculative decoding is an exact optimization at temperature zero, so its emitted token IDs must match ordinary greedy decoding token for token. A previous MTP regression under FULL CUDA graphs instead exposed the complete padded speculative width rather than truncating it to the accepted prefix. Rejected positions use a placeholder token ID, which is converted to token ID 0 before logprob gathering and can therefore become silent output corruption if padded positions escape the output-assembly boundary. With the GLM-5.1 tokenizer, token ID 0 renders as !, producing responses filled with repeated exclamation marks. The engine continued running normally and reported healthy acceptance statistics, so liveness checks and acceptance-rate tests could not detect the problem. PIECEWISE execution also remained correct, meaning coverage that does not explicitly exercise a FULL graph mode can miss the regression. Comparing emitted token IDs in both graph modes, while proving that rejection actually occurred, provides a deterministic and tokenizer-independent guard for this defect class.

AI assistance was used for duplicate research, test design, implementation, validation, and PR preparation. The human submitter reviewed the resulting change and remains responsible for it end to end.

Co-authored-by: OpenAI Codex <codex@openai.com>
Signed-off-by: Andreas Karatzas <Andreas.Karatzas@amd.com>

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.

Tip: disable this comment in your organization's Code Review settings.

@AndreasKaratzas

Copy link
Copy Markdown
Member Author

/ci run

@github-actions

github-actions Bot commented Sep 2, 2026

Copy link
Copy Markdown

✅ Triggered Buildkite CI #86788 for commit a09382972a92.

@AndreasKaratzas
AndreasKaratzas merged commit 87deddc into main Sep 2, 2026
34 checks passed
@AndreasKaratzas
AndreasKaratzas deleted the akaratza_cohere_spec_mtp branch September 2, 2026 08:29
mylibrar pushed a commit to tanyuqian/vllm that referenced this pull request Sep 3, 2026
…-project#54893)

Signed-off-by: Andreas Karatzas <Andreas.Karatzas@amd.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
sheralskumar pushed a commit to sheralskumar/vllm that referenced this pull request Sep 8, 2026
…-project#54893)

Signed-off-by: Andreas Karatzas <Andreas.Karatzas@amd.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants