Skip to content

[ROCm] Restore Inkling MTP backend parity - #50806

Merged
tjtanaa merged 3 commits into
mainfrom
akaratza_inkling_mtp_parity
Aug 5, 2026
Merged

[ROCm] Restore Inkling MTP backend parity#50806
tjtanaa merged 3 commits into
mainfrom
akaratza_inkling_mtp_parity

Conversation

@AndreasKaratzas

Copy link
Copy Markdown
Member
  • Synchronize the backend-neutral AMD MTP orchestration with the NVIDIA implementation updated by PR [Model Runner V2][Spec Decode] Add multi-layer MTP speculator #48892.
  • Select and construct only the checkpoint depth layers needed by the configured speculative-token count.
  • Route each speculative step to its matching depth and ignore checkpoint weights for depths that were not constructed.
  • Preserve AMD-specific kernels through the existing relative imports for model, normalization, attention, and short-convolution components.
  • Keep the AMD and NVIDIA MTP source files byte-identical as required by the existing ROCm alignment contract.

PR #48892 added multi-layer MTP orchestration only to the NVIDIA copy and left the intentionally aligned AMD file stale. That omission causes the byte-alignment check to fail in the Inkling step. Inkling currently permits only one speculative token, so this restores the supported MTP=1 path without enabling unvalidated ROCm MTP>1 execution. The complete ROCm alignment test file passes locally.

Signed-off-by: Andreas Karatzas <akaratza@amd.com>
@mergify mergify Bot added the rocm Related to AMD ROCm label Aug 3, 2026
@github-project-automation github-project-automation Bot moved this to Todo in AMD Aug 3, 2026
@Fangzhou-Ai

Copy link
Copy Markdown
Contributor

Thanks @AndreasKaratzas LGTM!

@AndreasKaratzas
AndreasKaratzas marked this pull request as ready for review August 3, 2026 15:32

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.

Tip: disable this comment in your organization's Code Review settings.

@tjtanaa

tjtanaa commented Aug 4, 2026

Copy link
Copy Markdown
Member

Can you share some validation results?

@AndreasKaratzas

Copy link
Copy Markdown
Member Author

@tjtanaa Looks more or less the same to me.

Revision GSM8K flexible GSM8K strict Accepted / drafted Acceptance
Base 5c4fe4b 95.15% 93.63% 158,330 / 191,658 82.6107%
PR 7049a5d, run 1 96.44% 94.09% 153,960 / 186,256 82.6604%
PR 7049a5d, repeat 95.45% 93.40% 157,036 / 189,994 82.6531%

Draft acceptance rate is effectively unchanged: both head runs are approximately +0.04 - 0.05 percentage points versus base. GSM8K also remains within normal run-to-run variation.

@tjtanaa tjtanaa added the ready ONLY add when PR is ready to merge/full CI is needed label Aug 4, 2026
@tjtanaa
tjtanaa enabled auto-merge (squash) August 4, 2026 04:43
@tjtanaa
tjtanaa merged commit 33c5058 into main Aug 5, 2026
65 checks passed
@tjtanaa
tjtanaa deleted the akaratza_inkling_mtp_parity branch August 5, 2026 02:02
@github-project-automation github-project-automation Bot moved this from Todo to Done in AMD Aug 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready ONLY add when PR is ready to merge/full CI is needed rocm Related to AMD ROCm

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

3 participants