Repository navigation
Conversation
The draft block built its attention metadata eagerly before every replay and copied it into the captured buffers, keeping those kernels and copies on the host path of each draft step. Capture the three raw inputs instead, as the decode and target-verify paths already do, and expand them inside the graph at the capture sequence bound the eager build already used.
jhinpan
force-pushed
the
perf/amd-dspark-draft-metadata-in-graph
branch
from
September 24, 2026 04:51
be6f5f2 to
2cdd132
Compare
4 of 5 tasks
Collaborator
Author
|
Closing: on |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
The HIP DSV4 backend builds its decode and target-verify metadata inside the CUDA graph from three raw inputs (
DSV4RawDecodeMetadata,DSV4RawVerifyMetadata), so a replay only copies those inputs. The DSpark draft block still builds its full metadata eagerly before every replay and then copies it into the captured buffers, which keeps those kernels and copies on the host path of every draft step.Modifications
DSV4RawDraftBlockMetadataand route the draft-block graph through it. Replay refreshesreq_pool_indices,seq_lensandout_cache_loc; the graph expands them withinit_forward_metadata_dspark_draft_blockatMAX_SEQ_LEN_FOR_CAPTURE, the bound the eager path already passed.Accuracy Tests
GSM8K (64 fixed questions, natural EOS) and 8 retrieval probes on every server: 61-62/64 GSM8K and 8/8 retrieval in both arms. On this base batch-1 outputs already differ between fresh servers of the same arm (base vs base: 15/64 identical GSM8K sequences, 62/64 same answer), and the base-vs-PR agreement stays in that range (12-15/64 identical, 61-64/64 same answer).
Speed Tests and Profiling
MI355X x4, TP4/EP4, DeepSeek-V4.1-Flash
dba1be0a, this branch ate2e824dc58, the cookbook MI350X Low-Latency cell (DSpark block 5), AITER built as in this branch's Dockerfile. Both arms also carry #8's int64 FlashMLA store fix. Fresh servers per arm, real acceptance, no profiler during timing.Batched runs report cycle time (batch x acceptance length / decode tok/s inside the window where every request is decoding) because batched outputs, and with them acceptance, vary run to run in both arms.
Route (kineto, batch 1, 8 draft steps on rank 0, measured on the previous base; the route code is unchanged by the rebase): the draft-metadata kernels
_window_gather,_swa_page_indicesand_page_table_from_req_to_tokenlaunch throughhipModuleLaunchKernelbefore every draft replay on the base and throughhipGraphLaunchwith this change.Checklist
CI States
Latest PR Test (Base): ❌ Missing
run-cilabel -- add it to run CI tests.Latest PR Test (Extra): ❌ Blocked --
run-ciis required first.Latest PR Test (AMD ROCm 7.2): ➖ No AMD PR run found for this commit.