Skip to content

perf(amd): build the DSpark draft-block metadata inside the draft graph - #13

Closed
jhinpan wants to merge 1 commit into
kevin-mii:dsv41-amd-mainfrom
jhinpan:perf/amd-dspark-draft-metadata-in-graph
Closed

jhinpan wants to merge 1 commit into
kevin-mii:dsv41-amd-mainfrom
jhinpan:perf/amd-dspark-draft-metadata-in-graph

Conversation

@jhinpan

@jhinpan jhinpan commented Sep 23, 2026 •

Copy link
Copy Markdown
Collaborator

Motivation

The HIP DSV4 backend builds its decode and target-verify metadata inside the CUDA graph from three raw inputs (DSV4RawDecodeMetadata, DSV4RawVerifyMetadata), so a replay only copies those inputs. The DSpark draft block still builds its full metadata eagerly before every replay and then copies it into the captured buffers, which keeps those kernels and copies on the host path of every draft step.

Modifications

  • Add DSV4RawDraftBlockMetadata and route the draft-block graph through it. Replay refreshes req_pool_indices, seq_lens and out_cache_loc; the graph expands them with init_forward_metadata_dspark_draft_block at MAX_SEQ_LEN_FOR_CAPTURE, the bound the eager path already passed.
  • Extend the MI35x DSpark unit tests: capture records only the raw inputs, the in-graph build receives the captured tensors and the capture bound, and replay refreshes those tensors in place without an eager build.

Accuracy Tests

GSM8K (64 fixed questions, natural EOS) and 8 retrieval probes on every server: 61-62/64 GSM8K and 8/8 retrieval in both arms. On this base batch-1 outputs already differ between fresh servers of the same arm (base vs base: 15/64 identical GSM8K sequences, 62/64 same answer), and the base-vs-PR agreement stays in that range (12-15/64 identical, 61-64/64 same answer).

Speed Tests and Profiling

MI355X x4, TP4/EP4, DeepSeek-V4.1-Flash dba1be0a, this branch at e2e824dc58, the cookbook MI350X Low-Latency cell (DSpark block 5), AITER built as in this branch's Dockerfile. Both arms also carry #8's int64 FlashMLA store fix. Fresh servers per arm, real acceptance, no profiler during timing.

Workload Metric Base This PR Change (95% CI)
Batch 1, 4 real-text 4K prompts x 24 per server, ABBA decode tok/s 359.3 364.9 +1.57% [+0.72, +2.49]
wall per verify 9.16 ms 8.96 ms
Batch 8, distinct real-text 4K prompts, 1024 tokens, ABCCBA cycle time 15.15 ms 14.96 ms -1.3% [-1.7, -0.7]
Batch 32 cycle time 24.48 ms 24.36 ms -0.5% [-1.4, +0.3]
Batch 64 cycle time 31.75 ms 31.65 ms -0.2% [-0.5, +0.0]

Batched runs report cycle time (batch x acceptance length / decode tok/s inside the window where every request is decoding) because batched outputs, and with them acceptance, vary run to run in both arms.

Route (kineto, batch 1, 8 draft steps on rank 0, measured on the previous base; the route code is unchanged by the rebase): the draft-metadata kernels _window_gather, _swa_page_indices and _page_table_from_req_to_token launch through hipModuleLaunchKernel before every draft replay on the base and through hipGraphLaunch with this change.

Checklist


CI States

Latest PR Test (Base): ❌ Missing run-ci label -- add it to run CI tests.
Latest PR Test (Extra): ❌ Blocked -- run-ci is required first.
Latest PR Test (AMD ROCm 7.2): ➖ No AMD PR run found for this commit.

The draft block built its attention metadata eagerly before every replay
and copied it into the captured buffers, keeping those kernels and copies
on the host path of each draft step. Capture the three raw inputs instead,
as the decode and target-verify paths already do, and expand them inside
the graph at the capture sequence bound the eager build already used.
@jhinpan
jhinpan force-pushed the perf/amd-dspark-draft-metadata-in-graph branch from be6f5f2 to 2cdd132 Compare September 24, 2026 04:51
@jhinpan

jhinpan commented Sep 29, 2026

Copy link
Copy Markdown
Collaborator Author

Closing: on dsv41-amd-5-integration this is +2.9% at batch 1 on its own, but on top of #18 it adds nothing measurable (+0.14% [-0.03, +0.31] wall per verify at batch 1, -0.10% to +0.16% verify cycles at batch 8-64). Once the host no longer blocks on the publish event, the eager draft-metadata launches it removes are already hidden.

@jhinpan jhinpan closed this Sep 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant