Skip to content

Use direct prefix tiles in resident alias attention - #97

Closed
0z5a wants to merge 92 commits into
ThinkFlowLab:mainfrom
0z5a:perf/resident-alias-prefix
Closed

0z5a wants to merge 92 commits into
ThinkFlowLab:mainfrom
0z5a:perf/resident-alias-prefix

Conversation

@0z5a

@0z5a 0z5a commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

Incremental review: 0z5a/vllm-rlt#16, based on alias CUDA Graphs #96. The upstream diff includes the unmerged parent stack; use the fork PR for this change alone.

Purpose

Reuse the known first-alias position in resident attention so complete prefix tiles use the direct reader. Snapshot one additional integer per alias descriptor into the existing routing bank and refresh stable Graph storage before replay. Bank-reader and request-publication events preserve lifetime ordering.

Test Plan

  • Affected CPU checks: 29 passed, 12 CUDA skips. All 60 tiny CUDA arms pass at B1–128/C2B, including P33/P34 tile boundaries, mapped-host/H2D routing, cancellation and request reuse.
  • Cross-stream allocation/reset and bank-retirement probes pass; 24 Graph arms report 184 captures, 1,660 replays and 512 fallbacks.
  • Official Ouro-1.4B fixed-schedule E2E: seven balanced pairs at each B16/C32 and B32/C64 point on each RTX 5090, P128/D128, BF16, delayed gate 0.5/min2/max4. Each measured arm has a complete warmup. Both cards and heavy IO were reserved for the full run.
  • Independent audit confirms all 56 measured arms / 2,688 requests / 344,064 output tokens exactly match the frozen external reference. Warmups also match; paired work and Graph coverage are equal, with no new measured captures. All children/controller exited naturally 0.

Test Result

The complete prefix-aware alias stack regresses against native resident Graph execution at all four points.

GPU B / C Native Graph tokens/s Alias + prefix Graph tokens/s Paired speedup [95% CI] Throughput change
5090 #0 16 / 32 387.05 380.45 0.9825× [0.9790, 0.9837] -1.75%
5090 #0 32 / 64 452.56 432.99 0.9568× [0.9440, 0.9641] -4.32%
5090 #1 16 / 32 385.93 379.58 0.9797× [0.9787, 0.9892] -2.03%
5090 #1 32 / 64 453.33 435.25 0.9575× [0.9363, 0.9625] -4.25%

Tokens/s are arm medians; speedup is the median paired elapsed ratio. The 95% intervals bootstrap seven pairs within one persistent session per card and do not represent independent startup sessions. B32 retains the frozen 16-entry Graph cache and equal fallbacks.

This measures the entire alias stack against native Graph execution, not the incremental prefix change against its alias parent. It does not establish HTTP/SLO performance or natural-schedule quality equivalence. The separate parent Graph quality result remains 63/100 native versus 62/100 alias, with 15 token-different and 43 exit-different requests.

Full result, scope and immutable evidence · Implementation and CUDA checks.

0z5a added 30 commits October 2, 2026 22:57
Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
…ssion

Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
…ive footprint

Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
…ivals

Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
0z5a added 16 commits October 5, 2026 06:21
Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
0z5a added 12 commits October 5, 2026 10:05
Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
…zation

Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
…ting

Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
…zation

Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
…ting

Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
@0z5a 0z5a closed this Oct 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant