Repository navigation
feat(stinkytofu): swap CDNA5 DAG fillers by s_wait_alu scoreboard - #14352
Merged
Merged
Conversation
✅ All Checks Passed — Ready for Review
📖 Need help? See the Policy FAQ for details on every check and how to fix failures. |
|
🎉 All checks passed! This PR is ready for review. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## develop #14352 +/- ##
===========================================
+ Coverage 75.88% 75.89% +0.01%
===========================================
Files 2883 2883
Lines 480622 480796 +174
Branches 72281 72335 +54
===========================================
+ Hits 364704 364867 +163
- Misses 90872 90873 +1
- Partials 25046 25056 +10
*This pull request uses carry forward flags. Click here to find out more. 🚀 New features to boost your workflow:
|
A prefetch had only a lower bound (prefetchEarliestWmma_) and was issued whenever the lower levels happened to pick it: develop's promoteDuePrefetchHoist fires it at its hazard deadline, so its position followed the cycle hazard model, and any change to that model moved it. A group with no lead (a stage under PrefetchLeadMinStageWmmas) had no bound at all. Add promoteDuePrefetch to the Prefetch level of the promote table: a prefetch at its planned window that is issuable now is promoted, so the Prefetch level places it. In its window it goes after the window's fills, right before the next WMMA, as the hoist did: promoted at the start of the window it ran ahead of the window's ds_loads, which made its address VALU wait longer and the stage barrier wait for the ds (SQTT mxf8 k128k: +111 cycles per iteration). A group with no lead is planned in its tensor_load's window, as a single-stage loop already is. Asm on the 21 gfx1250 perf yamls against develop: 14 kernels identical, 6 differ by 2-11 lines, mxf8 k128k by 58. Prefetch positions in every MAF kernel are unchanged. The two FileCheck cases for no lead now expect the group in the load's window.
untilFree() counts the WMMAs or FIFO ops that must still issue before an instruction needs no s_wait_alu; requiredGap() gives the same count for a producer-consumer pair. Both run on a copy of InsertWaitAlu's scoreboard, so the DAG can ask instead of keeping its own wait model.
…e first WMMA untilFree saw every plain-VALU va_vdst as unhideable until a WMMA latched the form, which is the whole loop top. It now takes formWmma like requiredGap; both latch it only from an XDL WMMA, as the walk does, and stop counting at the arch's largest hide instead of 64 ops. Rewrite the design doc around swap-only filler moves and a filler pacer.
pickFreeBest ranks a candidate WaitAluTracker says needs an s_wait_alu below every one that does not, so another filler takes the slot. Loop s_wait_alu 577 -> 575 on the gfx1250 perf kernels (one kernel changed, no barrier moves). Holding such VALUs outright was worse (577 -> 634) and is left for the filler pacer.
pickFreeBest now asks the scoreboard once per candidate and demotes a filler only when untilFree says a later slot hides its wait. An EXEC writer's va_vdst(0) drain and the producer of a prefetch or of the last input of real work keep their place. untilFree skips the scoreboard copy when the counter is already clear. Loop s_wait_alu 577 -> 575 on the gfx1250 perf kernels, no barrier moves; adds dag_wait_alu_swap.stir.
A VALU computing a ds address moved later lands right before its ds and turns a weak vm_vsrc(6) into a strict va_vdst(0). The previous rank did exactly that in 3 kernels: 3 fewer waits but 2 more strict ones. Exempt fillers feeding a ds, VMEM, prefetch or tensor_load; the swap now leaves the 21 gfx1250 perf kernels unchanged.
KKyang
force-pushed
the
users/kkyang/stinkytofu-waitalu-placement
branch
from
October 25, 2026 13:38
fdbcb95 to
25f9437
Compare
KKyang
changed the base branch from
develop
to
users/kkyang/stinkytofu-prefetch-pin
October 25, 2026 13:39
cycheng
approved these changes
Oct 26, 2026
Base automatically changed from
users/kkyang/stinkytofu-prefetch-pin
to
develop
October 26, 2026 09:21
KKyang
added a commit
that referenced
this pull request
Oct 26, 2026
JIRA ID : AIHPBLAS-5298 ## Summary A CDNA5 prefetch only had a lower bound (`prefetchEarliestWmma_`) and was issued whenever the lower levels happened to pick it. On develop, `promoteDuePrefetchHoist` fires it at its hazard deadline, so its position follows the cycle hazard model, and any change to that model moves it. Prefetch groups with no lead (a stage under `PrefetchLeadMinStageWmmas`) had no bound at all. ## Changes - `promoteDuePrefetch` is added to the Prefetch level of the promote table. A prefetch at its planned window that is issuable now is promoted, so the Prefetch level places it. - In its window it goes after the window's fills, right before the next WMMA, as the hoist did. Fired at the start of the window it ran ahead of the window's `ds_load`s. - A group with no lead is planned in its tensor_load's window, as a single-stage loop already is. - The two FileCheck cases for no lead (`dag_prefetch_lead_off`, `dag_prefetch_lead_short_stage_test`) now expect the group in the load's window. The s_wait_alu scoreboard work (#14352, #14356) removes the hazard model, which would otherwise move these prefetches. ## Tests - stinkytofu ctest: 1800/1800. - GPU-less asm A/B on the 21 gfx1250 perf yamls against develop: 13 kernels identical, 6 differ by 2-11 lines, 2 by 58 and 96 lines. Prefetch positions in every MAF kernel are unchanged.
KKyang
added a commit
that referenced
this pull request
Oct 26, 2026
#14356) Stacked on #14352. The CDNA5 DAG drops its own cycle model of the s_wait_alu hazards and asks InsertWaitAlu's scoreboard (WaitAluTracker) instead. Design and per-step A/B: docs/developer/waitalu-placement.md. ## Changes - Producer first: pickFreeBest ranks free fillers by a need-by window, the planned window of the ds / prefetch / tensor_load they feed less the WMMAs requiredGap() says must sit between them. - Cycle hazard model removed: hazardGates_, getHazardWait, hazardDeadline and its cumCycles pre-scan, and the two hoist rules. In their place: - promoteDuePrefetchProducer (Prefetch level) issues a prefetch's producer once its need-by window comes, so it does not lose the slot to a ds. - A prefetch or VMEM load no level placed waits while WMMAs can issue, until untilFree() says its va_vdst is hidden. - SALU sgpr -> memory address only stalls in hardware: its producer goes first, nothing is held. - The tracker runs whenever InsertWaitAlu does (dagFeatures.waitAluScoreboard = EnableESM2; --wait-alu-scoreboard in stinkytofu-opt), not only with the strict-count hold. ## Results (21 gfx1250 perf kernels, GPU-less asm A/B vs develop) - Main loop s_wait_alu 577 -> 545; strict (count 0) 61 -> 66. - Whole kernels 26724 -> 26693; strict 16939 -> 16937. - SALU -> memory-address pairs under 8 cycles: 0 of 369 before and after. - Open: medium mxf4 loop waits 19 -> 22, and the +5 strict loop waits, not yet traced. ## Tests - Removed the four DAGSchedulerPassTest cases of the cycle gaps (behaviour removed). - dag_prefetch_lead_off / _short_stage_test run with the scoreboard and expect its placement. - New dag_filler_producer_first.stir. stinkytofu ctest 1805/1805. Part of #14351 --------- Co-authored-by: YangWen Huang <kkyang@kkyang.me> Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
Contributor
TheRock Submodule Bump ActivityNewest first
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
WaitAluTracker::untilFree()/requiredGap(): ask InsertWaitAlu's scoreboard how many ops until a wait is hidden, so the DAG keeps no wait model of its own.untilFreetakes a WMMA form before the first WMMA (it used to treat the whole loop top as unhideable).pickFreeBestdemotes a filler only when a later slot drops its s_wait_alu; fillers feeding a ds, VMEM, prefetch or tensor_load keep their slot.docs/developer/waitalu-placement.md.Results (21 gfx1250 perf kernels)
Tests
WaitAluTrackerTest.cpp(unit),dag_wait_alu_swap.stir(FileCheck).Closes #14351