Skip to content

fix(stinkytofu): count a ds credit wait as elapsed time; merge CDNA5 DAG Phases C and D #14805

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
KKyang merged 2 commits into from
Nov 2, 2026
Merged

fix(stinkytofu): count a ds credit wait as elapsed time; merge CDNA5 DAG Phases C and D #14805

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
c76cefa
refactor(stinkytofu): merge CDNA5 DAG Phases C and D into one NonWmma…
KKyang Oct 30, 2026
769263f
fix(stinkytofu): count a ds credit wait as elapsed time in the CDNA5 DAG
KKyang Oct 30, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion shared/stinkytofu/src/transforms/asm/dag/AGENTS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ falls through to the next.
| --- | --- |
| `decidePromote()` | Pure decision. The first rule in `kRules` that fires sets `promotedPhase_` and `promotedNode_`. Mutates no queue. |
| Phase B | Issues the best WMMA if no veto holds it. Runs only when nothing is promoted. |
| Phase C / D | `NonWmmaFill`: issues the promoted fill node, else the smallest pickable non-WMMA (C inside the WMMA window, D outside). |
| Phase C+D | `NonWmmaFill`: issues the promoted fill node, else the smallest pickable non-WMMA, inside or outside the WMMA window. With nothing pickable inside a window, time advances to the oldest queued WMMA's end and the pick runs once more. |
| Phase E | Forced WMMA: the most-ready WMMA, before barriers. |
| Barrier | The only place a barrier issues: the promoted one, or the drain case. |
| Phase G | Fallback: the least-blocked ready node, so the schedule always makes progress. |
Expand Down
57 changes: 10 additions & 47 deletions shared/stinkytofu/src/transforms/asm/dag/CDNA5.hpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -647,9 +647,6 @@ class CDNA5ReadyQueue : public ReadyQueue {
int dsTotalThisRegion_ = 0;
int dsIssuedThisRegion_ = 0;
int wmmaTotalThisRegion_ = 0;
// Credit-wait cycles charged to DS placement in the current WMMA.
// Kept separate from coIssueCyclePos_, the real hardware/hazard timeline.
int dsSchedulingBudgetUsed_ = 0;

// (A) RAW data-ready gate. Per reg index: remaining modeled latency until a
// producer's result is safe to consume (e.g. ds_load LDS->VGPR, 56 cyc). Any
Expand Down Expand Up @@ -816,7 +813,6 @@ class CDNA5ReadyQueue : public ReadyQueue {
// ValuIssue - the same for a VALU, which can only issue in the window's co-issue slots.
enum class TimeKind { Elapsed, Issue, ValuIssue };
void advanceTime(int cycles, TimeKind kind);
void chargeDsCreditWait(int wait);
int computeValuAdvanceCycles(int issueCycles) const;
void updateWMMAStatus(DAGNode* node);
void stampDataReady(const StinkyInstruction& inst, int extraDelay = 0);
Expand Down Expand Up @@ -1150,12 +1146,6 @@ int CDNA5ReadyQueue::freeCoIssueSpace() const {
return activeWmmaLatency_ - coIssueCyclePos_;
}

// A ds_load that waits for a free credit charges the wait to the ds scheduling budget of the
// current WMMA window. It is not elapsed time: the credit queue does not age.
void CDNA5ReadyQueue::chargeDsCreditWait(int wait) {
dsSchedulingBudgetUsed_ += wait;
}

// Before the region's first WMMA there is no co-issue window, but the most-ready
// WMMA still stalls on its own sources. Work that finishes issuing within that
// stall does not delay the WMMA. 0 once a WMMA has issued or when it is ready.
Expand Down Expand Up @@ -1583,7 +1573,6 @@ DAGNode* CDNA5ReadyQueue::pickOneFromWMMA(DAGNode* pick) {
}
nonWmmaFillsSinceActiveWmma_ = 0; // new window: restart WMMA->WMMA fill count
fillsThisWindow_ = 0;
dsSchedulingBudgetUsed_ = 0;
// Advance by WMMA issue cycles after opening a new timeline window.
// This keeps coIssueCyclePos_ aligned with elapsed cycles right after WMMA
// issue.
Expand Down Expand Up @@ -1627,7 +1616,7 @@ bool CDNA5ReadyQueue::findSmallestPickableNonWmma(DAGNode* pickedDS, DAGNode** o
bool dsHeldBySpread = false;

// Ordering, lowest key first: (1) genuinely free work; (2) a ds_load waiting
// for a credit that fits the active WMMA's scheduling budget; (3) work that
// for a credit that fits what is left of the active WMMA's window; (3) work that
// still needs a real RAW/hazard stall. Within a tier, global_read beats
// other non-WMMA kinds, then smallest id wins.
// Producer-side hazard hoisting is handled separately by
Expand Down Expand Up @@ -1658,7 +1647,7 @@ bool CDNA5ReadyQueue::findSmallestPickableNonWmma(DAGNode* pickedDS, DAGNode** o
int dsCreditWait = 0;
if (dsBaseOk) {
dsCreditWait = dsReadInflight_.full() ? std::max(1, dsReadInflight_.minResidual()) : 0;
const int schedulingPos = coIssueCyclePos_ + dsSchedulingBudgetUsed_;
const int schedulingPos = coIssueCyclePos_;
int schedulingSpace = activeWmmaLatency_ - schedulingPos;
for (int pos = schedulingPos; pos < activeWmmaLatency_; ++pos) {
if (isBlockedCycle(pos)) {
Expand Down Expand Up @@ -2605,54 +2594,29 @@ DAGNode* CDNA5ReadyQueue::pickOne() {
}
}

// Phase C+D — NonWmmaFill: fill with non-WMMA work (inside then outside the
// window).
// Phase C+D — NonWmmaFill: fill with non-WMMA work, inside or outside a WMMA window.
if (isPromote(PromotePhase::NonWmmaFill)) {
if (promotedPhase_ == PromotePhase::NonWmmaFill && promotedNode_)
return rememberPick(popNonWmma(promotedNode_, promotedKind_));

// Phase C — inside WMMA latency window.
if (coIssueCyclePos_ < activeWmmaLatency_) {
// Inside a window with nothing pickable, time moves on and the pick is tried once more.
for (bool inWindow = coIssueCyclePos_ < activeWmmaLatency_;; inWindow = false) {
DAGNode* smallestPickable = nullptr;
int pickKind = -1;
int pickWait = 0;
if (findSmallestPickableNonWmma(pickedDS, &smallestPickable, &pickKind, &pickWait)) {
PASS_DEBUG(std::cerr << "[CDNA5 pickOne] Phase C picked non-WMMA dagId="
PASS_DEBUG(std::cerr << "[CDNA5 pickOne] NonWmmaFill picked dagId="
<< smallestPickable->id << " kind=" << pickKind
<< " wait=" << pickWait << "\n");
// A ds credit wait consumes only its independent scheduling
// budget. RAW/hazard waits remain genuine elapsed stalls.
if (pickWait > 0) {
if (pickKind == kLocalRead) {
chargeDsCreditWait(pickWait);
} else {
advanceTime(pickWait, TimeKind::Elapsed);
}
}
<< " wait=" << pickWait << " inWindow=" << inWindow << "\n");
// Every wait (ds credit, RAW, hazard) is elapsed time.
if (pickWait > 0) advanceTime(pickWait, TimeKind::Elapsed);
return rememberPick(popNonWmma(smallestPickable, pickKind));
}

if (!inWindow) break;
// Only to the oldest queued WMMA's end: the queue is not drained (depth 1: window end).
advanceTime(std::min(activeWmmaLatency_ - coIssueCyclePos_, soonestQueueEnd()),
TimeKind::Elapsed);
}

// Phase D — outside WMMA latency.
DAGNode* smallestPickable = nullptr;
int pickKind = -1;
int pickWait = 0;
if (findSmallestPickableNonWmma(pickedDS, &smallestPickable, &pickKind, &pickWait)) {
// Same split as Phase C: a ds credit wait is pacing-only; RAW/hazard
// waits are genuine elapsed stalls.
if (pickWait > 0) {
if (pickKind == kLocalRead) {
chargeDsCreditWait(pickWait);
} else {
advanceTime(pickWait, TimeKind::Elapsed);
}
}
return rememberPick(popNonWmma(smallestPickable, pickKind));
}
}

// Phase G — final safety net: force-pick the least-blocked ready node to guarantee progress.
Expand Down Expand Up @@ -2815,7 +2779,6 @@ void CDNA5ReadyQueue::onInit(IRList::iterator regionStart, IRList::iterator regi
openBarrierIds_.clear();
nonWmmaFillsSinceActiveWmma_ = 0;
fillsThisWindow_ = 0;
dsSchedulingBudgetUsed_ = 0;
globalReadInflight_ = InFlightQueue(globalReadQueueDepth());
dsReadInflight_ = InFlightQueue(dsReadQueueDepth());
currentBB_ = (regionStart != regionEnd) ? regionStart->getParent() : nullptr;
Expand Down
8 changes: 6 additions & 2 deletions shared/stinkytofu/tests/unit/asm/DAGSchedulerPassTest.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2354,7 +2354,9 @@ TEST_F(DAGSchedulerPassTest, WmmaRefill_FixedBatchIssuesBatchSizeWmmasPerRefill)
pfc.dagFeatures.wmmaBatchSize = 4;
ctx.setPassFeatureConfig(pfc);
pass->run(*func, ctx, am);
EXPECT_EQ(wdShape(mnemonicSequence(*bb)), "WddWWWWWWWWWdWdWdddd");
// A credit wait is elapsed time and ages the credit queue, so the second load after it no
// longer waits and needs no refill WMMA in front of it.
EXPECT_EQ(wdShape(mnemonicSequence(*bb)), "WddWWWWWWWWWddWWdddd");
}

TEST_F(DAGSchedulerPassTest, WmmaRefill_DynamicFillsToTarget) {
Expand All @@ -2372,7 +2374,9 @@ TEST_F(DAGSchedulerPassTest, WmmaRefill_DynamicFillsToTarget) {
pfc.dagFeatures.wmmaRefillToCycles = 32;
ctx.setPassFeatureConfig(pfc);
pass->run(*func, ctx, am);
EXPECT_EQ(wdShape(mnemonicSequence(*bb)), "WddWWWWWWWWWdWdWdddd");
// A credit wait is elapsed time and ages the credit queue, so the second load after it no
// longer waits and needs no refill WMMA in front of it.
EXPECT_EQ(wdShape(mnemonicSequence(*bb)), "WddWWWWWWWWWddWWdddd");
}

// Even ds_load spread: the ds_loads follow the WMMAs instead of bursting at the start.
Expand Down
Loading