feat(stinkytofu): issue a barrier once its dependency threshold is met

shared/stinkytofu/docs/user/scheduler-tuning-parameters.md

@@ -99,8 +99,7 @@ the region's WMMAs. This replaces the old cap / throttle / slot-first / hide-bud
 |---|---|---|---|
 | `DsSpread` | true | `--ds-spread` / `--no-ds-spread` | Pace the region's ds_loads by WMMA batch (`DsBatchWmmas` WMMAs). Each batch takes the region's even share `ceil(ds_loads / usable batches)`, where the last `ceil(LDS return time / WMMA window) + kDrainSlackWmmas` WMMAs of the region are kept free for the last load to return. A sparse region therefore spreads its loads, a dense one gets a share above what the LDS rate fits (the credits throttle it), and a batch that began behind the even line takes `DsCatchUpPace` more. Nothing is held once the region's last WMMA has issued; a held ds_load never holds a WMMA. false = a ds_load issues as soon as it is ready and a credit is free (the credit queue is the only pacing). |
 | `DsBatchWmmas` | 4 | `--ds-batch-wmmas=N` | WMMAs per ds_load pacing batch (independent of the queue refill batch `WmmaBatchSize`). |
-| `DsReadCyclesPerLoad` | 6 (`stinkytofu-opt`: 0 = off) | `--ds-read-cycles-per-load=N` | LDS return rate: cycles per ds_load. Capped at the region's even spread (`WMMAs x WMMA latency / ds_loads`). Decides how long the stage's ds_loads take to issue: `ceil(ds_loads x N / WMMA window)` WMMA windows from the region start. The barrier is placed after the last of them; the gap from the last ds_load to the barrier is the drain, set by the barrier cover. Replaces the old throttle latency T (one ds_load per `T / DsReadQueueDepth` cycles). Used only with `DsSpread`. |
-| `WmmaBarrierCoverCycles` | -1 (automatic) | `--wmma-barrier-cover-cycles=N` | Queued WMMA cover a ready barrier waits for before it issues. -1 = a tail of 4 WMMAs (needs `DsSpread`); 0 = off; always capped at a full queue. Needs the queue model on. |
+| `DsReadCyclesPerLoad` | 6 (`stinkytofu-opt`: 0 = off) | `--ds-read-cycles-per-load=N` | LDS return rate: cycles per ds_load. Capped at the region's even spread (`WMMAs x WMMA latency / ds_loads`). Decides how long the stage's ds_loads take to issue: `ceil(ds_loads x N / WMMA window)` WMMA windows from the region start. The barrier is placed after the last of them; the gap from the last ds_load to the barrier is the drain already inside the barrier threshold. Replaces the old throttle latency T (one ds_load per `T / DsReadQueueDepth` cycles). Used only with `DsSpread`. |
 | `DsCatchUpPace` | 2 | `--ds-catch-up-pace=N` | ds_loads a batch may take above the even share when it began behind the even line. 0 = no catching up. |
 | `DsReadQueueDepth` | 0 → HW 16 | `--ds-read-queue-depth=N` | In-flight ds_load credits modeled for the LDS return queue. A credit lives `HWModel::lds.readCreditLifetime` cycles. |
 | `DsReadDrainLatency` | 0 → dynamic | `--ds-read-drain-latency=N` | Barrier timing only: cycles a barrier waits for its ds_loads to return. 0 derives it from the matching loads. |
@@ -122,9 +121,10 @@ loses its tuning unless it is translated:
 | `DsReadPerCap` A, `DsIssueCapSpanCycles` X, `DsIssueCapMode` | burst cap: at most A ds_loads per span X (default one WMMA window) | no equivalent: the per-batch share (`DsBatchWmmas`, `DsCatchUpPace`) replaces it |
 | `DsReadPerWmma`, `DsReadThrottleTransitionFactor`, `DsReadThrottleTransitionEntries`, `DsSlotFirst` | per-WMMA ds count, loads past the queue depth, ds before fillers | removed |
 | `BarrierHalfSlack` | WMMA windows kept inside a barrier signal/wait pair (0 = together) | removed: the separation slack is the constant 1 |
+| `WmmaBarrierCoverCycles` | extra WMMAs a ready barrier waited for after its threshold (`-1` = 4) | removed: the threshold already includes the last WMMA to ds dependency and its drain |
 | `DsReadDrainLatency` | barrier timing only | unchanged |
 
-The matching `stinkytofu-opt` flags and `--enable-wmma-hide-budget-prescan` are gone too.
+The matching `stinkytofu-opt` flags, `--enable-wmma-hide-budget-prescan`, and `--wmma-barrier-cover-cycles` are gone too. The last is still accepted and ignored.
 TensileLite's `StinkyTofuDsIssueCapMode` and `StinkyTofuDsIssueCapSpanCycles` global parameters are
 gone.
 
@@ -159,7 +159,7 @@ Per WMMA window, at most `globalReadPerWmma` (arch default 1) tensor_loads issue
 | `WarGateWmmas` | -1 → derived `L / I` | `--war-gate-wmmas=N` | WMMAs a ds_load waits before overwriting a vgpr a WMMA read. Active only with `EnableESM2` + `EnableESM2TrackValuVsrc`. |
 | `ClusterBarrierRule3SignalLeadCycles` | -1 → heuristic | – | Cluster-barrier Rule 3 signal lead: 200 if the main loop's WMMA latency sum > 500, else 100. See [Insert cluster barrier pass](../developer/cluster-barrier.md). |
 | `ReduceClusterBarrierPerLoop` | -1 → heuristic | – | `1`: first Rule 3 handshake of each enclosing loop, then another after each 2000 estimated cycles. Nearby HalfPLR / ExpandPointerSwap copies still share one handshake. `0`: every loop trigger handshakes. Unset turns on when the main loop's WMMA latency sum > 2500 and that sum / ds_load count >= 5. See [Insert cluster barrier pass](../developer/cluster-barrier.md). |
-| *CLI only* `mergeBarrierThreshold` | 0 → 11 | `--merge-barrier-threshold=N` | Max cycle distance for `StinkyMergeBarrierPass` to merge adjacent barrier groups. |
+| *CLI only* `mergeBarrierThreshold` | unused | `--merge-barrier-threshold=N` | Parsed but ignored. `StinkyMergeBarrierPass` merges adjacent groups only when no matrix instruction is issued between them. |
 
 ## Arch defaults and hardware facts
 

shared/stinkytofu/include/stinkytofu/bindings/python/Module.hpp

@@ -145,7 +145,6 @@
     X(WarGateWmmas, int, -1)              /* WMMA src -> ds_load overwrite gap; -1 = derived */    \
     X(PrefetchLeadWmmas, int, 25)         /* 0 = prefetch issues when ready */                     \
     X(PrefetchLeadMinStageWmmas, int, 64) /* shorter stages run with no lead */                    \
-    X(WmmaBarrierCoverCycles, int, -1)    /* cover kept for a barrier; < 0 = automatic */          \
     X(DsSpread, bool, true)               /* even ds_load spread over the WMMAs */                 \
     X(DsBatchWmmas, int, 4)               /* WMMAs per ds_load pacing batch */                     \
     X(DsCatchUpPace, int, 2)              /* extra ds_loads per batch when behind the even line */ \

shared/stinkytofu/include/stinkytofu/core/Types.hpp

@@ -121,12 +121,6 @@ struct PassFeatureConfig {
         /// (see DsSpreadPacer), and a load issues only while the ds queue has a free credit. Off =
         /// ds_loads issue as soon as they are ready and a credit is free.
         bool dsSpread = true;
-        /// Queued WMMA cover (cycles) to have in the pipe when a barrier issues, so the WMMAs hide
-        /// its stall. The WMMAs before the barrier's threshold are reserved for it: the queue is
-        /// refilled to this cover ahead of other work, and ds_loads the barrier waits on are
-        /// released that many WMMAs earlier. Applies with dsSpread (or when set > 0) and the
-        /// queue model; < 0 = automatic (72 cycles, at most a full queue), 0 = off.
-        int wmmaBarrierCoverCycles = -1;
         /// Cycles the LDS needs to return one ds_load (its return rate, the old throttle latency /
         /// queue depth). The windows the ds_loads in front of a barrier need are at least
         /// loads * cycles / WMMA window. 0 = off (the even-spread density only).
@@ -142,11 +136,11 @@ struct PassFeatureConfig {
         /// WMMA windows by the region's matrix latency. 0 disables the extra
         /// gap. Mirrors ModuleOptions::TensorLoadDsLoadGapCycles.
         int tensorLoadDsLoadGapCycles = 64;
-        /// Max cycle-distance between two adjacent barrier groups for
-        /// StinkyMergeBarrierPass to merge them into a single multi-token
-        /// barrier group. 0 = use the CDNA5 default (kCdna5MergeBarrierThreshold).
-        /// Internal tuning knob only — deliberately not surfaced as a module
-        /// option, so TensileLite cannot set it.
+        /// Unused by StinkyMergeBarrierPass. Adjacent groups merge only when no
+        /// matrix instruction is issued between them. Kept so existing callers
+        /// and --merge-barrier-threshold= still parse. Internal knob only —
+        /// deliberately not surfaced as a module option, so TensileLite cannot
+        /// set it.
         int mergeBarrierThreshold = 0;
         /// Mirrors ModuleOptions::ClusterBarrier: InsertClusterBarrierPass will run
         /// after the scheduler and plant SCC-clobbering handshakes around workgroup

shared/stinkytofu/include/stinkytofu/transforms/asm/StinkyMergeBarrierPass.hpp

@@ -36,11 +36,11 @@ class Pass;
 /// exactly an adjacent s_barrier_signal / s_barrier_wait pair with the same
 /// non-empty memory-token set. Two consecutive legal groups may be fused only
 /// when Layer2BarrierOverlapAnalysis contains the final-order-validated,
-/// directional earlier-to-later pair, their token sets differ, and their
-/// modeled cycle-distance is below the merge threshold
-/// (PassFeatureConfig::DagFeatures::mergeBarrierThreshold, or the CDNA5 default
-/// when unset): the earlier pair absorbs the later tokens and the redundant
-/// second signal/wait pair is removed.
+/// directional earlier-to-later pair, their token sets differ, and no matrix
+/// instruction is issued between them. One issued WMMA occupies the WMMA queue
+/// long enough to hide the later barrier, so cycle distance is not a gate. The
+/// earlier pair absorbs the later tokens and the redundant second signal/wait
+/// pair is removed.
 STINKYTOFU_EXPORT std::unique_ptr<Pass> createStinkyMergeBarrierPass();
 
 }  // namespace stinkytofu

shared/stinkytofu/src/conversion/rocisa/ToStinkyTofuUtils.cpp

@@ -1760,7 +1760,11 @@ void init_stinkytofu(nb::module_ m) {  // NOLINT(misc-use-internal-linkage)
 #undef DEBUG_SET_MODULE_OPTION
                 for (auto kv : options) {
                     const std::string key = nb::cast<std::string>(kv.first);
-                    if (isRemovedDsPacingOption(key))
+                    if (key == "WmmaBarrierCoverCycles")
+                        std::cerr << "[StinkyTofu] WARNING: module option '" << key
+                                  << "' was removed; a barrier issues when its threshold is met, "
+                                  << "and the option is ignored\n";
+                    else if (isRemovedDsPacingOption(key))
                         std::cerr
                             << "[StinkyTofu] WARNING: module option '" << key
                             << "' was removed with the legacy ds_load pacing and is ignored\n";

shared/stinkytofu/src/pipeline/backend/Gfx1250Backend.cpp

@@ -198,8 +198,6 @@ bool buildGfx1250Pipeline(ModulePassManager& mpm, StinkyAsmModule& module, const
                 passFeatureConfig.dagFeatures.lockDsReadOrder = moduleOptions.LockDsReadOrder;
                 passFeatureConfig.dagFeatures.evenSpreadFillers = moduleOptions.EvenSpreadFillers;
                 passFeatureConfig.dagFeatures.dsSpread = moduleOptions.DsSpread;
-                passFeatureConfig.dagFeatures.wmmaBarrierCoverCycles =
-                    moduleOptions.WmmaBarrierCoverCycles;
                 passFeatureConfig.dagFeatures.dsCatchUpPace = moduleOptions.DsCatchUpPace;
                 passFeatureConfig.dagFeatures.dsBatchWmmas = moduleOptions.DsBatchWmmas;
                 passFeatureConfig.dagFeatures.dsReadCyclesPerLoad =

shared/stinkytofu/src/transforms/asm/StinkyMergeBarrierPass.cpp

@@ -42,9 +42,9 @@
 // ======================
 // Runs immediately after StinkyDAGSchedulerPass. Within loop bodies it looks
 // for two (or more) barrier groups that Layer 2 identified as a directional
-// overlapping pair and the scheduler placed only a few cycles apart. It fuses
-// them into a single group carrying the union of their memory tokens, dropping
-// the redundant second signal/wait pair.
+// overlapping pair and the scheduler placed with no matrix instruction between
+// them. It fuses them into a single group carrying the union of their memory
+// tokens, dropping the redundant second signal/wait pair.
 //
 // A legal "barrier group" is exactly one adjacent s_barrier_signal /
 // s_barrier_wait pair that shares the same non-empty LDS token set, e.g.:
@@ -53,23 +53,19 @@
 // Anything else (lone signal/wait, signal/.../wait with filler between, mismatched
 // tokens) is ignored. Two consecutive legal groups G1 (tokens T1) and G2 (tokens
 // T2) may merge only when Layer2BarrierOverlapAnalysis contains G1 -> G2,
-// T1 ≠ T2, and the modeled cycle-distance is below the configured threshold.
+// T1 ≠ T2, and no matrix instruction is issued between them. Issuing a WMMA
+// occupies the WMMA queue long enough to hide the later barrier, so cycle
+// distance is not a merge gate.
 //
-// The merge never moves instructions: it only unions G2's tokens into G1's
-// barriers and drops G2's redundant signal/wait pair. That is correct precisely
-// when nothing between the two groups is ordered against the merged barrier — no
-// other barrier and no producer/consumer of a merged token (T1 ∪ T2) sits in
-// between. When such an instruction exists the merge is skipped, because keeping
-// the barrier in place would move it to the wrong side of that dependency.
+// The merge never moves ordinary instructions: it unions G2's tokens into G1
+// and drops G2. G1 stays where it is, so a producer or consumer of T1 may sit
+// between the groups. G2 is the barrier that moves earlier. A producer or
+// consumer of T2 between the groups would end up on the wrong side of it, and
+// another barrier between them would too, so those pairs are left unmerged.
 
 namespace {
 using namespace stinkytofu;
 
-// CDNA5 (Gfx1250) default merge distance, in cycles. Used when
-// dagFeatures.mergeBarrierThreshold holds the sentinel value (0). Explicit
-// non-sentinel config wins. Mirrors the kCdna5* tunables in dag/CDNA5.hpp.
-constexpr int kCdna5MergeBarrierThreshold = 11;
-
 // An adjacent s_barrier_signal / s_barrier_wait pair with one shared token set.
 struct BarrierGroup {
     std::vector<StinkyInstruction*> barriers;  // [signal, wait]
@@ -161,21 +157,6 @@ std::vector<BarrierGroup> collectBarrierGroups(BasicBlock& bb) {
     return groups;
 }
 
-// Sum of issueCycles over the instructions strictly between \p afterIt and
-// \p beforeIt (both exclusive). Non-StinkyInstruction IR contributes nothing.
-int cycleDistance(IRList::iterator afterIt, IRList::iterator beforeIt) {
-    int cycles = 0;
-    for (auto it = std::next(afterIt); it != beforeIt; ++it) {
-        auto* inst = dyn_cast<StinkyInstruction>(it.getNodePtr());
-        if (inst == nullptr) continue;
-        if (isMatrixInstruction(*inst))
-            cycles += inst->latencyCycles;
-        else
-            cycles += inst->issueCycles;
-    }
-    return cycles;
-}
-
 // Add \p tokenId as an LDS pseudo register to \p regs if not already present.
 void addUniqueLds(std::vector<StinkyRegister>& regs, uint32_t tokenId, bool /*isDest*/) {
     for (const StinkyRegister& r : regs)
@@ -227,33 +208,34 @@ void setMergedBarrierComment(StinkyInstruction* barrier,
 }
 
 // Attempt to merge the two consecutive groups g1 (earlier) and g2 (later) inside
-// \p bb. Returns true on success (IR mutated). \p threshold is in cycles.
-bool tryMergePair(BasicBlock& bb, const BarrierGroup& g1, const BarrierGroup& g2, int threshold,
+// \p bb. Returns true on success (IR mutated).
+bool tryMergePair(BasicBlock& bb, const BarrierGroup& g1, const BarrierGroup& g2,
                   Layer2OverlapResult& overlaps) {
     if (!isLayer2OverlapPair(g1, g2, overlaps)) return false;
 
     // Only distinct token sets are merge candidates. Same-token consecutive
     // legal groups are successive syncs of one token and must both remain.
     if (g1.tokens == g2.tokens) return false;
 
-    const int dist = cycleDistance(g1.lastIt, g2.firstIt);
-    if (dist >= threshold) return false;
-
-    // Merged barrier will guard the union of both token sets.
-    std::unordered_set<uint32_t> mergedTokens = g1.tokens;
-    mergedTokens.insert(g2.tokens.begin(), g2.tokens.end());
-
-    // Merge without moving instructions. This is only correct when nothing
-    // strictly between the two groups is ordered against the merged barrier:
-    // no other barrier, and no producer/consumer of a merged token. Otherwise
-    // folding G2 back onto G1's position would cross that dependency — bail.
+    // G1 stays put and absorbs G2's tokens; G2 is the barrier that moves
+    // earlier. An instruction between the groups may touch G1's tokens. It
+    // must not touch G2's tokens, and another barrier must not sit between
+    // them: either one would be crossed by folding G2 back onto G1.
+    // A matrix instruction between them also blocks the merge. Issuing one
+    // WMMA, including scaled WMMA, fills the WMMA queue for long enough to
+    // hide G2, so the groups stay separate no matter how few cycles the
+    // modeled latency reports.
     for (auto it = std::next(g1.lastIt); it != g2.firstIt; ++it) {
         auto* inst = dyn_cast<StinkyInstruction>(it.getNodePtr());
         if (inst == nullptr) continue;
         if (isBarrier(*inst)) return false;
-        if (touchesTokens(*inst, mergedTokens)) return false;
+        if (isMatrixInstruction(*inst)) return false;
+        if (touchesTokens(*inst, g2.tokens)) return false;
     }
 
+    std::unordered_set<uint32_t> mergedTokens = g1.tokens;
+    mergedTokens.insert(g2.tokens.begin(), g2.tokens.end());
+
     // Safe: fold g2's tokens into every barrier of g1, then drop g2's redundant
     // signal/wait pair. No instruction is moved.
     for (StinkyInstruction* barrier : g1.barriers) {
@@ -268,13 +250,13 @@ bool tryMergePair(BasicBlock& bb, const BarrierGroup& g1, const BarrierGroup& g2
 
 // Repeatedly merge mergeable adjacent barrier-group pairs in \p bb until a fixed
 // point. Chained close groups collapse into one multi-token group.
-void mergeBarriersInBlock(BasicBlock& bb, int threshold, Layer2OverlapResult& overlaps) {
+void mergeBarriersInBlock(BasicBlock& bb, Layer2OverlapResult& overlaps) {
     bool changed = true;
     while (changed) {
         changed = false;
         std::vector<BarrierGroup> groups = collectBarrierGroups(bb);
         for (size_t i = 0; i + 1 < groups.size(); ++i) {
-            if (tryMergePair(bb, groups[i], groups[i + 1], threshold, overlaps)) {
+            if (tryMergePair(bb, groups[i], groups[i + 1], overlaps)) {
                 changed = true;
                 break;  // group layout changed; rebuild before continuing
             }
@@ -299,13 +281,6 @@ class StinkyMergeBarrierPass : public StinkyInstPass {
         if (passCtx.getGemmTileConfig().arch[0] != 12 || passCtx.getGemmTileConfig().arch[1] != 5)
             return preserveCFGAnalyses();
 
-        const int cfg = passCtx.getPassFeatureConfig().dagFeatures.mergeBarrierThreshold;
-        const int threshold = cfg > 0 ? cfg : kCdna5MergeBarrierThreshold;
-
-        if (threshold == -1) {
-            return preserveCFGAnalyses();
-        }
-
         // Copy because chained merges redirect relations away from removed barriers.
         Layer2OverlapResult layer2Overlaps = AM.getResult<Layer2BarrierOverlapAnalysis>(func);
 
@@ -321,7 +296,7 @@ class StinkyMergeBarrierPass : public StinkyInstPass {
         for (BasicBlock& bb : func) {
             if (!loopBodyBBs.count(&bb)) continue;
             if (!passCtx.shouldProcessBasicBlock(bb)) continue;
-            mergeBarriersInBlock(bb, threshold, layer2Overlaps);
+            mergeBarriersInBlock(bb, layer2Overlaps);
         }
         return preserveCFGAnalyses();
     }