[AMDGPU] asyncmark support for ASYNC_CNT by ssahasra · Pull Request #185813 · llvm/llvm-project

ssahasra · 2026-03-11T06:18:01Z

The ASYNC_CNT is used to track the progress of asynchronous copies between global and LDS memories. By including it in asyncmark, the compiler can now assist the programmer in generating waits for ASYNC_CNT.

Assisted-By: Claude Sonnet 4.5

This is part of a stack:

Fixes: LCOMPILER-332

Async operations transfer data between global memory and LDS. Their progress is tracked by the ASYNC_CNT counter on GFX1250 and later architectures. This change introduces the representation of that counter in SIInsertWaitCnts. For now, the programmer must manually insert s_wait_asyncnt instructions. Later changes will add compiler assistance for generating the waits by including this counter in the asyncmark instructions. Assisted-by: Claude Sonnet 4.5

The ASYNC_CNT is used to track the progress of asynchronous copies between global and LDS memories. By including it in asyncmark, the compiler can now assist the programmer in generating waits for ASYNC_CNT. Assisted-By: Claude Sonnet 4.5

llvmbot · 2026-03-11T06:18:39Z

@llvm/pr-subscribers-backend-amdgpu

Author: Sameer Sahasrabuddhe (ssahasra)

Changes

The ASYNC_CNT is used to track the progress of asynchronous copies between global and LDS memories. By including it in asyncmark, the compiler can now assist the programmer in generating waits for ASYNC_CNT.

Assisted-By: Claude Sonnet 4.5

Patch is 22.99 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/185813.diff

8 Files Affected:

(modified) llvm/lib/Target/AMDGPU/AMDGPU.td (+3)
(modified) llvm/lib/Target/AMDGPU/AMDGPUInstructionSelector.cpp (+1-2)
(modified) llvm/lib/Target/AMDGPU/GCNSubtarget.h (+2)
(modified) llvm/lib/Target/AMDGPU/SIInsertWaitcnts.cpp (+14-7)
(modified) llvm/lib/Target/AMDGPU/SOPInstructions.td (+1-1)
(modified) llvm/lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.h (+1-1)
(removed) llvm/test/CodeGen/AMDGPU/asyncmark-err.ll (-19)
(added) llvm/test/CodeGen/AMDGPU/asyncmark-gfx12plus.ll (+366)

diff --git a/llvm/lib/Target/AMDGPU/AMDGPU.td b/llvm/lib/Target/AMDGPU/AMDGPU.td
index 4259bf4c1b0bf..42a9c91a0fc3f 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPU.td
+++ b/llvm/lib/Target/AMDGPU/AMDGPU.td
@@ -1298,6 +1298,9 @@ defm VMemToLDSLoad : AMDGPUSubtargetFeature<"vmem-to-lds-load-insts",
   "w/lds bit set or global_load_lds. This does not include scratch_load_lds."
 >;
 
+// Manual predicate for hasAsyncMark() which combines HasVMemToLDSLoad and GFX1250Plus
+def HasAsyncMark : Predicate<"Subtarget->hasAsyncMark()">;
+
 defm LdsBarrierArriveAtomic : AMDGPUSubtargetFeature<"lds-barrier-arrive-atomic",
   "Has LDS barrier-arrive atomic instructions"
 >;
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUInstructionSelector.cpp b/llvm/lib/Target/AMDGPU/AMDGPUInstructionSelector.cpp
index 81e224355411b..35f5e89f95986 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUInstructionSelector.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPUInstructionSelector.cpp
@@ -2398,8 +2398,7 @@ bool AMDGPUInstructionSelector::selectG_INTRINSIC_W_SIDE_EFFECTS(
     return selectTensorLoadStore(I, IntrinsicID);
   case Intrinsic::amdgcn_asyncmark:
   case Intrinsic::amdgcn_wait_asyncmark:
-    // FIXME: Not supported on GFX12 yet. Will need a new feature when we do.
-    if (!Subtarget->hasVMemToLDSLoad())
+    if (!Subtarget->hasAsyncMark())
       return false;
     break;
   case Intrinsic::amdgcn_exp_compr:
diff --git a/llvm/lib/Target/AMDGPU/GCNSubtarget.h b/llvm/lib/Target/AMDGPU/GCNSubtarget.h
index b9c7bad4cef0d..b35c661b1b608 100644
--- a/llvm/lib/Target/AMDGPU/GCNSubtarget.h
+++ b/llvm/lib/Target/AMDGPU/GCNSubtarget.h
@@ -220,6 +220,8 @@ class GCNSubtarget final : public AMDGPUGenSubtargetInfo,
 
   bool hasScalarSubwordLoads() const { return getGeneration() >= GFX12; }
 
+  bool hasAsyncMark() const { return hasVMemToLDSLoad() || HasAsynccnt; }
+
   TrapHandlerAbi getTrapHandlerAbi() const {
     return isAmdHsaOS() ? TrapHandlerAbi::AMDHSA : TrapHandlerAbi::NONE;
   }
diff --git a/llvm/lib/Target/AMDGPU/SIInsertWaitcnts.cpp b/llvm/lib/Target/AMDGPU/SIInsertWaitcnts.cpp
index a804ba35bade7..16f278ca0ee6a 100644
--- a/llvm/lib/Target/AMDGPU/SIInsertWaitcnts.cpp
+++ b/llvm/lib/Target/AMDGPU/SIInsertWaitcnts.cpp
@@ -649,6 +649,14 @@ class SIInsertWaitcnts {
     return SIInstrInfo::mayWriteLDSThroughDMA(MI) && isAsync(MI);
   }
 
+  bool shouldUpdateAsyncMark(const MachineInstr &MI, InstCounterType T) const {
+    if (!isAsyncLdsDmaWrite(MI))
+      return false;
+    if (SIInstrInfo::usesASYNC_CNT(MI))
+      return T == ASYNC_CNT;
+    return T == LOAD_CNT;
+  }
+
   bool isVmemAccess(const MachineInstr &MI) const;
   bool generateWaitcntInstBefore(MachineInstr &MI,
                                  WaitcntBrackets &ScoreBrackets,
@@ -1252,12 +1260,7 @@ void WaitcntBrackets::updateByEvent(WaitEventType E, MachineInstr &Inst) {
         setVMemScore(LDSDMA_BEGIN + Slot, T, CurrScore);
     }
 
-    // FIXME: Not supported on GFX12 yet. Newer async operations use other
-    // counters too, so will need a map from instruction or event types to
-    // counter types.
-    if (Context->isAsyncLdsDmaWrite(Inst) && T == LOAD_CNT) {
-      assert(!SIInstrInfo::usesASYNC_CNT(Inst) &&
-             "unexpected GFX1250 instruction");
+    if (Context->shouldUpdateAsyncMark(Inst, T)) {
       AsyncScore[T] = CurrScore;
     }
 
@@ -2104,7 +2107,11 @@ bool WaitcntGeneratorGFX12Plus::applyPreexistingWaitcnt(
       II.eraseFromParent();
       Modified = true;
     } else if (Opcode == AMDGPU::WAIT_ASYNCMARK) {
-      reportFatalUsageError("WAIT_ASYNCMARK is not ready for GFX12 yet");
+      // Update the Waitcnt, but don't erase the wait.asyncmark() itself. It
+      // shows up in the assembly as a comment with the original parameter N.
+      unsigned N = II.getOperand(0).getImm();
+      AMDGPU::Waitcnt OldWait = ScoreBrackets.determineAsyncWait(N);
+      Wait = Wait.combined(OldWait);
     } else {
       std::optional<InstCounterType> CT = counterTypeForInstr(Opcode);
       assert(CT.has_value());
diff --git a/llvm/lib/Target/AMDGPU/SOPInstructions.td b/llvm/lib/Target/AMDGPU/SOPInstructions.td
index ce6e862104b4f..e3c1ba58197c8 100644
--- a/llvm/lib/Target/AMDGPU/SOPInstructions.td
+++ b/llvm/lib/Target/AMDGPU/SOPInstructions.td
@@ -1716,7 +1716,7 @@ def S_WAITCNT_lds_direct : SPseudoInstSI<(outs), (ins)> {
    let hasSideEffects = 0;
 }
 
-let SubtargetPredicate = HasVMemToLDSLoad in {
+let SubtargetPredicate = HasAsyncMark in {
 def ASYNCMARK : SPseudoInstSI<(outs), (ins),
   [(int_amdgcn_asyncmark)]> {
    let maybeAtomic = 0;
diff --git a/llvm/lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.h b/llvm/lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.h
index 9cec56090172b..fd6afe509239c 100644
--- a/llvm/lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.h
+++ b/llvm/lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.h
@@ -1207,7 +1207,7 @@ class Waitcnt {
   Waitcnt() = default;
   // Pre-gfx12 constructor.
   Waitcnt(unsigned VmCnt, unsigned ExpCnt, unsigned LgkmCnt, unsigned VsCnt)
-      : LoadCnt(VmCnt), ExpCnt(ExpCnt), DsCnt(LgkmCnt), StoreCnt(VsCnt) {}
+    : LoadCnt(VmCnt), ExpCnt(ExpCnt), DsCnt(LgkmCnt), StoreCnt(VsCnt) {}
 
   // gfx12+ constructor.
   Waitcnt(unsigned LoadCnt, unsigned ExpCnt, unsigned DsCnt, unsigned StoreCnt,
diff --git a/llvm/test/CodeGen/AMDGPU/asyncmark-err.ll b/llvm/test/CodeGen/AMDGPU/asyncmark-err.ll
deleted file mode 100644
index f929cb3e380b7..0000000000000
--- a/llvm/test/CodeGen/AMDGPU/asyncmark-err.ll
+++ /dev/null
@@ -1,19 +0,0 @@
-; RUN: split-file %s %t
-; RUN: not --crash llc -filetype=null -global-isel=0 -mtriple=amdgcn -mcpu=gfx1250 %t/mark.ll 2>&1 | FileCheck --ignore-case %s
-; RUN: not         llc -filetype=null -global-isel=1 -mtriple=amdgcn -mcpu=gfx1250 %t/mark.ll 2>&1 | FileCheck --ignore-case %s
-; RUN: not --crash llc -filetype=null -global-isel=0 -mtriple=amdgcn -mcpu=gfx1250 %t/wait.ll 2>&1 | FileCheck --ignore-case %s
-; RUN: not         llc -filetype=null -global-isel=1 -mtriple=amdgcn -mcpu=gfx1250 %t/wait.ll 2>&1 | FileCheck --ignore-case %s
-
-; CHECK: LLVM ERROR: Cannot select
-
-;--- mark.ll
-define void @async_err() {
-  call void @llvm.amdgcn.asyncmark()
-  ret void
-}
-
-;--- wait.ll
-define void @async_err() {
-  call void @llvm.amdgcn.wait.asyncmark(i16 0)
-  ret void
-}
diff --git a/llvm/test/CodeGen/AMDGPU/asyncmark-gfx12plus.ll b/llvm/test/CodeGen/AMDGPU/asyncmark-gfx12plus.ll
new file mode 100644
index 0000000000000..9e0acd51472c5
--- /dev/null
+++ b/llvm/test/CodeGen/AMDGPU/asyncmark-gfx12plus.ll
@@ -0,0 +1,366 @@
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 6
+; RUN: llc -march=amdgcn -mcpu=gfx1250  < %s | FileCheck %s -check-prefixes=GFX1250
+
+; Test async mark/wait with global_load_lds and global loads
+; This version uses wave barriers to enforce program order so that unrelated vmem
+; instructions do not get reordered before reaching this point.
+
+define void @interleaved_with_wave_barrier(ptr addrspace(1) %foo, ptr addrspace(3) %lds, ptr addrspace(1) %bar, ptr addrspace(1) %out) {
+; GFX1250-LABEL: interleaved_with_wave_barrier:
+; GFX1250:       ; %bb.0: ; %entry
+; GFX1250-NEXT:    s_wait_loadcnt_dscnt 0x0
+; GFX1250-NEXT:    s_wait_kmcnt 0x0
+; GFX1250-NEXT:    v_dual_mov_b32 v7, v6 :: v_dual_mov_b32 v9, v4
+; GFX1250-NEXT:    v_dual_mov_b32 v8, v3 :: v_dual_mov_b32 v6, v5
+; GFX1250-NEXT:    v_add_nc_u64_e32 v[4:5], 0x54, v[0:1]
+; GFX1250-NEXT:    v_add_nc_u32_e32 v3, 0x54, v2
+; GFX1250-NEXT:    global_load_b32 v10, v[8:9], off offset:44
+; GFX1250-NEXT:    global_load_b32 v11, v[0:1], off offset:4
+; GFX1250-NEXT:    ; wave barrier
+; GFX1250-NEXT:    global_load_async_to_lds_b32 v3, v[4:5], off offset:4 th:TH_LOAD_NT nv
+; GFX1250-NEXT:    v_add_nc_u64_e32 v[4:5], 0x58, v[8:9]
+; GFX1250-NEXT:    v_add_nc_u32_e32 v3, 0x58, v2
+; GFX1250-NEXT:    ; wave barrier
+; GFX1250-NEXT:    ; asyncmark
+; GFX1250-NEXT:    global_load_b32 v0, v[0:1], off offset:8
+; GFX1250-NEXT:    ; wave barrier
+; GFX1250-NEXT:    global_load_async_to_lds_b32 v3, v[4:5], off offset:4 th:TH_LOAD_LU nv
+; GFX1250-NEXT:    ; wave barrier
+; GFX1250-NEXT:    global_load_b32 v1, v[8:9], off offset:48
+; GFX1250-NEXT:    ; asyncmark
+; GFX1250-NEXT:    ; wait_asyncmark(1)
+; GFX1250-NEXT:    s_wait_asynccnt 0x1
+; GFX1250-NEXT:    ds_load_b32 v3, v2 offset:84
+; GFX1250-NEXT:    ; wait_asyncmark(0)
+; GFX1250-NEXT:    s_wait_asynccnt 0x0
+; GFX1250-NEXT:    ds_load_b32 v2, v2 offset:88
+; GFX1250-NEXT:    s_wait_loadcnt 0x2
+; GFX1250-NEXT:    v_add_nc_u32_e32 v4, v11, v10
+; GFX1250-NEXT:    s_wait_loadcnt_dscnt 0x101
+; GFX1250-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(SKIP_1) | instid1(VALU_DEP_1)
+; GFX1250-NEXT:    v_add3_u32 v0, v4, v3, v0
+; GFX1250-NEXT:    s_wait_loadcnt_dscnt 0x0
+; GFX1250-NEXT:    v_add3_u32 v0, v0, v1, v2
+; GFX1250-NEXT:    global_store_b32 v[6:7], v0, off
+; GFX1250-NEXT:    s_set_pc_i64 s[30:31]
+entry:
+  ; First batch: global load, global load, async global-to-LDS
+  %bar_gep11 = getelementptr i32, ptr addrspace(1) %bar, i32 11
+  %bar_v11 = load i32, ptr addrspace(1) %bar_gep11
+  %foo_gep1 = getelementptr i32, ptr addrspace(1) %foo, i32 1
+  %foo_v1 = load i32, ptr addrspace(1) %foo_gep1
+  %lds_gep21 = getelementptr i32, ptr addrspace(3) %lds, i32 21
+  %bar_gep21 = getelementptr i32, ptr addrspace(1) %foo, i32 21
+  call void @llvm.amdgcn.wave.barrier()
+  call void @llvm.amdgcn.global.load.async.to.lds.b32(ptr addrspace(1) %bar_gep21, ptr addrspace(3) %lds_gep21, i32 4, i32 u0x21)
+  call void @llvm.amdgcn.wave.barrier()
+  call void @llvm.amdgcn.asyncmark()
+
+  ; Second batch: global load, async global-to-LDS, global load
+  %foo_gep2 = getelementptr i32, ptr addrspace(1) %foo, i32 2
+  %foo_v2 = load i32, ptr addrspace(1) %foo_gep2
+  %bar_gep22 = getelementptr i32, ptr addrspace(1) %bar, i32 22
+  %lds_gep22 = getelementptr i32, ptr addrspace(3) %lds, i32 22
+  call void @llvm.amdgcn.wave.barrier()
+  call void @llvm.amdgcn.global.load.async.to.lds.b32(ptr addrspace(1) %bar_gep22, ptr addrspace(3) %lds_gep22, i32 4, i32 u0x23)
+  call void @llvm.amdgcn.wave.barrier()
+  %bar_gep12 = getelementptr i32, ptr addrspace(1) %bar, i32 12
+  %bar_v12 = load i32, ptr addrspace(1) %bar_gep12
+  call void @llvm.amdgcn.asyncmark()
+
+  ; Wait for first async mark and read from LDS
+  ; This results in vmcnt(3) corresponding to the second batch.
+  call void @llvm.amdgcn.wait.asyncmark(i16 1)
+  %lds_val21 = load i32, ptr addrspace(3) %lds_gep21
+
+  ; Wait for the next lds dma
+  ; This results in vmcnt(1), corresponding to %bar_v12. Could have been combined with the lgkmcnt(1) for %lds_val21.
+  ; Notable that the asyncmark is sufficient to prevent the optimizer from coalescing the previous ds_read with the next one.
+  call void @llvm.amdgcn.wait.asyncmark(i16 0)
+  %lds_val22 = load i32, ptr addrspace(3) %lds_gep22
+  %sum1 = add i32 %foo_v1, %bar_v11
+  %sum2 = add i32 %sum1, %lds_val21
+  %sum3 = add i32 %sum2, %foo_v2
+  ; Finally a vmcnt(0) for %bar_v12, which was not included in the async mark that followed it.
+  %sum4 = add i32 %sum3, %bar_v12
+  %sum5 = add i32 %sum4, %lds_val22
+  store i32 %sum5, ptr addrspace(1) %out
+
+  ret void
+}
+
+; A perfect loop that is unlikely to exist in real life. It uses only async LDS
+; DMA operations, and result in vmcnt waits that exactly match the stream of
+; those outstanding operations.
+
+define amdgpu_kernel void @test_pipelined_loop(ptr addrspace(1) %foo, ptr addrspace(3) %lds, ptr addrspace(1) %bar, ptr addrspace(1) %out, i32 %n) {
+; GFX1250-LABEL: test_pipelined_loop:
+; GFX1250:       ; %bb.0: ; %prolog
+; GFX1250-NEXT:    s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1
+; GFX1250-NEXT:    s_clause 0x1
+; GFX1250-NEXT:    s_load_b96 s[0:2], s[4:5], 0x24 nv
+; GFX1250-NEXT:    s_load_b32 s3, s[4:5], 0x44 nv
+; GFX1250-NEXT:    s_wait_kmcnt 0x0
+; GFX1250-NEXT:    v_dual_mov_b32 v0, 0 :: v_dual_mov_b32 v1, s2
+; GFX1250-NEXT:    s_add_co_i32 s6, s2, 4
+; GFX1250-NEXT:    s_mov_b32 s7, s2
+; GFX1250-NEXT:    v_mov_b32_e32 v2, s6
+; GFX1250-NEXT:    s_mov_b32 s6, 2
+; GFX1250-NEXT:    global_load_async_to_lds_b32 v1, v0, s[0:1] offset:4 nv
+; GFX1250-NEXT:    v_mov_b32_e32 v1, 4
+; GFX1250-NEXT:    ; asyncmark
+; GFX1250-NEXT:    global_load_async_to_lds_b32 v2, v1, s[0:1] offset:4 nv
+; GFX1250-NEXT:    v_mov_b32_e32 v1, 0
+; GFX1250-NEXT:    s_add_nc_u64 s[0:1], s[0:1], 8
+; GFX1250-NEXT:    ; asyncmark
+; GFX1250-NEXT:  .LBB1_1: ; %loop_body
+; GFX1250-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX1250-NEXT:    s_add_co_i32 s8, s7, 8
+; GFX1250-NEXT:    s_add_co_i32 s6, s6, 1
+; GFX1250-NEXT:    v_mov_b32_e32 v2, s8
+; GFX1250-NEXT:    global_load_async_to_lds_b32 v2, v0, s[0:1] offset:4 nv
+; GFX1250-NEXT:    v_mov_b32_e32 v2, s7
+; GFX1250-NEXT:    ; asyncmark
+; GFX1250-NEXT:    ; wait_asyncmark(2)
+; GFX1250-NEXT:    s_wait_asynccnt 0x2
+; GFX1250-NEXT:    s_add_co_i32 s7, s7, 4
+; GFX1250-NEXT:    s_cmp_lt_i32 s6, s3
+; GFX1250-NEXT:    ds_load_b32 v2, v2
+; GFX1250-NEXT:    s_add_nc_u64 s[0:1], s[0:1], 4
+; GFX1250-NEXT:    s_wait_dscnt 0x0
+; GFX1250-NEXT:    v_add_nc_u32_e32 v1, v1, v2
+; GFX1250-NEXT:    s_cbranch_scc1 .LBB1_1
+; GFX1250-NEXT:  ; %bb.2: ; %epilog
+; GFX1250-NEXT:    s_lshl2_add_u32 s0, s3, s2
+; GFX1250-NEXT:    ; wait_asyncmark(1)
+; GFX1250-NEXT:    s_wait_asynccnt 0x1
+; GFX1250-NEXT:    s_add_co_i32 s0, s0, -8
+; GFX1250-NEXT:    s_delay_alu instid0(SALU_CYCLE_1)
+; GFX1250-NEXT:    v_dual_mov_b32 v2, 0 :: v_dual_mov_b32 v0, s0
+; GFX1250-NEXT:    s_load_b64 s[0:1], s[4:5], 0x34 nv
+; GFX1250-NEXT:    ds_load_b32 v0, v0
+; GFX1250-NEXT:    ; wait_asyncmark(0)
+; GFX1250-NEXT:    s_wait_dscnt 0x0
+; GFX1250-NEXT:    s_wait_asynccnt 0x0
+; GFX1250-NEXT:    v_add_nc_u32_e32 v0, v1, v0
+; GFX1250-NEXT:    s_wait_kmcnt 0x0
+; GFX1250-NEXT:    global_store_b32 v2, v0, s[0:1]
+; GFX1250-NEXT:    s_endpgm
+prolog:
+  ; Load first iteration
+  call void @llvm.amdgcn.global.load.async.to.lds.b32(ptr addrspace(1) %foo, ptr addrspace(3) %lds, i32 4, i32 u0x20)
+  call void @llvm.amdgcn.asyncmark()
+
+  ; Load second iteration
+  %lds_gep1 = getelementptr i32, ptr addrspace(3) %lds, i32 1
+  %foo_gep1 = getelementptr i32, ptr addrspace(1) %foo, i32 1
+  call void @llvm.amdgcn.global.load.async.to.lds.b32(ptr addrspace(1) %foo_gep1, ptr addrspace(3) %lds_gep1, i32 4, i32 u0x20)
+  call void @llvm.amdgcn.asyncmark()
+
+  br label %loop_body
+
+loop_body:
+  %i = phi i32 [ 2, %prolog ], [ %i.next, %loop_body ]
+  %sum = phi i32 [ 0, %prolog ], [ %sum_i, %loop_body ]
+
+  ; Load next iteration
+  %lds_gep_cur = getelementptr i32, ptr addrspace(3) %lds, i32 %i
+  %foo_gep_cur = getelementptr i32, ptr addrspace(1) %foo, i32 %i
+  call void @llvm.amdgcn.global.load.async.to.lds.b32(ptr addrspace(1) %foo_gep_cur, ptr addrspace(3) %lds_gep_cur, i32 4, i32 u0x20)
+  call void @llvm.amdgcn.asyncmark()
+
+  ; Wait for iteration i-2 and process
+  call void @llvm.amdgcn.wait.asyncmark(i16 2)
+  %lds_idx = sub i32 %i, 2
+  %lds_gep_read = getelementptr i32, ptr addrspace(3) %lds, i32 %lds_idx
+  %lds_val = load i32, ptr addrspace(3) %lds_gep_read
+
+  %sum_i = add i32 %sum, %lds_val
+
+  %i.next = add i32 %i, 1
+  %cmp = icmp slt i32 %i.next, %n
+  br i1 %cmp, label %loop_body, label %epilog
+
+epilog:
+  ; Process remaining iterations
+  call void @llvm.amdgcn.wait.asyncmark(i16 1)
+  %lds_n_2 = sub i32 %n, 2
+  %lds_gep_n_2 = getelementptr i32, ptr addrspace(3) %lds, i32 %lds_n_2
+  %lds_val_n_2 = load i32, ptr addrspace(3) %lds_gep_n_2
+  %sum_e2 = add i32 %sum_i, %lds_val_n_2
+  %out_gep_e1 = getelementptr i32, ptr addrspace(1) %out, i32 %lds_n_2
+
+  call void @llvm.amdgcn.wait.asyncmark(i16 0)
+  %lds_n_1 = sub i32 %n, 1
+  %lds_gep_n_1 = getelementptr i32, ptr addrspace(3) %lds, i32 %lds_n_1
+  %lds_val_n_1 = load i32, ptr addrspace(3) %lds_gep_n_1
+  %sum_e1 = add i32 %sum_e2, %lds_val_n_1
+  store i32 %sum_e2, ptr addrspace(1) %bar
+
+  ret void
+}
+
+; Software pipelined loop with async global-to-LDS and global loads
+
+define amdgpu_kernel void @test_pipelined_loop_with_global(ptr addrspace(1) %foo, ptr addrspace(3) %lds, ptr addrspace(1) %bar, ptr addrspace(1) %out, i32 %n) {
+; GFX1250-LABEL: test_pipelined_loop_with_global:
+; GFX1250:       ; %bb.0: ; %prolog
+; GFX1250-NEXT:    s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1
+; GFX1250-NEXT:    s_clause 0x1
+; GFX1250-NEXT:    s_load_b96 s[8:10], s[4:5], 0x24 nv
+; GFX1250-NEXT:    s_load_b128 s[0:3], s[4:5], 0x34 nv
+; GFX1250-NEXT:    v_mov_b32_e32 v0, 0
+; GFX1250-NEXT:    s_wait_kmcnt 0x0
+; GFX1250-NEXT:    s_load_b32 s6, s[8:9], 0x0
+; GFX1250-NEXT:    s_load_b32 s7, s[0:1], 0x0
+; GFX1250-NEXT:    v_mov_b32_e32 v1, s10
+; GFX1250-NEXT:    s_add_co_i32 s11, s10, 4
+; GFX1250-NEXT:    s_delay_alu instid0(SALU_CYCLE_1)
+; GFX1250-NEXT:    v_dual_mov_b32 v3, 4 :: v_dual_mov_b32 v4, s11
+; GFX1250-NEXT:    s_load_b32 s11, s[4:5], 0x44 nv
+; GFX1250-NEXT:    global_load_async_to_lds_b32 v1, v0, s[8:9] offset:4 nv
+; GFX1250-NEXT:    ; asyncmark
+; GFX1250-NEXT:    s_clause 0x1
+; GFX1250-NEXT:    global_load_b32 v1, v0, s[8:9] offset:4
+; GFX1250-NEXT:    global_load_b32 v2, v0, s[0:1] offset:4
+; GFX1250-NEXT:    s_wait_xcnt 0x0
+; GFX1250-NEXT:    s_add_nc_u64 s[0:1], s[0:1], 8
+; GFX1250-NEXT:    s_add_nc_u64 s[4:5], s[8:9], 8
+; GFX1250-NEXT:    s_wait_kmcnt 0x0
+; GFX1250-NEXT:    v_dual_mov_b32 v5, s6 :: v_dual_mov_b32 v6, s7
+; GFX1250-NEXT:    s_mov_b64 s[6:7], s[2:3]
+; GFX1250-NEXT:    global_load_async_to_lds_b32 v4, v3, s[8:9] offset:4 nv
+; GFX1250-NEXT:    s_wait_loadcnt 0x0
+; GFX1250-NEXT:    v_dual_mov_b32 v3, v1 :: v_dual_mov_b32 v4, v2
+; GFX1250-NEXT:    s_mov_b32 s8, 2
+; GFX1250-NEXT:    s_mov_b32 s9, s10
+; GFX1250-NEXT:    ; asyncmark
+; GFX1250-NEXT:  .LBB2_1: ; %loop_body
+; GFX1250-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX1250-NEXT:    s_add_co_i32 s12, s9, 8
+; GFX1250-NEXT:    s_wait_loadcnt 0x0
+; GFX1250-NEXT:    v_dual_mov_b32 v7, v4 :: v_dual_mov_b32 v9, s12
+; GFX1250-NEXT:    v_mov_b32_e32 v8, v3
+; GFX1250-NEXT:    s_clause 0x1
+; GFX1250-NEXT:    global_load_b32 v3, v0, s[4:5]
+; GFX1250-NEXT:    global_load_b32 v4, v0, s[0:1]
+; GFX1250-NEXT:    v_dual_add_nc_u32 v10, v5, v6 :: v_dual_mov_b32 v6, v2
+; GFX1250-NEXT:    global_load_async_to_lds_b32 v9, v0, s[4:5] offset:4 nv
+; GFX1250-NEXT:    v_mov_b32_e32 v9, s9
+; GFX1250-NEXT:    ; asyncmark
+; GFX1250-NEXT:    ; wait_asyncmark(2)
+; GFX1250-NEXT:    s_wait_asynccnt 0x2
+; GFX1250-NEXT:    s_wait_asynccnt 0x2
+; GFX1250-NEXT:    s_add_co_i32 s8, s8, 1
+; GFX1250-NEXT:    s_add_co_i32 s9, s9, 4
+; GFX1250-NEXT:    ds_load_b32 v9, v9
+; GFX1250-NEXT:    v_mov_b32_e32 v5, v1
+; GFX1250-NEXT:    s_cmp_lt_i32 s8, s11
+; GFX1250-NEXT:    s_wait_xcnt 0x0
+; GFX1250-NEXT:    s_add_nc_u64 s[0:1], s[0:1], 4
+; GFX1250-NEXT:    s_add_nc_u64 s[4:5], s[4:5], 4
+; GFX1250-NEXT:    s_wait_dscnt 0x0
+; GFX1250-NEXT:    v_add_nc_u32_e32 v9, v10, v9
+; GFX1250-NEXT:    global_store_b32 v0, v9, s[6:7]
+; GFX1250-NEXT:    s_wait_xcnt 0x0
+; GFX1250-NEXT:    s_add_nc_u64 s[6:7], s[6:7], 4
+; GFX1250-NEXT:    s_cbranch_scc1 .LBB2_1
+; GFX1250-NEXT:  ; %bb.2: ; %epilog
+; GFX1250-NEXT:    s_add_co_i32 s0, s11, -2
+; GFX1250-NEXT:    ; wait_asyncmark(1)
+; GFX1250-NEXT:    s_wait_asynccnt 0x1
+; GFX1250-NEXT:    s_lshl2_add_u32 s1, s0, s10
+; GFX1250-NEXT:    s_delay_alu instid0(SALU_CYCLE_1)
+; GFX1250-NEXT:    v_dual_add_nc_u32 v2, v8, v7 :: v_dual_mov_b32 v0, s1
+; GFX1250-NEXT:    ds_load_b32 v1, v0
+; GFX1250-NEXT:    s_wait_dscnt 0x0
+; GFX1250-NEXT:    v_dual_mov_b32 v5, s0 :: v_dual_add_nc_u32 v1, v2, v1
+; GFX1250-NEXT:    global_store_b32 v5, v1, s[2:3] scale_offset
+; GFX1250-NEXT:    ; wait_asyncmark(0)
+; GFX1250-NEXT:    s_wait_asynccnt 0x0
+; GFX1250-NEXT:    ds_load_b32 v0, v0 offset:4
+; GFX1250-NEXT:    s_wait_loadcnt 0x0
+; GFX1250-NEXT:    s_wait_xcnt 0x0
+; GFX1250-NEXT:    v_add_nc_u32_e32 v1, v3, v4
+; GFX1250-NEXT:    s_wait_dscnt 0x0
+; GFX1250-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GFX1250-NEXT:    v_add_nc_u32_e32 v0, v1, v0
+; GFX1250-NEXT:    global_store_b32 v5, v0, s[2:3] offset:4 scale_offset
+; GFX1250-NEXT:    s_endpgm
+prolog:
+  ; Load first iteration
+  %v0 = load i32, ptr addrspace(1) %foo
+  %g0 = load i32, ptr addrspace(1) %bar
+  call void @llvm.amdgcn.global.load.async.to.lds.b32(ptr addrspace(1) %foo, ptr addrspace(3) %lds, i32 4, i32 u0x20)
+  c...
[truncated]

github-actions · 2026-03-11T06:19:59Z

✅ With the latest revision this PR passed the C/C++ code formatter.

Pierre-vh · 2026-03-11T09:22:13Z

@@ -0,0 +1,366 @@
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 6
+; RUN: llc -march=amdgcn -mcpu=gfx1250  < %s | FileCheck %s -check-prefixes=GFX1250


Add GISel run line

I missed GISel in the pre-gfx12 version as well. I would prefer to have this current PR submitted and then separately work on an new patch to implement the missing pieces and update the tests for both versions (pre-gfx12 and gfx12-plus).

arsenm · 2026-03-11T09:55:17Z

@@ -0,0 +1,366 @@
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 6
+; RUN: llc -march=amdgcn -mcpu=gfx1250  < %s | FileCheck %s -check-prefixes=GFX1250


Don't use -march

krzysz00

Overall seems reasonable

krzysz00 · 2026-03-11T18:37:41Z

+  %lds_val21 = load i32, ptr addrspace(3) %lds_gep21
+
+  ; Wait for the next lds dma
+  ; This results in vmcnt(1), corresponding to %bar_v12. Could have been combined with the lgkmcnt(1) for %lds_val21.


Stale comments?

Fixed. Tried to remove all copy-pasta from the original test.

Async operations transfer data between global memory and LDS. Their progress is tracked by the ASYNC_CNT counter on GFX1250 and later architectures. This change introduces the representation of that counter in SIInsertWaitCnts. For now, the programmer must manually insert s_wait_asyncnt instructions. Later changes will add compiler assistance for generating the waits by including this counter in the asyncmark instructions. Assisted-by: Claude Sonnet 4.5 This is part of a stack: - #185813 - #185810

Async operations transfer data between global memory and LDS. Their progress is tracked by the ASYNC_CNT counter on GFX1250 and later architectures. This change introduces the representation of that counter in SIInsertWaitCnts. For now, the programmer must manually insert s_wait_asyncnt instructions. Later changes will add compiler assistance for generating the waits by including this counter in the asyncmark instructions. Assisted-by: Claude Sonnet 4.5 This is part of a stack: - llvm#185813 - llvm#185810

ssahasra · 2026-04-04T01:56:49Z

As far as I can see, the only pending issue is that globalisel was never tested or even implemented. I will do it as a separate patch. Can we let this one through?

llvm-ci · 2026-04-07T05:42:10Z

LLVM Buildbot has detected a new failure on builder llvm-clang-x86_64-darwin running on doug-worker-3 while building llvm at step 2 "checkout".

Full details are available at: https://lab.llvm.org/buildbot/#/builders/23/builds/19021

Here is the relevant piece of the build log for the reference

Step 2 (checkout) failure: update (failure)
...
 * [new tag]                   llvmorg-5.0.1-rc1  -> llvmorg-5.0.1-rc1
 * [new tag]                   llvmorg-5.0.1-rc2  -> llvmorg-5.0.1-rc2
 * [new tag]                   llvmorg-5.0.1-rc3  -> llvmorg-5.0.1-rc3
 * [new tag]                   llvmorg-5.0.2      -> llvmorg-5.0.2
 * [new tag]                   llvmorg-5.0.2-rc1  -> llvmorg-5.0.2-rc1
 * [new tag]                   llvmorg-5.0.2-rc2  -> llvmorg-5.0.2-rc2
 * [new tag]                   llvmorg-6.0.0      -> llvmorg-6.0.0
 * [new tag]                   llvmorg-6.0.0-rc1  -> llvmorg-6.0.0-rc1
 * [new tag]                   llvmorg-6.0.0-rc2  -> llvmorg-6.0.0-rc2
 * [new tag]                   llvmorg-6.0.0-rc3  -> llvmorg-6.0.0-rc3
 * [new tag]                   llvmorg-6.0.1      -> llvmorg-6.0.1
 * [new tag]                   llvmorg-6.0.1-rc1  -> llvmorg-6.0.1-rc1
 * [new tag]                   llvmorg-6.0.1-rc2  -> llvmorg-6.0.1-rc2
 * [new tag]                   llvmorg-6.0.1-rc3  -> llvmorg-6.0.1-rc3
 * [new tag]                   llvmorg-7.0.0      -> llvmorg-7.0.0
 * [new tag]                   llvmorg-7.0.0-rc1  -> llvmorg-7.0.0-rc1
 * [new tag]                   llvmorg-7.0.0-rc2  -> llvmorg-7.0.0-rc2
 * [new tag]                   llvmorg-7.0.0-rc3  -> llvmorg-7.0.0-rc3
 * [new tag]                   llvmorg-7.0.1      -> llvmorg-7.0.1
 * [new tag]                   llvmorg-7.0.1-rc1  -> llvmorg-7.0.1-rc1
 * [new tag]                   llvmorg-7.0.1-rc2  -> llvmorg-7.0.1-rc2
 * [new tag]                   llvmorg-7.0.1-rc3  -> llvmorg-7.0.1-rc3
 * [new tag]                   llvmorg-7.1.0      -> llvmorg-7.1.0
 * [new tag]                   llvmorg-7.1.0-rc1  -> llvmorg-7.1.0-rc1
 * [new tag]                   llvmorg-8.0.0      -> llvmorg-8.0.0
 * [new tag]                   llvmorg-8.0.0-rc1  -> llvmorg-8.0.0-rc1
 * [new tag]                   llvmorg-8.0.0-rc2  -> llvmorg-8.0.0-rc2
 * [new tag]                   llvmorg-8.0.0-rc3  -> llvmorg-8.0.0-rc3
 * [new tag]                   llvmorg-8.0.0-rc4  -> llvmorg-8.0.0-rc4
 * [new tag]                   llvmorg-8.0.0-rc5  -> llvmorg-8.0.0-rc5
 * [new tag]                   llvmorg-8.0.1      -> llvmorg-8.0.1
 * [new tag]                   llvmorg-8.0.1-rc1  -> llvmorg-8.0.1-rc1
 * [new tag]                   llvmorg-8.0.1-rc2  -> llvmorg-8.0.1-rc2
 * [new tag]                   llvmorg-8.0.1-rc3  -> llvmorg-8.0.1-rc3
 * [new tag]                   llvmorg-8.0.1-rc4  -> llvmorg-8.0.1-rc4
 * [new tag]                   llvmorg-9.0.0      -> llvmorg-9.0.0
 * [new tag]                   llvmorg-9.0.0-rc1  -> llvmorg-9.0.0-rc1
 * [new tag]                   llvmorg-9.0.0-rc2  -> llvmorg-9.0.0-rc2
 * [new tag]                   llvmorg-9.0.0-rc3  -> llvmorg-9.0.0-rc3
 * [new tag]                   llvmorg-9.0.0-rc4  -> llvmorg-9.0.0-rc4
 * [new tag]                   llvmorg-9.0.0-rc5  -> llvmorg-9.0.0-rc5
 * [new tag]                   llvmorg-9.0.0-rc6  -> llvmorg-9.0.0-rc6
 * [new tag]                   llvmorg-9.0.1      -> llvmorg-9.0.1
 * [new tag]                   llvmorg-9.0.1-rc1  -> llvmorg-9.0.1-rc1
 * [new tag]                   llvmorg-9.0.1-rc2  -> llvmorg-9.0.1-rc2
 * [new tag]                   llvmorg-9.0.1-rc3  -> llvmorg-9.0.1-rc3
fatal: reference is not a tree: f9adee2f6b1f06ae1623a044128997482da5132c
From https://github.com/llvm/llvm-project
 * branch                      main       -> FETCH_HEAD
fatal: reference is not a tree: f9adee2f6b1f06ae1623a044128997482da5132c

The ASYNC_CNT is used to track the progress of asynchronous copies between global and LDS memories. By including it in asyncmark, the compiler can now assist the programmer in generating waits for ASYNC_CNT. Assisted-By: Claude Sonnet 4.5 This is part of a stack: - llvm#185813 - llvm#185810 Fixes: LCOMPILER-332

Update MIR test to use gfx1250 async LDS DMA Switch from gfx950 + BUFFER_LOAD_DWORD_LDS_IDXEN to gfx1250 + GLOBAL_LOAD_ASYNC_TO_LDS_B32 as suggested in review. On GFX1250 the async load is tracked via ASYNC_CNT rather than VM_CNT, which properly exercises the ASYNCMARK/WAIT_ASYNCMARK machinery. The join block carries WAIT_ASYNCMARK 0; in asyncmark_in_else the pass expands it to S_WAIT_ASYNCCNT 0 to drain the pending async count. Rebased onto upstream main to pick up GFX1250 ASYNCMARK support added in llvm#185813. Regenerate CHECK lines with update_mir_test_checks.py.

The ASYNC_CNT is used to track the progress of asynchronous copies between global and LDS memories. By including it in asyncmark, the compiler can now assist the programmer in generating waits for ASYNC_CNT. Assisted-By: Claude Sonnet 4.5 This is part of a stack: - llvm#185813 - llvm#185810 Fixes: LCOMPILER-332

ssahasra added 2 commits March 11, 2026 10:46

[AMDGPU] asyncmark support for ASYNC_CNT

121f1a8

The ASYNC_CNT is used to track the progress of asynchronous copies between global and LDS memories. By including it in asyncmark, the compiler can now assist the programmer in generating waits for ASYNC_CNT. Assisted-By: Claude Sonnet 4.5

ssahasra requested review from Pierre-vh, RyanRio, arsenm, jayfoad, krzysz00 and nhaehnle March 11, 2026 06:18

llvmbot added the backend:AMDGPU label Mar 11, 2026

ssahasra mentioned this pull request Mar 11, 2026

[AMDGPU] Introduce ASYNC_CNT on GFX1250 #185810

Merged

Pierre-vh reviewed Mar 11, 2026

View reviewed changes

arsenm reviewed Mar 11, 2026

View reviewed changes

krzysz00 reviewed Mar 11, 2026

View reviewed changes

ssahasra force-pushed the users/ssahasra/async-cnt-gfx1250 branch from c912af8 to 68e3556 Compare March 20, 2026 09:12

Base automatically changed from users/ssahasra/async-cnt-gfx1250 to main March 20, 2026 09:52

ssahasra added 2 commits March 25, 2026 15:56

Merge branch 'main' into users/ssahasra/asyncmark-gfx1250

56714ad

don't use -march, remove stale pre-gfx12 comments

c93970a

arsenm approved these changes Apr 6, 2026

View reviewed changes

RyanRio approved these changes Apr 6, 2026

View reviewed changes

ssahasra merged commit f9adee2 into main Apr 7, 2026
10 checks passed

ssahasra deleted the users/ssahasra/asyncmark-gfx1250 branch April 7, 2026 01:53

ssahasra mentioned this pull request Apr 29, 2026

[AMDGPU][SIInsertWaitcnts] Fix iota_range assertion when OtherMarks is empty in mergeAsyncMarks() #193499

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[AMDGPU] asyncmark support for ASYNC_CNT#185813

[AMDGPU] asyncmark support for ASYNC_CNT#185813
ssahasra merged 4 commits into
mainfrom
users/ssahasra/asyncmark-gfx1250

ssahasra commented Mar 11, 2026 •

edited

Loading

Uh oh!

llvmbot commented Mar 11, 2026

Uh oh!

github-actions Bot commented Mar 11, 2026 •

edited

Loading

Uh oh!

Pierre-vh Mar 11, 2026

Uh oh!

ssahasra Mar 26, 2026

Uh oh!

arsenm Mar 11, 2026

Uh oh!

ssahasra Mar 26, 2026

Uh oh!

krzysz00 left a comment

Uh oh!

krzysz00 Mar 11, 2026

Uh oh!

ssahasra Mar 26, 2026

Uh oh!

ssahasra commented Apr 4, 2026

Uh oh!

Uh oh!

llvm-ci commented Apr 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

		@@ -0,0 +1,366 @@
		; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 6
		; RUN: llc -march=amdgcn -mcpu=gfx1250 < %s \| FileCheck %s -check-prefixes=GFX1250

Conversation

ssahasra commented Mar 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

llvmbot commented Mar 11, 2026

Uh oh!

github-actions Bot commented Mar 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Pierre-vh Mar 11, 2026

Choose a reason for hiding this comment

Uh oh!

ssahasra Mar 26, 2026

Choose a reason for hiding this comment

Uh oh!

arsenm Mar 11, 2026

Choose a reason for hiding this comment

Uh oh!

ssahasra Mar 26, 2026

Choose a reason for hiding this comment

Uh oh!

krzysz00 left a comment

Choose a reason for hiding this comment

Uh oh!

krzysz00 Mar 11, 2026

Choose a reason for hiding this comment

Uh oh!

ssahasra Mar 26, 2026

Choose a reason for hiding this comment

Uh oh!

ssahasra commented Apr 4, 2026

Uh oh!

Uh oh!

llvm-ci commented Apr 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

ssahasra commented Mar 11, 2026 •

edited

Loading

github-actions Bot commented Mar 11, 2026 •

edited

Loading