[RISCV] Custom lower VP_LOAD/VP_STORE of mask vectors by wangpc-pp · Pull Request #200146 · llvm/llvm-project

wangpc-pp · 2026-05-28T09:46:02Z

For scalable mask vectors (<vscale x N x i1>), VP_LOAD/VP_STORE
were left at the default Expand action.

VectorLegalizer::Expand falls back to UnrollVectorOp, which
only supports fixed-length vectors and ends up calling
getVectorNumElements() on a scalable type, crashing the compiler.

This is reachable from llvm.vp.load / llvm.vp.store of nxvNi1.

Fix it by:

Setting VP_LOAD/VP_STORE to Custom for all legal BoolVecVTs.
Adding a mask-element-type fast path in lowerMaskedLoad /
lowerMaskedStore that lowers the operation to the mask-register
instructions vlm.v / vsm.v (mirroring the existing path in
lowerFixedLengthVectorLoadToRVV / lowerFixedLengthVectorStoreToRVV).
Adds tests in llvm/test/CodeGen/RISCV/rvv/vpload.ll and vpstore.ll
for nxv{1,2,4,8,16,32,64}i1.

This fixes #199896.

This PR is done by TRAE (ByteDance's coding agent).

For scalable mask vectors (`<vscale x N x i1>`), `VP_LOAD/VP_STORE` were left at the default `Expand` action. `VectorLegalizer::Expand` falls back to `UnrollVectorOp`, which only supports fixed-length vectors and ends up calling `getVectorNumElements()` on a scalable type, crashing the compiler. This is reachable from `llvm.vp.load` / `llvm.vp.store` of `nxvNi1`. Fix it by: 1. Setting `VP_LOAD`/`VP_STORE` to `Custom` for all legal `BoolVecVTs`. 2. Adding a mask-element-type fast path in `lowerMaskedLoad` / `lowerMaskedStore` that lowers the operation to the mask-register instructions `vlm.v` / `vsm.v` (mirroring the existing path in `lowerFixedLengthVectorLoadToRVV` / `lowerFixedLengthVectorStoreToRVV`). 3. Adds tests in `llvm/test/CodeGen/RISCV/rvv/vpload.ll` and `vpstore.ll` for `nxv{1,2,4,8,16,32,64}i1. This fixes llvm#199896. This PR is done by TRAE (ByteDance's coding agent).

llvmorg-github-actions · 2026-05-28T09:46:47Z

@llvm/pr-subscribers-backend-risc-v

Author: Pengcheng Wang (wangpc-pp)

Changes

For scalable mask vectors (<vscale x N x i1>), VP_LOAD/VP_STORE
were left at the default Expand action.

VectorLegalizer::Expand falls back to UnrollVectorOp, which
only supports fixed-length vectors and ends up calling
getVectorNumElements() on a scalable type, crashing the compiler.

This is reachable from llvm.vp.load / llvm.vp.store of nxvNi1.

Fix it by:

Setting VP_LOAD/VP_STORE to Custom for all legal BoolVecVTs.
Adding a mask-element-type fast path in lowerMaskedLoad /
lowerMaskedStore that lowers the operation to the mask-register
instructions vlm.v / vsm.v (mirroring the existing path in
lowerFixedLengthVectorLoadToRVV / lowerFixedLengthVectorStoreToRVV).
Adds tests in llvm/test/CodeGen/RISCV/rvv/vpload.ll and vpstore.ll
for `nxv{1,2,4,8,16,32,64}i1.

This fixes #199896.

This PR is done by TRAE (ByteDance's coding agent).

Full diff: https://github.com/llvm/llvm-project/pull/200146.diff

3 Files Affected:

(modified) llvm/lib/Target/RISCV/RISCVISelLowering.cpp (+30)
(modified) llvm/test/CodeGen/RISCV/rvv/vpload.ll (+80)
(modified) llvm/test/CodeGen/RISCV/rvv/vpstore.ll (+80)

diff --git a/llvm/lib/Target/RISCV/RISCVISelLowering.cpp b/llvm/lib/Target/RISCV/RISCVISelLowering.cpp
index 8add1d540c3ca..a56af20f50926 100644
--- a/llvm/lib/Target/RISCV/RISCVISelLowering.cpp
+++ b/llvm/lib/Target/RISCV/RISCVISelLowering.cpp
@@ -979,6 +979,8 @@ RISCVTargetLowering::RISCVTargetLowering(const TargetMachine &TM,
                          Expand);
       setOperationAction(ISD::VP_MERGE, VT, Custom);
 
+      setOperationAction({ISD::VP_LOAD, ISD::VP_STORE}, VT, Custom);
+
       setOperationAction({ISD::CTTZ_ELTS, ISD::CTTZ_ELTS_ZERO_POISON,
                           ISD::VP_CTTZ_ELTS, ISD::VP_CTTZ_ELTS_ZERO_POISON},
                          VT, Custom);
@@ -13762,6 +13764,21 @@ SDValue RISCVTargetLowering::lowerMaskedLoad(SDValue Op,
 
   MVT XLenVT = Subtarget.getXLenVT();
 
+  // VP_LOAD of a mask vector lowers directly to vlm.v on the mask register.
+  if (VT.getVectorElementType() == MVT::i1) {
+    assert(isa<VPLoadSDNode>(Op) &&
+           "MaskedLoad of i1 vector should not reach here");
+    if (!VL)
+      VL = getDefaultVLOps(VT, VT, DL, DAG, Subtarget).second;
+    SDValue IntID = DAG.getTargetConstant(Intrinsic::riscv_vlm, DL, XLenVT);
+    SDValue Ops[] = {Chain, IntID, BasePtr, VL};
+    SDVTList VTs = DAG.getVTList({VT, MVT::Other});
+    SDValue Result =
+        DAG.getMemIntrinsicNode(ISD::INTRINSIC_W_CHAIN, DL, VTs, Ops, MemVT,
+                                MMO);
+    return DAG.getMergeValues({Result, Result.getValue(1)}, DL);
+  }
+
   MVT ContainerVT = VT;
   if (VT.isFixedLengthVector()) {
     ContainerVT = getContainerForFixedLengthVector(VT);
@@ -13909,6 +13926,19 @@ SDValue RISCVTargetLowering::lowerMaskedStore(SDValue Op,
   MVT VT = Val.getSimpleValueType();
   MVT XLenVT = Subtarget.getXLenVT();
 
+  // VP_STORE of a mask vector lowers directly to vsm.v on the mask register.
+  if (VT.getVectorElementType() == MVT::i1) {
+    assert(isa<VPStoreSDNode>(Op) &&
+           "MaskedStore of i1 vector should not reach here");
+    if (!VL)
+      VL = getDefaultVLOps(VT, VT, DL, DAG, Subtarget).second;
+    SDValue IntID = DAG.getTargetConstant(Intrinsic::riscv_vsm, DL, XLenVT);
+    return DAG.getMemIntrinsicNode(ISD::INTRINSIC_VOID, DL,
+                                   DAG.getVTList(MVT::Other),
+                                   {Chain, IntID, Val, BasePtr, VL}, MemVT,
+                                   MMO);
+  }
+
   MVT ContainerVT = VT;
   if (VT.isFixedLengthVector()) {
     ContainerVT = getContainerForFixedLengthVector(VT);
diff --git a/llvm/test/CodeGen/RISCV/rvv/vpload.ll b/llvm/test/CodeGen/RISCV/rvv/vpload.ll
index 2ece316c7e54a..74a3c45734bfd 100644
--- a/llvm/test/CodeGen/RISCV/rvv/vpload.ll
+++ b/llvm/test/CodeGen/RISCV/rvv/vpload.ll
@@ -560,3 +560,83 @@ define <vscale x 8 x i8> @vpload_all_active_nxv8i8(ptr %ptr) {
   %load = call <vscale x 8 x i8> @llvm.vp.load.nxv8i8.p0(ptr %ptr, <vscale x 8 x i1> splat (i1 true), i32 %evl)
   ret <vscale x 8 x i8> %load
 }
+
+define <vscale x 1 x i1> @vpload_nxv1i1(ptr %ptr, i32 zeroext %evl) {
+; CHECK-LABEL: vpload_nxv1i1:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
+; CHECK-NEXT:    vlm.v v0, (a0)
+; CHECK-NEXT:    ret
+  %load = call <vscale x 1 x i1> @llvm.vp.load.nxv1i1.p0(ptr %ptr, <vscale x 1 x i1> splat (i1 true), i32 %evl)
+  ret <vscale x 1 x i1> %load
+}
+
+define <vscale x 2 x i1> @vpload_nxv2i1(ptr %ptr, i32 zeroext %evl) {
+; CHECK-LABEL: vpload_nxv2i1:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
+; CHECK-NEXT:    vlm.v v0, (a0)
+; CHECK-NEXT:    ret
+  %load = call <vscale x 2 x i1> @llvm.vp.load.nxv2i1.p0(ptr %ptr, <vscale x 2 x i1> splat (i1 true), i32 %evl)
+  ret <vscale x 2 x i1> %load
+}
+
+define <vscale x 4 x i1> @vpload_nxv4i1(ptr %ptr, i32 zeroext %evl) {
+; CHECK-LABEL: vpload_nxv4i1:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
+; CHECK-NEXT:    vlm.v v0, (a0)
+; CHECK-NEXT:    ret
+  %load = call <vscale x 4 x i1> @llvm.vp.load.nxv4i1.p0(ptr %ptr, <vscale x 4 x i1> splat (i1 true), i32 %evl)
+  ret <vscale x 4 x i1> %load
+}
+
+define <vscale x 8 x i1> @vpload_nxv8i1(ptr %ptr, i32 zeroext %evl) {
+; CHECK-LABEL: vpload_nxv8i1:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
+; CHECK-NEXT:    vlm.v v0, (a0)
+; CHECK-NEXT:    ret
+  %load = call <vscale x 8 x i1> @llvm.vp.load.nxv8i1.p0(ptr %ptr, <vscale x 8 x i1> splat (i1 true), i32 %evl)
+  ret <vscale x 8 x i1> %load
+}
+
+define <vscale x 16 x i1> @vpload_nxv16i1(ptr %ptr, i32 zeroext %evl) {
+; CHECK-LABEL: vpload_nxv16i1:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, ma
+; CHECK-NEXT:    vlm.v v0, (a0)
+; CHECK-NEXT:    ret
+  %load = call <vscale x 16 x i1> @llvm.vp.load.nxv16i1.p0(ptr %ptr, <vscale x 16 x i1> splat (i1 true), i32 %evl)
+  ret <vscale x 16 x i1> %load
+}
+
+define <vscale x 32 x i1> @vpload_nxv32i1(ptr %ptr, i32 zeroext %evl) {
+; CHECK-LABEL: vpload_nxv32i1:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli zero, a1, e8, m4, ta, ma
+; CHECK-NEXT:    vlm.v v0, (a0)
+; CHECK-NEXT:    ret
+  %load = call <vscale x 32 x i1> @llvm.vp.load.nxv32i1.p0(ptr %ptr, <vscale x 32 x i1> splat (i1 true), i32 %evl)
+  ret <vscale x 32 x i1> %load
+}
+
+define <vscale x 64 x i1> @vpload_nxv64i1(ptr %ptr, i32 zeroext %evl) {
+; CHECK-LABEL: vpload_nxv64i1:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli zero, a1, e8, m8, ta, ma
+; CHECK-NEXT:    vlm.v v0, (a0)
+; CHECK-NEXT:    ret
+  %load = call <vscale x 64 x i1> @llvm.vp.load.nxv64i1.p0(ptr %ptr, <vscale x 64 x i1> splat (i1 true), i32 %evl)
+  ret <vscale x 64 x i1> %load
+}
+
+define <vscale x 16 x i1> @vpload_nxv16i1_masked(ptr %ptr, <vscale x 16 x i1> %m, i32 zeroext %evl) {
+; CHECK-LABEL: vpload_nxv16i1_masked:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, ma
+; CHECK-NEXT:    vlm.v v0, (a0)
+; CHECK-NEXT:    ret
+  %load = call <vscale x 16 x i1> @llvm.vp.load.nxv16i1.p0(ptr %ptr, <vscale x 16 x i1> %m, i32 %evl)
+  ret <vscale x 16 x i1> %load
+}
diff --git a/llvm/test/CodeGen/RISCV/rvv/vpstore.ll b/llvm/test/CodeGen/RISCV/rvv/vpstore.ll
index 9fd8b9d23cb5e..8d16c213551ed 100644
--- a/llvm/test/CodeGen/RISCV/rvv/vpstore.ll
+++ b/llvm/test/CodeGen/RISCV/rvv/vpstore.ll
@@ -462,3 +462,83 @@ define void @vpstore_all_active_nxv8i8(<vscale x 8 x i8> %val, ptr %ptr) {
   call void @llvm.vp.store.nxv8i8.p0(<vscale x 8 x i8> %val, ptr %ptr, <vscale x 8 x i1> splat (i1 true), i32 %evl)
   ret void
 }
+
+define void @vpstore_nxv1i1(<vscale x 1 x i1> %val, ptr %ptr, i32 zeroext %evl) {
+; CHECK-LABEL: vpstore_nxv1i1:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli zero, a1, e8, mf8, ta, ma
+; CHECK-NEXT:    vsm.v v0, (a0)
+; CHECK-NEXT:    ret
+  call void @llvm.vp.store.nxv1i1.p0(<vscale x 1 x i1> %val, ptr %ptr, <vscale x 1 x i1> splat (i1 true), i32 %evl)
+  ret void
+}
+
+define void @vpstore_nxv2i1(<vscale x 2 x i1> %val, ptr %ptr, i32 zeroext %evl) {
+; CHECK-LABEL: vpstore_nxv2i1:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli zero, a1, e8, mf4, ta, ma
+; CHECK-NEXT:    vsm.v v0, (a0)
+; CHECK-NEXT:    ret
+  call void @llvm.vp.store.nxv2i1.p0(<vscale x 2 x i1> %val, ptr %ptr, <vscale x 2 x i1> splat (i1 true), i32 %evl)
+  ret void
+}
+
+define void @vpstore_nxv4i1(<vscale x 4 x i1> %val, ptr %ptr, i32 zeroext %evl) {
+; CHECK-LABEL: vpstore_nxv4i1:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli zero, a1, e8, mf2, ta, ma
+; CHECK-NEXT:    vsm.v v0, (a0)
+; CHECK-NEXT:    ret
+  call void @llvm.vp.store.nxv4i1.p0(<vscale x 4 x i1> %val, ptr %ptr, <vscale x 4 x i1> splat (i1 true), i32 %evl)
+  ret void
+}
+
+define void @vpstore_nxv8i1(<vscale x 8 x i1> %val, ptr %ptr, i32 zeroext %evl) {
+; CHECK-LABEL: vpstore_nxv8i1:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli zero, a1, e8, m1, ta, ma
+; CHECK-NEXT:    vsm.v v0, (a0)
+; CHECK-NEXT:    ret
+  call void @llvm.vp.store.nxv8i1.p0(<vscale x 8 x i1> %val, ptr %ptr, <vscale x 8 x i1> splat (i1 true), i32 %evl)
+  ret void
+}
+
+define void @vpstore_nxv16i1(<vscale x 16 x i1> %val, ptr %ptr, i32 zeroext %evl) {
+; CHECK-LABEL: vpstore_nxv16i1:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, ma
+; CHECK-NEXT:    vsm.v v0, (a0)
+; CHECK-NEXT:    ret
+  call void @llvm.vp.store.nxv16i1.p0(<vscale x 16 x i1> %val, ptr %ptr, <vscale x 16 x i1> splat (i1 true), i32 %evl)
+  ret void
+}
+
+define void @vpstore_nxv32i1(<vscale x 32 x i1> %val, ptr %ptr, i32 zeroext %evl) {
+; CHECK-LABEL: vpstore_nxv32i1:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli zero, a1, e8, m4, ta, ma
+; CHECK-NEXT:    vsm.v v0, (a0)
+; CHECK-NEXT:    ret
+  call void @llvm.vp.store.nxv32i1.p0(<vscale x 32 x i1> %val, ptr %ptr, <vscale x 32 x i1> splat (i1 true), i32 %evl)
+  ret void
+}
+
+define void @vpstore_nxv64i1(<vscale x 64 x i1> %val, ptr %ptr, i32 zeroext %evl) {
+; CHECK-LABEL: vpstore_nxv64i1:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli zero, a1, e8, m8, ta, ma
+; CHECK-NEXT:    vsm.v v0, (a0)
+; CHECK-NEXT:    ret
+  call void @llvm.vp.store.nxv64i1.p0(<vscale x 64 x i1> %val, ptr %ptr, <vscale x 64 x i1> splat (i1 true), i32 %evl)
+  ret void
+}
+
+define void @vpstore_nxv16i1_masked(<vscale x 16 x i1> %val, ptr %ptr, <vscale x 16 x i1> %m, i32 zeroext %evl) {
+; CHECK-LABEL: vpstore_nxv16i1_masked:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vsetvli zero, a1, e8, m2, ta, ma
+; CHECK-NEXT:    vsm.v v0, (a0)
+; CHECK-NEXT:    ret
+  call void @llvm.vp.store.nxv16i1.p0(<vscale x 16 x i1> %val, ptr %ptr, <vscale x 16 x i1> %m, i32 %evl)
+  ret void
+}

github-actions · 2026-05-28T09:47:37Z

✅ With the latest revision this PR passed the C/C++ code formatter.

wangpc-pp · 2026-05-28T09:47:58Z

I just noticed that you have a similar PR(lukel97#14). @lukel97 😄

lukel97 · 2026-05-28T09:50:10Z

I just noticed that you have a similar PR(lukel97#14). @lukel97 😄

Haha yes, great minds think alike :) That was with github copilot, and I think both our LLMs made the same PR. But I don't think it's correct because we can't lower the mask semantics of vp.load, since there's no masked variant of vlm.m?

I'm pretty sure the cost model returns an invalid cost for vp.loads of i1 types, but there's a fold somewhere else that's turning an i8 vp.load and truncate into an illegal vp.load of i1

wangpc-pp · 2026-05-28T10:04:48Z

I just noticed that you have a similar PR(lukel97#14). @lukel97 😄

Haha yes, great minds think alike :) That was with github copilot, and I think both our LLMs made the same PR. But I don't think it's correct because we can't lower the mask semantics of vp.load, since there's no masked variant of vlm.m?

I'm pretty sure the cost model returns an invalid cost for vp.loads of i1 types, but there's a fold somewhere else that's turning an i8 vp.load and truncate into an illegal vp.load of i1

Oh you are right! I should check it before committing. Yeah, I noticed the InstCombine part as well. Since you are looking at it now, I will close this PR and switch to other items.

lukel97 · 2026-05-28T10:06:28Z

I just noticed that you have a similar PR(lukel97#14). @lukel97 😄

Haha yes, great minds think alike :) That was with github copilot, and I think both our LLMs made the same PR. But I don't think it's correct because we can't lower the mask semantics of vp.load, since there's no masked variant of vlm.m?
I'm pretty sure the cost model returns an invalid cost for vp.loads of i1 types, but there's a fold somewhere else that's turning an i8 vp.load and truncate into an illegal vp.load of i1

Oh you are right! I should check it before committing. Yeah, I noticed the InstCombine part as well. Since you are looking at it now, I will close this PR and switch to other items.

I'm not actively looking at it, just steering copilot. Feel free to also let TRAE investigate it!

llvmorg-github-actions Bot added the backend:RISC-V label May 28, 2026

wangpc-pp requested review from lukel97 and topperc May 28, 2026 09:46

clang-format

bb3b48f

wangpc-pp closed this May 28, 2026

wangpc-pp deleted the main-riscv-vp-load-mask branch May 28, 2026 10:04

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[RISCV] Custom lower VP_LOAD/VP_STORE of mask vectors#200146

[RISCV] Custom lower VP_LOAD/VP_STORE of mask vectors#200146
wangpc-pp wants to merge 2 commits into
llvm:mainfrom
wangpc-pp:main-riscv-vp-load-mask

wangpc-pp commented May 28, 2026 •

edited

Loading

Uh oh!

llvmorg-github-actions Bot commented May 28, 2026

Uh oh!

github-actions Bot commented May 28, 2026 •

edited

Loading

Uh oh!

wangpc-pp commented May 28, 2026

Uh oh!

lukel97 commented May 28, 2026 •

edited

Loading

Uh oh!

wangpc-pp commented May 28, 2026

Uh oh!

lukel97 commented May 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

wangpc-pp commented May 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

llvmorg-github-actions Bot commented May 28, 2026

Uh oh!

github-actions Bot commented May 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wangpc-pp commented May 28, 2026

Uh oh!

lukel97 commented May 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wangpc-pp commented May 28, 2026

Uh oh!

lukel97 commented May 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

wangpc-pp commented May 28, 2026 •

edited

Loading

github-actions Bot commented May 28, 2026 •

edited

Loading

lukel97 commented May 28, 2026 •

edited

Loading