Skip to content

[VPlan] Split out EVL exit cond transform from canonicalizeEVLLoops. NFC #178181

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
lukel97 merged 6 commits into from
Feb 2, 2026
Merged

[VPlan] Split out EVL exit cond transform from canonicalizeEVLLoops. NFC #178181

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
753bdde
[VPlan] Split out EVL exit cond transform from canonicalizeEVLLoops. NFC
lukel97 Jan 27, 2026
0a49ca0
Address review comments
lukel97 Jan 29, 2026
f8952aa
Verify EVL IV is immediately after canonical IV
lukel97 Jan 30, 2026
c0e3ae7
Move EVL verification to verifyPhiRecipes
lukel97 Feb 2, 2026
0edc12b
Update doc comment
lukel97 Feb 2, 2026
26e6cb2
Remove redundant null canIV check
lukel97 Feb 2, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7430,6 +7430,8 @@ DenseMap<const SCEV *, Value *> LoopVectorizationPlanner::executePlan(
// Expand BranchOnTwoConds after dissolution, when latch has direct access to
// its successors.
VPlanTransforms::expandBranchOnTwoConds(BestVPlan);
// Switch the exit condition to AVLNext == 0 for EVL tail folded loops.

@fhahn fhahn Jan 29, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: transform/convert may be more in line with owrding elswhere

Suggested change
// Switch the exit condition to AVLNext == 0 for EVL tail folded loops.
// Convert the exit condition to AVLNext == 0 for EVL tail folded loops.

@lukel97 lukel97 Jan 29, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed, thanks

VPlanTransforms::convertEVLExitCond(BestVPlan);
// Canonicalize EVL loops after regions are dissolved.
VPlanTransforms::canonicalizeEVLLoops(BestVPlan);
VPlanTransforms::materializeBackedgeTakenCount(BestVPlan, VectorPH);
Expand Down
67 changes: 40 additions & 27 deletions llvm/lib/Transforms/Vectorize/VPlanTransforms.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3247,9 +3247,9 @@ void VPlanTransforms::addExplicitVectorLength(
Plan.setUF(1);
}

void VPlanTransforms::canonicalizeEVLLoops(VPlan &Plan) {
// Find EVL loop entries by locating VPEVLBasedIVPHIRecipe.
// There should be only one EVL PHI in the entire plan.
/// Find EVL loop entries by locating VPEVLBasedIVPHIRecipe.
/// There should be only one EVL PHI in the entire plan.

@fhahn fhahn Jan 29, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just to note that now we need to traverse the full VPlan twice if there's no EVL recipe.

would be good if there would a good way to exit early one we can't find a EVL recipe in the header if possible. There's getFirstLoopHeader which works on plain CFG but would require VPDT, which is likely not worth it unless already available.

I think later we assert that the first phi in EVL's parent is a canonical IV, via below. Maybe we could try to check this early, and bail out once we found a block with a canonical IV-like but not a VPEVLBasedIVPHIRecipe recipes. Not sure if we may be able to check if the increment of the phi is used by the expected branch-on-cond.

 [[maybe_unused]] auto *CanIV = cast<VPPhi>(&*EVLPhi->getParent()->begin());
  assert(
      match(LatchExitingBr,
            m_BranchOnCond(m_SpecificCmp(
                CmpInst::ICMP_EQ, m_Specific(CanIV->getIncomingValue(1)),
                m_Specific(&Plan.getVectorTripCount())))) &&

@lukel97 lukel97 Jan 29, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that you mention it, we can just do the branch condition transform before the loop regions are dissolved and use getCanonicalIV, the EVL IV will immediately follow it if it exists 0a49ca0

@fhahn fhahn Jan 29, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

moving makes sense, thanks

static VPEVLBasedIVPHIRecipe *findEVLPhi(VPlan &Plan) {
VPEVLBasedIVPHIRecipe *EVLPhi = nullptr;

for (VPBasicBlock *VPBB : VPBlockUtils::blocksOnly<VPBasicBlock>(
Expand All @@ -3260,28 +3260,17 @@ void VPlanTransforms::canonicalizeEVLLoops(VPlan &Plan) {
EVLPhi = PhiR;
}

return EVLPhi;
}

void VPlanTransforms::canonicalizeEVLLoops(VPlan &Plan) {
VPEVLBasedIVPHIRecipe *EVLPhi = findEVLPhi(Plan);
// Early return if no EVL PHI is found.
if (!EVLPhi)
return;

VPBasicBlock *HeaderVPBB = EVLPhi->getParent();
VPValue *EVLIncrement = EVLPhi->getBackedgeValue();
VPValue *AVL;
[[maybe_unused]] bool FoundAVL =
match(EVLIncrement,
m_c_Add(m_ZExtOrSelf(m_EVL(m_VPValue(AVL))), m_Specific(EVLPhi)));
assert(FoundAVL && "Didn't find AVL?");

// The AVL may be capped to a safe distance.
VPValue *SafeAVL;
if (match(AVL, m_Select(m_VPValue(), m_VPValue(SafeAVL), m_VPValue())))
AVL = SafeAVL;

VPValue *AVLNext;
[[maybe_unused]] bool FoundAVLNext =
match(AVL, m_VPInstruction<Instruction::PHI>(
m_Specific(Plan.getTripCount()), m_VPValue(AVLNext)));
assert(FoundAVLNext && "Didn't find AVL backedge?");

// Convert EVLPhi to concrete recipe.
auto *ScalarR =
Expand All @@ -3302,21 +3291,45 @@ void VPlanTransforms::canonicalizeEVLLoops(VPlan &Plan) {
VPRecipeBase *CanonicalIVIncrement = Backedge->getDefiningRecipe();
CanonicalIVIncrement->eraseFromParent();
CanonicalIV->eraseFromParent();
}

// Replace the use of VectorTripCount in the latch-exiting block.
// Before: (branch-on-cond (icmp eq EVLIVInc, VectorTripCount))
// After: (branch-on-cond icmp eq AVLNext, 0)
void VPlanTransforms::convertEVLExitCond(VPlan &Plan) {
VPEVLBasedIVPHIRecipe *EVLPhi = findEVLPhi(Plan);
if (!EVLPhi)
return;

// Bail if not an EVL tail folded loop.
VPValue *AVL;
if (!match(EVLPhi->getBackedgeValue(),
m_c_Add(m_ZExtOrSelf(m_EVL(m_VPValue(AVL))), m_Specific(EVLPhi))))
return;

// The AVL may be capped to a safe distance.
VPValue *SafeAVL;
if (match(AVL, m_Select(m_VPValue(), m_VPValue(SafeAVL), m_VPValue())))

@fhahn fhahn Jan 29, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not related to the patch itself, but it might be good to think if we can match something more precise here, in case some transform gets applied to the original select

@lukel97 lukel97 Jan 30, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I think we can make this more explicit. Will try to do in a follow up.

@fhahn fhahn Jan 30, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great thanks!

AVL = SafeAVL;

VPValue *AVLNext;
[[maybe_unused]] bool FoundAVLNext =
match(AVL, m_VPInstruction<Instruction::PHI>(
m_Specific(Plan.getTripCount()), m_VPValue(AVLNext)));
assert(FoundAVLNext && "Didn't find AVL backedge?");

VPBasicBlock *HeaderVPBB = EVLPhi->getParent();
VPBasicBlock *LatchExiting =
HeaderVPBB->getPredecessors()[1]->getEntryBasicBlock();
auto *LatchExitingBr = cast<VPInstruction>(LatchExiting->getTerminator());
if (match(LatchExitingBr, m_BranchOnCond(m_True())))
return;

assert(match(LatchExitingBr, m_BranchOnCond(m_SpecificCmp(
CmpInst::ICMP_EQ, m_VPValue(EVLIncrement),

@lukel97 lukel97 Jan 27, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This also fixes this m_VPValue, it should be a m_Specific

m_Specific(&Plan.getVectorTripCount())))) &&
"Expected BranchOnCond with ICmp comparing EVL increment with vector "
"trip count");
[[maybe_unused]] auto *CanIV = cast<VPPhi>(&*EVLPhi->getParent()->begin());
assert(
match(LatchExitingBr,
m_BranchOnCond(m_SpecificCmp(
CmpInst::ICMP_EQ, m_Specific(CanIV->getIncomingValue(1)),
m_Specific(&Plan.getVectorTripCount())))) &&
"Expected BranchOnCond with ICmp comparing CanIV increment with vector "
"trip count");

Type *AVLTy = VPTypeAnalysis(Plan).inferScalarType(AVLNext);
VPBuilder Builder(LatchExitingBr);
Expand Down
10 changes: 6 additions & 4 deletions llvm/lib/Transforms/Vectorize/VPlanTransforms.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -315,12 +315,14 @@ struct VPlanTransforms {
/// variable vector lengths instead of fixed lengths. This transformation:
/// * Makes EVL-Phi concrete.
// * Removes CanonicalIV and increment.
/// * Replaces the exit condition from
/// (branch-on-count CanonicalIVInc, VectorTripCount)
/// to
/// (branch-on-cond eq AVLNext, 0)
static void canonicalizeEVLLoops(VPlan &Plan);

/// Replaces the exit condition from
/// (branch-on-count CanonicalIVInc, VectorTripCount)
/// to
/// (branch-on-cond eq AVLNext, 0)
static void convertEVLExitCond(VPlan &Plan);

@fhahn fhahn Jan 29, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: there's no clear order to definitions, but convertEVLExitCond runs before canonicalizeEVLLoops, so might be worth moving first?

@lukel97 lukel97 Jan 29, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed, thanks


/// Lower abstract recipes to concrete ones, that can be codegen'd.
static void convertToConcreteRecipes(VPlan &Plan);

Expand Down