[KEP] Update KEP of the flavor fungibility feature by pajakd · Pull Request #6133 · kubernetes-sigs/kueue

pajakd · 2025-07-22T11:10:12Z

What type of PR is this?

/kind documentation

What this PR does / why we need it:

Which issue(s) this PR fixes:

Part of #5424

Special notes for your reviewer:

Does this PR introduce a user-facing change?

NONE

k8s-ci-robot · 2025-07-22T11:10:15Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

netlify · 2025-07-22T11:10:26Z

✅ Deploy Preview for kubernetes-sigs-kueue canceled.

Name	Link
🔨 Latest commit	`0f2d41d`
🔍 Latest deploy log	https://app.netlify.com/projects/kubernetes-sigs-kueue/deploys/68822ee39d20050008c4bae5

tenzen-y

First questions.

tenzen-y · 2025-07-22T12:52:24Z

-After we try to schedule a podset in a resource flavor, we decide whether to traverse to the next flavor base on the `flavorFungibility`. If the assignment mode is `NoFit`, we will always try the next flavor until the last one. When the assignment mode is `Preempt`, we can return the current assignment if `WhenCanPreempt` is `Preempt`. Otherwise if the assignment mode is `Fit`, we try the next flavor only when we need borrowing in the current flavor and `WhenCanBorrow` is `TryNextFlavor`.
+- (`Fit`, `NoBorrow`) is the most prefered
+- (`Fit`, `Borrow`) is prefered over (`Preempt`, `NoBorrow`) if `WhenCanPreemptAndBorrow = AvoidPreemption`
+- (`Preempt`, `NoBorrow`) is prefered over (`Fit`, `Borrow`) if `WhenCanPreemptAndBorrow = AvoidBorrowing`


(Preempt, NoBorrow) is prefered over (Fit, Borrow) if WhenCanPreemptAndBorrow = AvoidBorrowing

What does this mean? Does the whenCanPreemptAndBorrow: AvoidBorrowing respect Preempt and NoBorrow over the Fit? Or does this strategy avoid only Fit with borrow?

IIUC, Fit contains 2 patterns: (1 the workload fits nominalQuota (2 the workload fits borrowed quotas.

Right, so in my notation we have 4 interesting cases (Fit, NoBorrow), (Fit, Borrow), (Preempt, NoBorrow), (Preempt, Borrow).

WhenCanPreemptAndBorrow = PreferBorrowing (current) the order is (Fit, NoBorrow), (Fit, Borrow), (Preempt, NoBorrow), (Preempt, Borrow)

WhenCanPreemptAndBorrow = PreferPreemption (new) the order is (Fit, NoBorrow), (Preempt, NoBorrow), (Fit, Borrow), (Preempt, Borrow)

tenzen-y · 2025-07-22T12:53:34Z

  TryNextFlavor FlavorFungibilityPolicy = "TryNextFlavor"
 )

+type FlavorSelectionStrategy string


Do you have any reason to choose strategy suffix?
Could we select FlavorSelectionPolicy to align with FlavorFungibilityPolicy?

Oh, of course. Changed to FlavorSelectionStrategy

mimowo · 2025-07-22T13:01:23Z

+   AvoidPreemption FlavorSelectionStrategy = "AvoidPreemption"
+   AvoidBorrowing  FlavorSelectionStrategy = "AvoidBorrowing"


As also asked in the PR impl: #6132 (comment), I'm not convinced to the naming, the "Avoid" seems stronger than prefer. So, it is likely users would interpret like "do nothing" when all flavors are (Preempt, NoBorrow).

So I would be leaning to "PreferPreemption", "PreferBorrowing", or an enum field "preference: Preemption/Borrowing", wdyt?

I also think that new strategies should not be "Avoid" since it would be better to mention what will happen rather than what will NOT happen.

OK. So I guess more people prefer prefer :) Updated.

mimowo · 2025-07-23T11:01:21Z

+  //    other workloads.
+  // - `PreferPreemption`: prefer to allocate in a flavor that uses only the nominal quota
+  //    even if it means preempting other workloads.
+  // +kubebuilder:validation:Enum={PreferBorrowing,PreferPreemption}


Why not just Borrow,Preempt? Since the field is already whenCanPreemptAndBorrow it is already clear we have the two options.

mimowo · 2025-07-23T11:41:52Z

+  //    even if it means preempting other workloads.
+  // +kubebuilder:validation:Enum={PreferBorrowing,PreferPreemption}
+  // +kubebuilder:default="PreferBorrowing"
+  WhenCanPreemptAndBorrow FlavorSelectionPolicy `json:"preferredStrategy,omitempty"`


Actually, I find this API still confusing a lot. Its form is similar to whenCanPreempt and whenCanBorrow, but the semantics is very differenent. whenCanPreempt and whenCanBorrow indicate actually: stop looking for new flavors. They mean when flavor which preempt -> do something, when flavor which can borrow -> do something.

The naming convention suggests the new field has the same nature, when a flavor which can preempt and borrow is encountered -> do something.

However, the nature of the field is very different, it means "once the set of flavors is determined, choose the one fitting the preference: Borrow or Preempt".

cc @tenzen-y @pajakd

Let me propose something.

The API I'm thinking towards is

flavorFungability: whenCanBorrow: Stop / TryNextFlavor whenCanPreempt: Stop / TryNextFlavor preferenceOrder: ["Preempt", "PreemptWhileBorrowing", "Borrow"]

Now, the field is preferenceOrder because some users may want to position "PreemptWhileBorrowing" differently. So, by saying just "Preempt / Borrow" we are too constraining for the future.

I think what happens here is some implicit ordering currently "Borrow", "Preempt", "PreemptWhileBorrowing".

As mentioned in https://github.com/kubernetes-sigs/kueue/pull/6133/files#r2225166242 the story is really for a user which is using FairSharing, and we got feedback due to the very nature of fair sharing they want "Preempt", "PreemptWhileBorrowing", "Borrow".

However, I can imagine some users will want ["Preempt", "PreemptWhileBorrowing", "Borrow"] (say outside of FairSharing), so long term something like preferenceOrder might be helpful.

So, I propose to do not introduce new API for 0.13, just a beta level feature gate (for now), FairSharingWithFlavorFungabilityPrefersPreemption. I think this is a sensible default when FairSharing is used for the reasons explained in the Story. The feature gate will give us bailout option if some users are unhappy with the change (I think this is unlikely), yet will give us time to think of better API. I'm also ok to make the feature gate alpha for now.

Ok, let me expand on the dependency on FairSharing as it was, rightfully, questioned when discussing with @pajakd @gabesaba .

What we want with this feature is to avoid borrowing when reclaim is enabled. Because if a workload borrows it is likely a preemption target of the workloads in CQs which are entitles to reclaim.

Initially I thought, and was mistaken, that FairSharing enables all CQs in the system to reclaim (which is the common configuration), but in fact a CQ may still disabled preemption.reclaimWithinCohort.

Let me rework the feature gate idea then. I think what makes sense it still to use implicit defaults until we have a better idea of what is needed.

So my proposal is:

whenCanBorrow: Borrow & whenCanPreempt: TryNextFlavor -> implicit default to for preference to Borrow

whenCanBorrow: TryNextFlavor & whenCanPreempt: Preempt -> implicit default to for preference to Preempt

whenCanBorrow: Borrow & whenCanPreempt: Preempt -> implicit default does not matter because we just choose first

whenCanBorrow: TryNextFlavor & whenCanPreempt: TryNextFlavor -> make implicit default depending on the enablement of FairSharing. This is enough to capture the idea of FairSharing to mix borrowing with reclaim, yet will only be scoped to CQs which have whenCanBorrow: TryNextFlavor & whenCanPreempt: TryNextFlavor.

EDIT: The intention for tying the default to FairSharing (in 4.) is the guesstimate of user preferences for "Preempt". The reason is that FairSharing with reclamation disabled is rather corner case usage, so I think it is reasonable to assume users who use FairSharing will prefer "Preempt" over "Borrowing".

All of this I would hide behind feature gate FlavorFungabilityImplicitPreferenceDefault.

Once we hear from users no complains for a couple of releases (most likely scenario) we can just graduate FlavorFungabilityImplicitPreferenceDefault. If some users report issues we can think of preferenceOrder idea to override the implicit default. However, for now I think we can improve the default behavior using feature gate without expanding the API.

I don't think that we should graduate FlavorFungabilityImplicitPreferenceDefault FG to default enablement ones (beta, ga) since IIRC, pattern 2 (whenCanBorrow: TryNextFlavor & whenCanPreempt: Preempt) breaks the current behavior.

If we want to ship this behavior as a default, we should extend the API.

To base the discussion on the k8s guidelines, I think this would be it: Deprecating a feature or behavior.

In our case we are not deprecating the feature, but "behavior". I think the relevant rule would be "Rule #7: Deprecated behaviors must function for no less than 1 year after their announced deprecation.", which I'm reading: if we graduate FlavorFungabilityImplicitPreferenceDefault to Beta, then we need to keep it for 1 year in Beta to allow users to return to the old behavior. So, I think the path exists. wdyt?

Still, if you feel strongly about it, I'm ok with API, but I wouldn't like to rush with new APIs. So far all the API proposals I've seen have some drawbacks. So, I'm ok to make the feature gate as "Alpha", but with a note that we intend to replace it with an API. wdyt?

fyi: we synced on slack with @tenzen-y, and we agreed to introduce an alpha feature gate as a temporary measure to collect feedback from users.

However, going forward instead of graduating the feature gate to Beta, we will introduce an API. This will give us more time to design and discuss the API without rushing. We can add a comment to the feature gate that it is not meant for graduation and its role should be replaced with API.

Unless someone proposes an API which everyone loves, but it seems challenging. So far all proposals have some drawbacks IMO.

@mimowo Thank you for summarizing our discussion.
As Michal mentioned in the above comment, we introduce a temporary FG (FlavorFungabilityImplicitPreferenceDefault ) for early feedback users for v0,13. However, to avoid the significant specification changes (Borrow -> Preemption), we do not change the behavior as a default. So, we will revist the API consideration for the v0.14. After we define the APIs for new behavior for v0.14, we will remove the FG (FlavorFungabilityImplicitPreferenceDefault).

@gabesaba @pajakd Could you address those decisions?

Updated the implementation PR (#6132) accordingly. Now I'll update this PR to include the decisions.

mimowo · 2025-07-23T15:33:16Z

xref as KEP for #6132

@pajakd please update the descption to say "KEP for #..." or "Part of #..." , then GH would x-ref automatically

mimowo · 2025-07-24T14:11:10Z

/lgtm
/approve
I believe all comments are addressed. We can always follow up if needed.

k8s-ci-robot · 2025-07-24T14:11:18Z

LGTM label has been added.

Details

Git tree hash: 27b2ffca085aa18641a1e08976ac648dd27d09ca

k8s-ci-robot · 2025-07-24T14:11:18Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mimowo, pajakd

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [mimowo]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

) * Update kep * Avoid -> prefer * Strategy -> policy * Added a new story * WIP * Update KEP * Update story * Update story * Remove dependency on FR

Update kep

2f8a363

k8s-ci-robot requested review from mbobrovskyi and mimowo July 22, 2025 11:10

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Jul 22, 2025

pajakd mentioned this pull request Jul 22, 2025

Flavor Fungibility Prefers Fit over NoBorrow #6132

Merged

tenzen-y reviewed Jul 22, 2025

View reviewed changes

mimowo reviewed Jul 22, 2025

View reviewed changes

Comment thread keps/582-preempt-based-on-flavor-order/README.md Outdated

pajakd added 2 commits July 22, 2025 17:52

Avoid -> prefer

eec9b8b

Strategy -> policy

475daa0

mimowo reviewed Jul 23, 2025

View reviewed changes

Comment thread keps/582-preempt-based-on-flavor-order/README.md Outdated

mimowo reviewed Jul 23, 2025

View reviewed changes

Comment thread keps/582-preempt-based-on-flavor-order/README.md Outdated

mimowo reviewed Jul 23, 2025

View reviewed changes

Added a new story

1a86af6

pajakd added 2 commits July 24, 2025 09:18

WIP

8c229c2

Update KEP

78b3416

pajakd marked this pull request as ready for review July 24, 2025 10:59

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 24, 2025

k8s-ci-robot requested a review from mimowo July 24, 2025 10:59

Update story

0112cc6

pajakd added 2 commits July 24, 2025 11:25

Update story

b898cac

Remove dependency on FR

0f2d41d

k8s-ci-robot assigned mimowo Jul 24, 2025

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jul 24, 2025

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 24, 2025

k8s-ci-robot merged commit 5448908 into kubernetes-sigs:main Jul 24, 2025
8 checks passed

k8s-ci-robot added this to the v0.13 milestone Jul 24, 2025

mimowo mentioned this pull request Jul 25, 2025

FlavorFungability: replace FlavorFungibilityImplicitPreferenceDefault feature gate with API #6184

Closed

3 tasks

vladikkuzn mentioned this pull request Oct 17, 2025

[KEP] FlavorFungability: replace FlavorFungibilityImplicitPreferenceDefault feature gate with API #7317

Merged

		AvoidPreemption FlavorSelectionStrategy = "AvoidPreemption"
		AvoidBorrowing FlavorSelectionStrategy = "AvoidBorrowing"

Conversation

pajakd commented Jul 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Uh oh!

k8s-ci-robot commented Jul 22, 2025

Uh oh!

netlify Bot commented Jul 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for kubernetes-sigs-kueue canceled.

Uh oh!

tenzen-y left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pajakd Jul 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mimowo Jul 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mimowo Jul 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tenzen-y Jul 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mimowo commented Jul 23, 2025

Uh oh!

mimowo commented Jul 24, 2025

Uh oh!

k8s-ci-robot commented Jul 24, 2025

Uh oh!

k8s-ci-robot commented Jul 24, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

pajakd commented Jul 22, 2025 •

edited

Loading

netlify Bot commented Jul 22, 2025 •

edited

Loading

pajakd Jul 22, 2025 •

edited

Loading

mimowo Jul 23, 2025 •

edited

Loading

mimowo Jul 23, 2025 •

edited

Loading

tenzen-y Jul 23, 2025 •

edited

Loading