Add v1beta1 API #532

alculquicondor · 2023-01-26T21:11:21Z

What type of PR is this?

/kind feature

/kind api-change

What this PR does / why we need it:

Adds a v1beta1 API with changes summarized in this doc https://docs.google.com/document/d/1Uu4hfGxux4Wh_laqZMLxXdEVdty06Sb2DwB035hj700/edit?usp=sharing&resourcekey=0-b7mU7mGPCkEfhjyYDsXOBg (join https://groups.google.com/a/kubernetes.io/g/wg-batch to access)

Which issue(s) this PR fixes:

Ref #23

Special notes for your reviewer:

The first commit contains a pure copy of the v1alpha2 API objects, to facilitate reviewing the diff.

This PR keeps the storage version as v1alpha2 so that tests continue passing. I plan to remove in a follow up PR, along with updating the implementation of the controllers.

k8s-ci-robot · 2023-01-26T21:11:26Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: alculquicondor

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [alculquicondor]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

tenzen-y · 2023-01-26T21:34:12Z

/cc

mwielgus · 2023-01-28T00:21:01Z

apis/kueue/v1beta1/clusterqueue_types.go

+
+type ResourceGroup struct {
+	// resources is the list of resources covered by the flavors in this group.
+	Resources []corev1.ResourceName `json:"resources"`


The field has the same name as the one in FlavorQuota but a different type. Maybe one of them should have a different name? It looks a bit weird in yaml.

I think that's ok. They are in different contexts.

+1 with @mwielgus

ahg-g

any changes compared to v1alpha1 other than clusterqueue_types.go and the queue-name label?

ahg-g · 2023-01-30T02:55:46Z

apis/kueue/v1beta1/clusterqueue_types.go

+	// If the ClusterQueue belongs to a cohort, the sum of the quotas for each
+	// (flavor, resource) combination defines the maximum quantity that can be
+	// allocated by a ClusterQueue in the cohort.
+	Quota resource.Quantity `json:"quota"`


Following up on @liggitt suggestion, we can use AssuredLimit for absolute value, and in the future we can add AssuredLimitShares or AssuredLimitPercentage (depending on the semantics we want to enable) when we build hierarchal quota.

The source for this suggestion was API priority and fairness.

In alpha they used AssuredConcurrencyShares https://github.com/kubernetes/kubernetes/blob/f97d14c6c88e92cb505f8a9147294705a93247e3/staging/src/k8s.io/api/flowcontrol/v1alpha1/types.go#L437

But in beta they are using NominalConcurrencyShares https://github.com/kubernetes/kubernetes/blob/f97d14c6c88e92cb505f8a9147294705a93247e3/staging/src/k8s.io/api/flowcontrol/v1beta3/types.go#L471

Definition of nominal:

stated or expressed but not necessarily corresponding exactly to the real value.

It matches our intent, don't you think?

apis/kueue/v1beta1/clusterqueue_types.go

apis/kueue/v1beta1/localqueue_types.go

apis/kueue/v1beta1/clusterqueue_types.go

mwielgus · 2023-01-30T13:48:12Z

apis/kueue/v1beta1/clusterqueue_types.go

+
+	// Total is the total quantity of the resource used, including resources
+	// borrowed from the cohort.
+	Total resource.Quantity `json:"total,omitempty"`


Here seems to be a bit of inconsistency. In spec we have base + borrowed. Here is total + borrowed.

That is a good point, but I think it's rather useful:
If an administrator considers increasing the quota, it's useful for them to directly see the total usage.
And observing the borrowed quota is useful for adjusting borrowingLimit.

mwielgus · 2023-01-30T13:49:05Z

apis/kueue/v1beta1/clusterqueue_types.go

+	// name of the resource
+	Name corev1.ResourceName `json:"name"`
+
+	// Total is the total quantity of the resource used, including resources


What about resources borrowed BY the cohort?

We cannot quantify those. A ClusterQueue doesn't directly borrow from another ClusterQueue. They borrow from the pool of unused resources in the Cohort.

This is so that a single Workload can use borrowed quota from multiple CQs at the same time.

Well, we kind-of do. Imagine the situation when all capacity in the cohort is taken and the queue itself has no admitted workloads. Then borrowedByCohort = base queue quota. If All queues have low usage, within their quota then borrowedByCohort = 0. Right :)? So we can generalize it to:

max(0, SumOfUsageInOtherQueuesInCohort - SumOfQuotaInOtherQuesesInCohort)

Good point, but it can be added later. Not sure how much worth it is. We don't maintain those numbers in the cache. We just calculate them when we snapshot the cache to run the admission cycle.

apis/kueue/v1beta1/localqueue_types.go

mwielgus · 2023-01-30T13:52:50Z

apis/kueue/v1beta1/resourceflavor_types.go

+type ResourceFlavor struct {
+	metav1.TypeMeta   `json:",inline"`
+	metav1.ObjectMeta `json:"metadata,omitempty"`
+


Why no Spec and Status?

So far we haven't found a case for Status, so we skipped it for simplicity. Can you think of something?

This is similar to how PriorityClasses are defined https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption/#priorityclass

I think that we will have a problem IF we find a use case for status (like resource depleted notification or so). Without Spec and Status we will have to do another api version, deprecation, migration etc just to have that bit of info exposed. Empty Status doesn't cost us much at this moment, but may be very handy in the future.

I guess we don't need to add a status field, but we can add the spec. See update.

apis/kueue/v1beta1/workload_types.go

apis/kueue/v1beta1/clusterqueue_types.go

tenzen-y · 2023-01-31T07:22:03Z

apis/kueue/v1beta1/clusterqueue_types.go

+	// resourceGroups describes groups of resources.
+	// Each resource group defines the list of resources and a list of flavors
+	// that provide quotas for these resources.
+	// resourceGroups can be up to 16.
+	// +listType=atomic
+	// +kubebuilder:validation:MaxItems=16


Can we mention that it does not guarantee the actual availability of resources?
Moreover, can we add examples, the same as v1alpha2?

I rephrased a little the definition of nominalQuota below

The word, nominal sounds good.
Thanks for the updates!

apis/kueue/v1beta1/clusterqueue_types.go

apis/kueue/v1beta1/constants.go

tenzen-y · 2023-01-31T07:51:41Z

apis/kueue/v1beta1/clusterqueue_types.go

+//+kubebuilder:printcolumn:name="Strategy",JSONPath=".spec.queueingStrategy",type=string,description="The queueing strategy used to prioritize workloads",priority=1
+//+kubebuilder:printcolumn:name="Pending Workloads",JSONPath=".status.pendingWorkloads",type=integer,description="Number of pending workloads"
+//+kubebuilder:printcolumn:name="Admitted Workloads",JSONPath=".status.admittedWorkloads",type=integer,description="Number of admitted workloads that haven't finished yet",priority=1
+//+kubebuilder:storageversion


Do you plan to add the conversion webhooks?

Most likely not. But that's part of the justification of going to beta: that from now on we would have a conversion webhook if there are breaking changes from v1beta1 to v1beta2 or v1.
But if you know of users running in prod, it would be good to have their feedback.

Most likely not. But that's part of the justification of going to beta: that from now on we would have a conversion webhook if there are breaking changes from v1beta1 to v1beta2 or v1.

It sounds good to me.

But if you know of users running in prod, it would be good to have their feedback.

I haven't known of users running in production, ever.

alculquicondor · 2023-02-02T20:03:26Z

/retest

tenzen-y · 2023-02-02T22:13:53Z

apis/kueue/v1beta1/constants.go

+package v1beta1
+
+const (
+	ResourceInUseFinalizerName = "kueue.k8s.io/resource-in-use"


Is this domain correct?
Probably, kueue.x-k8s.io/resource-in-use?

Ah, this is my misunderstanding.
Since the API will be graduated to beta, kueue.k8s.io/resource-in-use is correct.

We'll stay in x-k8s. We have consulted with @liggitt in any case, to make sure we are following best practices in our first beta release.

Sounds good. Thanks for letting me know.

"kueue.k8s.io/resource-in-use" or"kueue.x-k8s.io/resource-in-use"?

I swear I had changed it before :|

It should be kueue.x-k8s.io. Fixed.

mwielgus · 2023-02-02T23:57:30Z

apis/kueue/v1beta1/resourceflavor_types.go

+
+// ResourceFlavor is the Schema for the resourceflavors API.
+type ResourceFlavor struct {
+	metav1.TypeMeta   `json:",inline"`


Can we have a switch to turn off a resource flavor in all of the queues (possibly as a follow up PR)?

That's beyond the scope of this PR and it deserves it's own kep with stories.

tenzen-y

Also, how do reviewers feel about merging this as is and have a separate PR for swapping the controllers into the new APIs?

I'm ok.

tenzen-y · 2023-02-07T17:09:14Z

apis/kueue/v1beta1/clusterqueue_types.go

+	// If the ClusterQueue belongs to a cohort, the sum of the quotas for each
+	// (flavor, resource) combination defines the maximum quantity that can be
+	// allocated by a ClusterQueue in the cohort.
+	NominalQuota resource.Quantity `json:"quota"`


Suggested change

NominalQuota resource.Quantity `json:"quota"`

NominalQuota resource.Quantity `json:"nominalQuota"`

tenzen-y · 2023-02-07T17:13:36Z

apis/kueue/v1beta1/clusterqueue_types.go

+	// cohort. Only quota from [resource, flavor] pairs listed in the CQ can be
+	// borrowed.


Suggested change

// cohort. Only quota from [resource, flavor] pairs listed in the CQ can be

// borrowed.

// cohort. Only borrowingLimit from [resources, flavors] pairs listed in the CQ can be

// borrowed.

The word quota was intended. The documentation for how much quota is allowed to be borrowed is left for the description of the borrowingLimit field itself.

It sounds good to me.

netlify · 2023-02-07T18:34:42Z

✅ Deploy Preview for kubernetes-sigs-kueue canceled.

Name	Link
🔨 Latest commit	`6136eed`
🔍 Latest deploy log	https://app.netlify.com/sites/kubernetes-sigs-kueue/deploys/64022527101eb900082c18e7

tenzen-y · 2023-02-07T18:47:13Z

config/components/crd/bases/kueue.x-k8s.io_clusterqueues.yaml

-    shortNames:
-    - cq


Is the removal of shortName intended?

yeah.... too cryptic

It sounds good to me.
In this case, we may need to remove kubebuiler markers from v1alpha1 API.

tenzen-y · 2023-02-07T18:49:13Z

config/components/crd/bases/kueue.x-k8s.io_resourceflavors.yaml

@@ -13,7 +13,8 @@ spec:
    listKind: ResourceFlavorList
    plural: resourceflavors
    shortNames:
-    - rf


Is the removal of shortName intended?

same, too cryptic. flavor is better

tenzen-y · 2023-02-07T18:51:56Z

Generally, LGTM.

apis/kueue/v1beta1/clusterqueue_types.go

kerthcet · 2023-02-07T10:30:02Z

apis/kueue/v1beta1/clusterqueue_types.go

+
+type ResourceGroup struct {
+	// resources is the list of resources covered by the flavors in this group.
+	Resources []corev1.ResourceName `json:"resources"`


+1 with @mwielgus

kerthcet · 2023-02-07T10:58:10Z

apis/kueue/v1beta1/clusterqueue_types.go

+	// There could be up to 16 resources.
+	// +listType=map
+	// +listMapKey=name
+	// +kubebuilder:validation:MaxItems=16


What about MinItems here?

Yes, good catch, added.

apis/kueue/v1beta1/clusterqueue_types.go

kerthcet · 2023-02-08T07:12:10Z

apis/kueue/v1beta1/clusterqueue_types.go

+	// combination that this ClusterQueue is allowed to borrow from the unused
+	// quota of other ClusterQueues in the same cohort.


Suggested change

// combination that this ClusterQueue is allowed to borrow from the unused

// quota of other ClusterQueues in the same cohort.

// combination that this ClusterQueue is allowed to borrow from the unused

// quota of other ClusterQueues in the same cohort, but not guaranteed.

Not sure... I already mentioned this is unused quota. Or are you referring to something else? But please check the update for your previous comment.

kerthcet · 2023-02-08T07:42:36Z

apis/kueue/v1beta1/clusterqueue_types.go

+	ClusterQueueActive string = "Active"
+)
+
+type Usage struct {


Where we used this? Didn't find them.

I forgot to delete it :)

kerthcet · 2023-02-08T07:45:52Z

apis/kueue/v1beta1/constants.go

+package v1beta1
+
+const (
+	ResourceInUseFinalizerName = "kueue.k8s.io/resource-in-use"


"kueue.k8s.io/resource-in-use" or"kueue.x-k8s.io/resource-in-use"?

kerthcet · 2023-02-08T08:25:59Z

apis/kueue/v1beta1/workload_types.go

+
+	// template is the Pod template.
+	//
+	// The only allowed fields in template.metadata are labels and annotations.


Do you mean we'll validate this in Kueue? Or we'll ignore other fields directly.

Yes, we'll validate in Kueue.

kerthcet · 2023-02-08T09:03:19Z

apis/kueue/v1beta1/clusterqueue_types.go

+	// allocated by a ClusterQueue in the cohort.
+	NominalQuota resource.Quantity `json:"nominalQuota"`
+
+	// borrowingLimit is the maximum amount of quota for the [flavor, resource]


Maybe describe that MaximumQuota = NominalQuota + BorrowingLimit is more intuitive.

See update.

kerthcet · 2023-02-08T09:06:39Z

pkg/constants/constants.go

@@ -19,10 +19,13 @@ package constants
 import "time"

 const (
-	// QueueAnnotation is the annotation in the workload that holds the queue name.
+	// QueueLabel is the label key in the workload that holds the queue name.


Suggested change

// QueueLabel is the label key in the workload that holds the queue name.

// QueueLabel is the label key in the job that holds the queue name.

uhm... not sure. Others Job APIs should use the same label.

alculquicondor · 2023-02-08T16:27:11Z

/retest

alculquicondor · 2023-02-08T20:36:09Z

It looks like the CI was not liking having different shortnames across versions.

mimowo · 2023-02-09T08:06:43Z

apis/kueue/v1beta1/clusterqueue_types.go

+	//
+	// +kubebuilder:default=Never
+	// +kubebuilder:validation:Enum=Never;LowerPriority;Any
+	ReclaimWithinCohort PreemptionPolicy `json:"withinCohort,omitempty"`


Suggested change

ReclaimWithinCohort PreemptionPolicy `json:"withinCohort,omitempty"`

WithinCohort PreemptionPolicy `json:"withinCohort,omitempty"`

In order to align the struct name and the json field name.

Oops, this was a mistake in #514. It should be reclaimWithinCohort

mimowo · 2023-02-09T08:08:38Z

apis/kueue/v1beta1/workload_types.go

+	//
+	// The type of the condition could be:
+	//
+	// - Admitted: the Workload was admitted through a ClusterQueue.


Extend the list with PodsReady for consistency.

apis/kueue/v1alpha2/resourceflavor_types.go

ahg-g · 2023-02-08T22:06:05Z

apis/kueue/v1beta1/clusterqueue_types.go

+	// Gatekeeper should be used to enforce more advanced policies.
+	// Defaults to null which is a nothing selector (no namespaces eligible).
+	// If set to an empty selector `{}`, then all namespaces are eligible.
+	NamespaceSelector *metav1.LabelSelector `json:"namespaceSelector,omitempty"`


I wonder if we can disambiguate this by naming it allowedNamespcesSelector?

I don't anticipate other kind of namespaceSelector, so I don't see a strong benefit. There is also precedence in the NetworkPolicy https://kubernetes.io/docs/concepts/services-networking/network-policies/#networkpolicy-resource

The confusion I heard once was that this field selects the Jobs submitted to the queue. It would be better if the name coveys that this is about policy.

@liggitt any thoughts on naming here?

how does a namespace submit workloads to the cluster queue? what happens if a workload in a namespace not matched by this selector tries to use this cluster queue?

A namespace submits workloads to a LocalQueue. A LocalQueue references a ClusterQueue.

If the selector doesn't match, the Workload gets a Condition indicating that the Workload is not admitted because it doesn't match the namespaceSelector.

This is a basic form of policy enforcement, ideally the cluster admin sets up something like gatekeeper, but in most cases gatekeeper is an overkill.

kerthcet · 2023-02-10T03:01:25Z

apis/kueue/v1beta1/clusterqueue_types.go

+	// The list cannot be empty and it can contain up to 16 resources.
+	// +kubebuilder:validation:MinItems=1
+	// +kubebuilder:validation:MaxItems=16
+	CoveredResources []corev1.ResourceName `json:"coveredResources"`


I read the design docs https://docs.google.com/document/d/1Uu4hfGxux4Wh_laqZMLxXdEVdty06Sb2DwB035hj700/edit?resourcekey=0-b7mU7mGPCkEfhjyYDsXOBg#heading=h.dkobprho4lxe and I got the intension about adding a slice of resource names as a visual aid, just want to make sure we all agree on this. cc @ahg-g

In addition to being a visual aid, it makes the API more explicit, which is a good principle.

But I agree that it might add burden, specially for resource groups with a single resource. But, on the other hand, ClusterQueues should be APIs that don't change often.

kerthcet · 2023-02-10T03:16:10Z

apis/kueue/v1beta1/clusterqueue_types.go

+	Resources []ResourceUsage `json:"resources"`
+}
+
+type ResourceUsage struct {


Oh, that means borrowed, not borrowing. my brain....

kerthcet · 2023-02-13T03:05:40Z

Thanks @alculquicondor for the great work, LGTMed.

alculquicondor · 2023-02-14T16:58:18Z

Thank you all
I think we are pretty much settled now.
I have been working in transitioning the controllers to the new APIs. I'll keep this PR open (on hold) just in case I find some limitations in the implementation.

tenzen-y · 2023-02-14T17:02:43Z

Thank you all I think we are pretty much settled now. I have been working in transitioning the controllers to the new APIs. I'll keep this PR open (on hold) just in case I find some limitations in the implementation.

Sounds good. I'm looking forward to kueue v1beta1 :)
If you would help from others, feel free to send a ping to me.

kerthcet · 2023-02-23T09:40:16Z

pkg/constants/constants.go

@@ -19,10 +19,13 @@ package constants
 import "time"

 const (
-	// QueueAnnotation is the annotation in the workload that holds the queue name.
+	// QueueLabel is the label key in the workload that holds the queue name.
+	QueueLabel = "kueue.x-k8s.io/queue-name"


Revisit this label, I'm reading the api designs about MatchFields https://github.com/kubernetes/kubernetes/blob/fc8b5668f506b6d71ddc5243a21f5d0c5f8271d3/pkg/apis/core/types.go#L2558-L2563

This field was designed specially for daemonSet pod scheduling, one reason why we avoid to reuse MatchExpressions is because we can't model label as non-label usages(take daemonset for example, we're finding the exactly node not a set of nodes), in this case, it should be an unique localQueue name, not a set of queue names. Annotation is something we think not bad as an experimental feature, and considering we haven't populate queueName to job yet.

Are you saying that we should stay with an annotation instead of a label?

I'm not understanding what you mean by a unique localQueue name. Multiple workloads can point to the same queue.

I mean a workload can only belong to one localQueue, we're taking the name field into labels,, which seems like we're abusing the label.

This label is for the (custom) Job objects, not the Workloads. We call them workloads in general. Maybe we should call them jobs? not sure.

Change-Id: Ic4cc56f07e8ef5caa7b55ac02901a54708c35f20

Change-Id: I9a12e50eaa66c1897406b44e59f9e0813382645d

alculquicondor · 2023-03-08T15:46:46Z

/close
in favor of #604

k8s-ci-robot · 2023-03-08T15:47:59Z

@alculquicondor: Closed this PR.

In response to this:

/close
in favor of #604

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-ci-robot requested review from ArangoGutierrez and kerthcet January 26, 2023 21:11

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. approved Indicates a PR has been approved by an approver from all required OWNERS files. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Jan 26, 2023

alculquicondor force-pushed the v1beta1 branch from 2db8d86 to ff72832 Compare January 26, 2023 21:24

k8s-ci-robot requested a review from tenzen-y January 26, 2023 21:34

mwielgus reviewed Jan 28, 2023

View reviewed changes

ahg-g reviewed Jan 30, 2023

View reviewed changes

mwielgus reviewed Jan 30, 2023

View reviewed changes

apis/kueue/v1beta1/localqueue_types.go Show resolved Hide resolved

mwielgus reviewed Jan 30, 2023

View reviewed changes

apis/kueue/v1beta1/clusterqueue_types.go Outdated Show resolved Hide resolved

mwielgus reviewed Jan 30, 2023

View reviewed changes

apis/kueue/v1beta1/localqueue_types.go Show resolved Hide resolved

mwielgus reviewed Jan 30, 2023

View reviewed changes

apis/kueue/v1beta1/workload_types.go Outdated Show resolved Hide resolved

tenzen-y reviewed Jan 31, 2023

View reviewed changes

alculquicondor force-pushed the v1beta1 branch 2 times, most recently from b1751f2 to a1669ff Compare February 2, 2023 18:58

alculquicondor mentioned this pull request Feb 2, 2023

Graduate API to beta #23

Closed

4 tasks

alculquicondor changed the title ~~WIP Add v1beta1 API~~ Add v1beta1 API Feb 2, 2023

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Feb 2, 2023

tenzen-y reviewed Feb 2, 2023

View reviewed changes

mwielgus reviewed Feb 2, 2023

View reviewed changes

tenzen-y reviewed Feb 7, 2023

View reviewed changes

alculquicondor force-pushed the v1beta1 branch from 33cc687 to ab31e4f Compare February 7, 2023 18:39

tenzen-y reviewed Feb 7, 2023

View reviewed changes

kerthcet reviewed Feb 8, 2023

View reviewed changes

mimowo reviewed Feb 9, 2023

View reviewed changes

ahg-g reviewed Feb 9, 2023

View reviewed changes

kerthcet reviewed Feb 10, 2023

View reviewed changes

alculquicondor mentioned this pull request Feb 14, 2023

Adapt names to reviewed Kueue documentation GoogleCloudPlatform/kubernetes-engine-samples#493

Merged

rbarberop mentioned this pull request Feb 17, 2023

samples: Kueue cohort tutorial samples GoogleCloudPlatform/kubernetes-engine-samples#498

Closed

alculquicondor force-pushed the v1beta1 branch from 2d464a4 to c1f7f44 Compare February 21, 2023 19:53

kerthcet reviewed Feb 23, 2023

View reviewed changes

alculquicondor mentioned this pull request Feb 27, 2023

WaitForPodsReady: Requeue at the back of the queue after timeout #599

Closed

3 tasks

alculquicondor force-pushed the v1beta1 branch from c1f7f44 to 798c35d Compare March 2, 2023 22:09

alculquicondor mentioned this pull request Mar 2, 2023

New controller-runtime broke support for FieldOwner in status updates #601

Closed

alculquicondor force-pushed the v1beta1 branch from 798c35d to 24ffa19 Compare March 3, 2023 16:07

alculquicondor added 2 commits March 3, 2023 11:49

Copy v1alpha2 into v1beta1

4e3fff4

Change-Id: Ic4cc56f07e8ef5caa7b55ac02901a54708c35f20

v1beta1 API update proposals

6136eed

Change-Id: I9a12e50eaa66c1897406b44e59f9e0813382645d

alculquicondor force-pushed the v1beta1 branch from 24ffa19 to 6136eed Compare March 3, 2023 16:49

alculquicondor mentioned this pull request Mar 3, 2023

Migrate controllers to v1beta1 #604

Merged

k8s-ci-robot closed this Mar 8, 2023

tenzen-y mentioned this pull request Sep 7, 2023

KEP 973 workload-Priority #1081

Merged

	NominalQuota resource.Quantity `json:"quota"`
	NominalQuota resource.Quantity `json:"nominalQuota"`

		// cohort. Only quota from [resource, flavor] pairs listed in the CQ can be
		// borrowed.

		// combination that this ClusterQueue is allowed to borrow from the unused
		// quota of other ClusterQueues in the same cohort.

	// QueueLabel is the label key in the workload that holds the queue name.
	// QueueLabel is the label key in the job that holds the queue name.

	ReclaimWithinCohort PreemptionPolicy `json:"withinCohort,omitempty"`
	WithinCohort PreemptionPolicy `json:"withinCohort,omitempty"`

Add v1beta1 API #532

Add v1beta1 API #532

Conversation

alculquicondor commented Jan 26, 2023 • edited Loading

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

k8s-ci-robot commented Jan 26, 2023

tenzen-y commented Jan 26, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ahg-g left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alculquicondor Jan 31, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alculquicondor commented Feb 2, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tenzen-y left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

netlify bot commented Feb 7, 2023 • edited Loading

✅ Deploy Preview for kubernetes-sigs-kueue canceled.

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tenzen-y commented Feb 7, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alculquicondor commented Feb 8, 2023

alculquicondor commented Feb 8, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alculquicondor commented Jan 26, 2023 •

edited

Loading

alculquicondor Jan 31, 2023 •

edited

Loading

netlify bot commented Feb 7, 2023 •

edited

Loading