Promote STS minReadySeconds to beta #2824

ravisantoshgudimetla · 2021-07-14T20:14:08Z

One-line PR description:
Promote STS minReadySeconds to beta

Issue link: Add minReadySeconds to Statefulsets #2599

Other comments:

ravisantoshgudimetla · 2021-07-14T20:14:32Z

/sig apps

@soltysh

soltysh

/lgtm
/approve

soltysh · 2021-08-19T16:24:40Z

@ravisantoshgudimetla actually one nit, please update https://github.com/kubernetes/enhancements/blob/master/keps/sig-apps/2599-minreadyseconds-for-statefulsets/kep.yaml to reflect latest milestone and sate

soltysh · 2021-08-19T16:34:31Z

keps/prod-readiness/sig-apps/2599.yaml

@@ -1,3 +1,30 @@
 kep-number: 2599
 alpha:
  approver: "@ehashman"
+owning-sig: sig-apps


I think these changes are not necessary, not for PRR metadata file.

TY. Updated :)

soltysh

/lgtm

keps/sig-apps/2599-minreadyseconds-for-statefulsets/README.md

deads2k · 2021-08-30T20:52:14Z

keps/sig-apps/2599-minreadyseconds-for-statefulsets/README.md

@@ -438,6 +446,8 @@ Ideally, this should be a metric. Operations against the Kubernetes API (e.g.,
 checking if there are objects with field X set) may be a last resort. Avoid
 logs or events for this purpose.
 -->
+By checking the StatefulSets's `.status.AvailableReplicas` field. If that field is populated


isn't this set in cases when minreadyseconds==0?

they could count how many statefulsets are doing that directly, right?

Ohh you mean, I should set the value to be > 0?

keps/sig-apps/2599-minreadyseconds-for-statefulsets/README.md

deads2k · 2021-08-30T20:56:55Z

keps/sig-apps/2599-minreadyseconds-for-statefulsets/README.md

@@ -463,6 +473,7 @@ high level (needs more precise definitions) those may be things like:
    job creation time) for cron job <= 10%
  - 99,9% of /health requests per day finish with 200 code
 -->
+All the `Available` pods created should be more than the time specified in `.spec.minReadySeconds` 99.99% of the time.


how can a cluster-admin know this is happening? Or more generally, if the statefulset controller starts failing, is there a metric that tracks errors from the statefulset controller that could be alerted upon if it starts failing very often?

Based on the above metric.

It's not clear how to track this from the description of the above metric. Please be more specific.

Furthermore, I don't think this SLO is appropriate. There's nothing in the KEP that suggests this is engineered for (or that we can accurately measure) 99.99% correctness.

I think it's reasonable to think about expected latencies and correctness here, but 99.99% is a very stringent target and this SLO as written doesn't seem universal.

You may want to take a look at the updated question template: https://github.com/kubernetes/enhancements/blame/master/keps/NNNN-kep-template/README.md#L512-L513

I changed the SLO to 99% instead of 4 9's. I also added a sentence in the previous question on how the metric can be used to determine correctness of the functionality.

gracenng · 2021-09-02T04:12:24Z

Hi there 👋 1.23 enhancement shadow here.

PRR has not been approved for this enhancement stage and the implementable portion has not been checked in README.md. The current status of the enhancement is at risk. Enhancement freeze is Sept 9th.

Thanks

ehashman

Thanks @deads2k for taking a first pass at this while I was out of the office. I have some questions/feedback, the PRR questionnaire as filled out is lacking some detail.

ehashman · 2021-09-08T18:46:31Z

keps/sig-apps/2599-minreadyseconds-for-statefulsets/README.md


 ###### What specific metrics should inform a rollback?

 <!--
 What signals should users be paying attention to when the feature is young
 that might indicate a serious problem?
 -->
+`minReadySeconds` in StatefulSet doesn't get respected and all the `Ready` pods would be shown as `Available`. 


This is not a metric; how does one measure this?

By looking at the logs. I tried explaining it below or should we always use a metric? I thought looking at the logs is a signal that cluster-admin or developer can use to check if the feature is working or not.

ehashman · 2021-09-08T18:47:12Z

keps/sig-apps/2599-minreadyseconds-for-statefulsets/README.md

+the number of StatefulSets without `AvailableReplicas` growing overtime which can be used by
+cluster-admin to track th failures. We also a metric called `kube_statefulset_status_replicas_available`
+which we added recently to track the number of available replicas. The cluster-admin could use
+this metric to track the problems.


How would they use this metric? What are acceptable and unacceptable values for it?

ehashman · 2021-09-08T18:49:54Z

keps/sig-apps/2599-minreadyseconds-for-statefulsets/README.md

@@ -438,19 +450,20 @@ Ideally, this should be a metric. Operations against the Kubernetes API (e.g.,
 checking if there are objects with field X set) may be a last resort. Avoid
 logs or events for this purpose.
 -->
+By checking the `kube_statefulset_status_replicas_available` metric.


So if the metric exists, the feature is in use? Please specify what the user needs to check.

ehashman · 2021-09-08T18:50:29Z

keps/sig-apps/2599-minreadyseconds-for-statefulsets/README.md

-  - Details:
+  - Components exposing the metric: StatefulSet controller via kube_state_metrics
+
+The `kube_statefulset_status_replicas_available` gives the number of replicas available.


Can you give a little more detail on the metric labels? Will a timeseries exist for every workload that this is enabled for?

Yes. We use labels to identify the individual workload and timeseries would exist for it.

ehashman · 2021-09-08T18:56:26Z

keps/sig-apps/2599-minreadyseconds-for-statefulsets/README.md

@@ -463,6 +473,7 @@ high level (needs more precise definitions) those may be things like:
    job creation time) for cron job <= 10%
  - 99,9% of /health requests per day finish with 200 code
 -->
+All the `Available` pods created should be more than the time specified in `.spec.minReadySeconds` 99.99% of the time.


It's not clear how to track this from the description of the above metric. Please be more specific.

Furthermore, I don't think this SLO is appropriate. There's nothing in the KEP that suggests this is engineered for (or that we can accurately measure) 99.99% correctness.

I think it's reasonable to think about expected latencies and correctness here, but 99.99% is a very stringent target and this SLO as written doesn't seem universal.

You may want to take a look at the updated question template: https://github.com/kubernetes/enhancements/blame/master/keps/NNNN-kep-template/README.md#L512-L513

ehashman · 2021-09-08T18:57:18Z

keps/sig-apps/2599-minreadyseconds-for-statefulsets/README.md

@@ -589,6 +604,10 @@ details). For now, we leave it here.

 ###### How does this feature react if the API server and/or etcd is unavailable?

+ This feature will not work if the API server or etcd is unavailable as the controller-manager won't be even able get events or updates for StatefulSets. 


Does this feature have a dependency on events? They are lossy, that doesn't seem right.

None. I meant the update/create events not the events that individual components create.

ehashman · 2021-09-08T18:57:49Z

keps/sig-apps/2599-minreadyseconds-for-statefulsets/README.md

@@ -589,6 +604,10 @@ details). For now, we leave it here.

 ###### How does this feature react if the API server and/or etcd is unavailable?

+ This feature will not work if the API server or etcd is unavailable as the controller-manager won't be even able get events or updates for StatefulSets. 
+ If the API server and/or etcd is unavailable during the mid-rollout, the featuregate may be enabled but it won't have any effect on the StatefulSet as
+ the controller-manager cannot communicate with the API server


This also doesn't sound right; if the field is present on the object that the controller manager is working on, why would this stop working?

It needs to do an update on the status subresource of the STS object which is not possible without communicating with kube-apiserver

I suggest simple statement like:

The controller won't be able to progress, all currently queued resources are re-queued. This feature does not change current behavior of the controller in this regard.

ravisantoshgudimetla

Thank you for the review @ehashman. I addressed your comments and added few more sentences which can clarify things. PTAL

ravisantoshgudimetla · 2021-09-08T20:12:11Z

keps/sig-apps/2599-minreadyseconds-for-statefulsets/README.md


 ###### What specific metrics should inform a rollback?

 <!--
 What signals should users be paying attention to when the feature is young
 that might indicate a serious problem?
 -->
+`minReadySeconds` in StatefulSet doesn't get respected and all the `Ready` pods would be shown as `Available`. 


By looking at the logs. I tried explaining it below or should we always use a metric? I thought looking at the logs is a signal that cluster-admin or developer can use to check if the feature is working or not.

ravisantoshgudimetla · 2021-09-08T20:29:09Z

keps/sig-apps/2599-minreadyseconds-for-statefulsets/README.md

-  - Details:
+  - Components exposing the metric: StatefulSet controller via kube_state_metrics
+
+The `kube_statefulset_status_replicas_available` gives the number of replicas available.


Yes. We use labels to identify the individual workload and timeseries would exist for it.

ravisantoshgudimetla · 2021-09-08T20:40:37Z

keps/sig-apps/2599-minreadyseconds-for-statefulsets/README.md

@@ -589,6 +604,10 @@ details). For now, we leave it here.

 ###### How does this feature react if the API server and/or etcd is unavailable?

+ This feature will not work if the API server or etcd is unavailable as the controller-manager won't be even able get events or updates for StatefulSets. 
+ If the API server and/or etcd is unavailable during the mid-rollout, the featuregate may be enabled but it won't have any effect on the StatefulSet as
+ the controller-manager cannot communicate with the API server


It needs to do an update on the status subresource of the STS object which is not possible without communicating with kube-apiserver

ravisantoshgudimetla · 2021-09-08T20:41:18Z

keps/sig-apps/2599-minreadyseconds-for-statefulsets/README.md

@@ -589,6 +604,10 @@ details). For now, we leave it here.

 ###### How does this feature react if the API server and/or etcd is unavailable?

+ This feature will not work if the API server or etcd is unavailable as the controller-manager won't be even able get events or updates for StatefulSets. 


None. I meant the update/create events not the events that individual components create.

ravisantoshgudimetla · 2021-09-08T20:42:16Z

keps/sig-apps/2599-minreadyseconds-for-statefulsets/README.md

@@ -463,6 +473,7 @@ high level (needs more precise definitions) those may be things like:
    job creation time) for cron job <= 10%
  - 99,9% of /health requests per day finish with 200 code
 -->
+All the `Available` pods created should be more than the time specified in `.spec.minReadySeconds` 99.99% of the time.


I changed the SLO to 99% instead of 4 9's. I also added a sentence in the previous question on how the metric can be used to determine correctness of the functionality.

soltysh · 2021-09-09T10:29:54Z

keps/sig-apps/2599-minreadyseconds-for-statefulsets/README.md

+the number of StatefulSets without `AvailableReplicas` growing overtime which can be used by
+cluster-admin to track th failures. We also have a metric called `kube_statefulset_status_replicas_available`
+which we added recently to track the number of available replicas. The cluster-admin could use
+this metric to track the problems. If the value is immediately equal to the value of `Ready` replicas or if it is `0`, it can be considered as a feature failure.


@ravisantoshgudimetla let's focus just on the metrics, it'll be simpler and the question goal is to help cluster-admin identify problems with the feature (when enabled) and not necessarily regular users.

Removed it.

soltysh · 2021-09-09T10:32:42Z

keps/sig-apps/2599-minreadyseconds-for-statefulsets/README.md

-  - Components exposing the metric:
- [ ] Other (treat as last resort)
-  - Details:
+  - Components exposing the metric: StatefulSet controller via kube_state_metrics


kube-controller-manager is the component

soltysh · 2021-09-09T10:38:34Z

keps/sig-apps/2599-minreadyseconds-for-statefulsets/README.md

- [ ] Metrics
-  - Metric name:
+- [x] Metrics
+  - Metric name: `kube_statefulset_status_replicas_available`


Maybe availability ratio will be a better option for this one?

I think we can get the same ratio number using ready and available metrics. I don't have a strong opinion one or the other. Having said that, the current metrics are coming from kube_state_metrics and if we decide to go with this metric, we have to add the same metric for all the controllers. We can also revisit this during implementation.

soltysh · 2021-09-09T10:41:11Z

keps/sig-apps/2599-minreadyseconds-for-statefulsets/README.md

@@ -589,6 +604,10 @@ details). For now, we leave it here.

 ###### How does this feature react if the API server and/or etcd is unavailable?

+ This feature will not work if the API server or etcd is unavailable as the controller-manager won't be even able get events or updates for StatefulSets. 
+ If the API server and/or etcd is unavailable during the mid-rollout, the featuregate may be enabled but it won't have any effect on the StatefulSet as
+ the controller-manager cannot communicate with the API server


I suggest simple statement like:

The controller won't be able to progress, all currently queued resources are re-queued. This feature does not change current behavior of the controller in this regard.

ravisantoshgudimetla

Thank you for the feedback @soltysh. I included the changes you suggested. PTAL

ravisantoshgudimetla · 2021-09-09T12:39:15Z

keps/sig-apps/2599-minreadyseconds-for-statefulsets/README.md

+the number of StatefulSets without `AvailableReplicas` growing overtime which can be used by
+cluster-admin to track th failures. We also have a metric called `kube_statefulset_status_replicas_available`
+which we added recently to track the number of available replicas. The cluster-admin could use
+this metric to track the problems. If the value is immediately equal to the value of `Ready` replicas or if it is `0`, it can be considered as a feature failure.


Removed it.

ravisantoshgudimetla · 2021-09-09T12:41:37Z

keps/sig-apps/2599-minreadyseconds-for-statefulsets/README.md

- [ ] Metrics
-  - Metric name:
+- [x] Metrics
+  - Metric name: `kube_statefulset_status_replicas_available`


I think we can get the same ratio number using ready and available metrics. I don't have a strong opinion one or the other. Having said that, the current metrics are coming from kube_state_metrics and if we decide to go with this metric, we have to add the same metric for all the controllers. We can also revisit this during implementation.

keps/sig-apps/2599-minreadyseconds-for-statefulsets/README.md

soltysh

/lgtm
/approve

ehashman

/approve
for PRR

k8s-ci-robot · 2021-09-09T16:20:31Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ehashman, ravisantoshgudimetla, soltysh

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~keps/prod-readiness/OWNERS~~ [ehashman]
~~keps/sig-apps/OWNERS~~ [soltysh]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Jul 14, 2021

k8s-ci-robot requested review from kow3ns and soltysh July 14, 2021 20:14

k8s-ci-robot added kind/kep Categorizes KEP tracking issues and PRs modifying the KEP directory sig/apps Categorizes an issue or PR as relevant to SIG Apps. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Jul 14, 2021

ravisantoshgudimetla mentioned this pull request Jul 30, 2021

Promote min ready sec sts beta kubernetes/kubernetes#104045

Merged

soltysh approved these changes Aug 19, 2021

View reviewed changes

k8s-ci-robot assigned soltysh Aug 19, 2021

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 19, 2021

soltysh reviewed Aug 19, 2021

View reviewed changes

ravisantoshgudimetla force-pushed the add-minReadySeconds-beta branch from 2616d1a to 07b3177 Compare August 19, 2021 16:44

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 19, 2021

ravisantoshgudimetla force-pushed the add-minReadySeconds-beta branch 3 times, most recently from b611fb8 to 4d51bc1 Compare August 19, 2021 16:48

soltysh approved these changes Aug 19, 2021

View reviewed changes

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 19, 2021

deads2k reviewed Aug 30, 2021

View reviewed changes

keps/sig-apps/2599-minreadyseconds-for-statefulsets/README.md Outdated Show resolved Hide resolved

deads2k reviewed Aug 30, 2021

View reviewed changes

keps/sig-apps/2599-minreadyseconds-for-statefulsets/README.md Show resolved Hide resolved

deads2k reviewed Aug 30, 2021

View reviewed changes

keps/sig-apps/2599-minreadyseconds-for-statefulsets/README.md Show resolved Hide resolved

deads2k reviewed Aug 30, 2021

View reviewed changes

ravisantoshgudimetla force-pushed the add-minReadySeconds-beta branch from 4d51bc1 to 2eefb8e Compare September 1, 2021 12:59

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Sep 1, 2021

ravisantoshgudimetla force-pushed the add-minReadySeconds-beta branch from 2eefb8e to ff6c453 Compare September 1, 2021 13:07

ehashman requested changes Sep 8, 2021

View reviewed changes

ravisantoshgudimetla force-pushed the add-minReadySeconds-beta branch from ff6c453 to d7322b3 Compare September 8, 2021 20:43

ravisantoshgudimetla commented Sep 8, 2021

View reviewed changes

soltysh reviewed Sep 9, 2021

View reviewed changes

ravisantoshgudimetla force-pushed the add-minReadySeconds-beta branch from d7322b3 to ef9d1a8 Compare September 9, 2021 12:36

ravisantoshgudimetla commented Sep 9, 2021

View reviewed changes

soltysh reviewed Sep 9, 2021

View reviewed changes

keps/sig-apps/2599-minreadyseconds-for-statefulsets/README.md Show resolved Hide resolved

Promote STS minReadySeconds to beta

70800c1

ravisantoshgudimetla force-pushed the add-minReadySeconds-beta branch from ef9d1a8 to 70800c1 Compare September 9, 2021 13:56

soltysh approved these changes Sep 9, 2021

View reviewed changes

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Sep 9, 2021

ehashman approved these changes Sep 9, 2021

View reviewed changes

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Sep 9, 2021

k8s-ci-robot merged commit 851990a into kubernetes:master Sep 9, 2021

k8s-ci-robot added this to the v1.23 milestone Sep 9, 2021

ravisantoshgudimetla mentioned this pull request Nov 8, 2021

Add minReadySeconds to Statefulsets #2599

Closed

12 tasks

		@@ -589,6 +604,10 @@ details). For now, we leave it here.

		###### How does this feature react if the API server and/or etcd is unavailable?

		This feature will not work if the API server or etcd is unavailable as the controller-manager won't be even able get events or updates for StatefulSets.

Promote STS minReadySeconds to beta #2824

Promote STS minReadySeconds to beta #2824

Conversation

ravisantoshgudimetla commented Jul 14, 2021

ravisantoshgudimetla commented Jul 14, 2021

soltysh left a comment

Choose a reason for hiding this comment

soltysh commented Aug 19, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

soltysh left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gracenng commented Sep 2, 2021

ehashman left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ravisantoshgudimetla left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ravisantoshgudimetla left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

soltysh left a comment

Choose a reason for hiding this comment

ehashman left a comment

Choose a reason for hiding this comment

k8s-ci-robot commented Sep 9, 2021