1.22 feature blog for minReadySeconds in StatefulSets #28992

jlbutler · 2021-07-17T20:39:20Z

1.22 feature blog post for "Add minReadySeconds to Statefulsets" (kubernetes/enhancements#2599).

Target publish date is 08/27/21. [edit: updated publish date]

cc: @ravisantoshgudimetla

netlify · 2021-07-17T21:23:46Z

✔️ Deploy Preview for kubernetes-io-main-staging ready!

🔨 Explore the source changes: bd68cc9

🔍 Inspect the deploy log: https://app.netlify.com/sites/kubernetes-io-main-staging/deploys/61252b1fcbf3ba00073047f7

😎 Browse the preview: https://deploy-preview-28992--kubernetes-io-main-staging.netlify.app

sftim · 2021-07-28T19:08:31Z

content/en/blog/_posts/2021-08-16-minreadysecond-statefulsets.md

+
+## What problems does this solve?
+
+Today, once a `StatefulSet` `Pod` is in the `Ready` state it is considered `Available` to receive traffic. For some of the `StatefulSet` workloads, it may not be the case. For example, a workload like Prometheus with multiple instances of Alertmanager, it should be considered `Available` only when Alertmanager's state transfer is complete, not when the `Pod` is in `Ready` state. Since `minReadySeconds` adds buffer, the state transfer may be complete before the `Pod` becomes `Available`. While this is not a fool proof way of identifying if the state transfer is complete or not, it gives a way to the end user to express their intention of waiting for sometime before the `Pod` is considered `Available` and it is ready to serve requests. 


Q. Does “today” describe the situation before the v1.22 release, or after it?

It is before v1.22

As this article goes out after the release, let's reword to make that clear.

sftim · 2021-07-28T19:09:13Z

content/en/blog/_posts/2021-08-16-minreadysecond-statefulsets.md

@@ -0,0 +1,48 @@
+---
+layout: blog
+title: 'minReadySeconds for StatefulSets'


Is there a typical format for the titles of post-release blog articles? If so, let's follow that.

I am not sure of that. Where do we have style guide?

@jlbutler ^?

Apologies I thought I'd replied to this. As I mentioned last week, I also think this could use a more reader-friendly title.

I don't know of any specific guideline but they do tend to be more descriptive of functionality, rather than calling out details.

sftim

A bit more feedback.

Please do update the article to frame it as being published after the v1.22 release. That's the most important change that needs making. The other details are less essential.

sftim · 2021-08-02T14:28:56Z

content/en/blog/_posts/2021-08-16-minreadysecond-statefulsets.md

+KEP: https://github.com/kubernetes/enhancements/issues/2599
+API Changes: https://github.com/kubernetes/kubernetes/pull/100842
+Implementation PR: https://github.com/kubernetes/kubernetes/pull/101316


Suggested change

KEP: https://github.com/kubernetes/enhancements/issues/2599

API Changes: https://github.com/kubernetes/kubernetes/pull/100842

Implementation PR: https://github.com/kubernetes/kubernetes/pull/101316

- Read the KEP: [minReadySeconds for StatefulSets](https://github.com/kubernetes/enhancements/tree/master/keps/sig-apps/2599-minreadyseconds-for-statefulsets#readme)

- Read the documentation: [Minimum ready seconds](/docs/concepts/workloads/controllers/statefulset/#minimum-ready-seconds) for StatefulSet

- Review the [API definition](/docs/reference/kubernetes-api/workload-resources/stateful-set-v1/) for StatefulSet

sftim · 2021-08-02T14:30:10Z

content/en/blog/_posts/2021-08-16-minreadysecond-statefulsets.md

+
+## How do I get involved?
+
+Please reach out to us on [#sig-apps](https://kubernetes.slack.com/archives/C18NZM5K9) channel on slack, sig-apps mailing [email protected]


Suggested change

Please reach out to us on [#sig-apps](https://kubernetes.slack.com/archives/C18NZM5K9) channel on slack, sig-apps mailing list-[email protected]

Please reach out to us in the [#sig-apps](https://kubernetes.slack.com/archives/C18NZM5K9) channel on Slack (visit https://slack.k8s.io/ for an invitation if you need one), or on the SIG Apps mailing list: [email protected]

sftim · 2021-08-02T14:32:22Z

content/en/blog/_posts/2021-08-16-minreadysecond-statefulsets.md

+The `StatefulSet` controller watches for both `StatefulSets` and the `Pods` associated with them. When the feature gate associated with this feature is enabled, the `StatefulSet` controller identifies how long a particular `Pod` associated with a `StatefulSet` has been in the `Running` state.
+
+If this value is greater than or equal to the time specified by the end user in `.spec.minReadySeconds` field, the `StatefulSet` controller updates a field called `AvailableReplicas` in the `StatefulSet`'s status subresource to include this `Pod`. The `AvailableReplicas` in `StatefulSet`'s status is a int32 field which tracks the number of pods that are `Available`.


tiny nit:

Suggested change

The `StatefulSet` controller watches for both `StatefulSets` and the `Pods` associated with them. When the feature gate associated with this feature is enabled, the `StatefulSet` controller identifies how long a particular `Pod` associated with a `StatefulSet` has been in the `Running` state.

If this value is greater than or equal to the time specified by the end user in `.spec.minReadySeconds` field, the `StatefulSet` controller updates a field called `AvailableReplicas` in the `StatefulSet`'s status subresource to include this `Pod`. The `AvailableReplicas` in `StatefulSet`'s status is a int32 field which tracks the number of pods that are `Available`.

The statefulset controller watches for both `StatefulSets` and the `Pods` associated with them. When the feature gate associated with this feature is enabled, the statefulset controller identifies how long a particular `Pod` associated with a `StatefulSet` has been in the `Running` state.

If this value is greater than or equal to the time specified by the end user in `.spec.minReadySeconds` field, the statefulset controller updates a field called `availableReplicas` in the `StatefulSet`'s status subresource to include this `Pod`. The `.status.availableReplicas` field for a `StatefulSet` is an integer field that tracks the number of pods that are `Available`.

sftim · 2021-08-04T23:02:05Z

content/en/blog/_posts/2021-08-16-minreadysecond-statefulsets.md

+---
+layout: blog
+title: 'minReadySeconds for StatefulSets'
+date: 2021-08-27


It's handy to update the path to the Markdown file to match. Hugo doesn't care but us humans can get confused.

jlbutler · 2021-08-09T17:59:58Z

@ravisantoshgudimetla can you reply to review when you have a few cycles? thanks!

sftim · 2021-08-15T18:16:52Z

@jlbutler if you're willing to make the changes needed, feel free. That'll let us review.

If you want to get signoff on those changes from @ravisantoshgudimetla before we publish, add a /hold so we don't accidentally put it live without that signoff.

ravisantoshgudimetla · 2021-08-15T22:20:22Z

I'll get to this tomorrow. Thank you for the reviews @sftim

ravisantoshgudimetla

Made the changes locally. Will push the changes to remote branch once I have access to it. The earlier collaboration invite is expired now :(

ravisantoshgudimetla

@sftim - Made the changes you suggested modulo the title formatting. I am not sure which format I should use. Let me know if the current title looks good. TY

ravisantoshgudimetla · 2021-08-17T14:37:34Z

cc @soltysh

content/en/blog/_posts/2021-08-16-minreadysecond-statefulsets.md

Co-authored-by: Simon Pasquier <[email protected]>

sftim · 2021-08-19T09:35:58Z

I think the title for this article is OK and we should focus on getting this merged (we can update the title after merging if we think of a new one that we really like).

sftim · 2021-08-19T09:39:11Z

@ravisantoshgudimetla / @soltysh - between the two of you I'm happy that you can review each other's contributions here. Feel free to add /lgtm when you're happy that this can go out.

LGTM - for writing style etc

/hold
@jlbutler are you happy to revise this to show the new publication date?

/approve

ravisantoshgudimetla · 2021-08-19T19:47:40Z

@ravisantoshgudimetla / @soltysh - between the two of you I'm happy that you can review each other's contributions here. Feel free to add /lgtm when you're happy that this can go out.

@soltysh has already reviewed it. We worked with @jlbutler before opening this PR

chrisnegus

Looks good! Just a couple of small wording suggestions.

chrisnegus · 2021-08-23T14:14:41Z

content/en/blog/_posts/2021-08-16-minreadysecond-statefulsets.md

+
+## What problems does this solve?
+
+Prior to Kubernetes 1.22 release, once a `StatefulSet` `Pod` is in the `Ready` state it is considered `Available` to receive traffic. For some of the `StatefulSet` workloads, it may not be the case. For example, a workload like Prometheus with multiple instances of Alertmanager, it should be considered `Available` only when Alertmanager's state transfer is complete, not when the `Pod` is in `Ready` state. Since `minReadySeconds` adds buffer, the state transfer may be complete before the `Pod` becomes `Available`. While this is not a fool proof way of identifying if the state transfer is complete or not, it gives a way to the end user to express their intention of waiting for sometime before the `Pod` is considered `Available` and it is ready to serve requests. 


Suggested change

Prior to Kubernetes 1.22 release, once a `StatefulSet` `Pod` is in the `Ready` state it is considered `Available` to receive traffic. For some of the `StatefulSet` workloads, it may not be the case. For example, a workload like Prometheus with multiple instances of Alertmanager, it should be considered `Available` only when Alertmanager's state transfer is complete, not when the `Pod` is in `Ready` state. Since `minReadySeconds` adds buffer, the state transfer may be complete before the `Pod` becomes `Available`. While this is not a fool proof way of identifying if the state transfer is complete or not, it gives a way to the end user to express their intention of waiting for sometime before the `Pod` is considered `Available` and it is ready to serve requests.

Prior to Kubernetes 1.22 release, once a `StatefulSet` `Pod` is in the `Ready` state it is considered `Available` to receive traffic. For some of the `StatefulSet` workloads, that may not be the case. For example, a workload like Prometheus, with multiple instances of Alertmanager, should be considered `Available` only when Alertmanager's state transfer is complete, not when the `Pod` is in `Ready` state. Since `minReadySeconds` adds buffer, the state transfer may be complete before the `Pod` becomes `Available`. While this is not a foolproof way of identifying if the state transfer is complete or not, it gives a way to the end user to express their intention of waiting for some time before the `Pod` is considered `Available` and ready to serve requests.

chrisnegus · 2021-08-23T14:20:10Z

content/en/blog/_posts/2021-08-16-minreadysecond-statefulsets.md

+
+Prior to Kubernetes 1.22 release, once a `StatefulSet` `Pod` is in the `Ready` state it is considered `Available` to receive traffic. For some of the `StatefulSet` workloads, it may not be the case. For example, a workload like Prometheus with multiple instances of Alertmanager, it should be considered `Available` only when Alertmanager's state transfer is complete, not when the `Pod` is in `Ready` state. Since `minReadySeconds` adds buffer, the state transfer may be complete before the `Pod` becomes `Available`. While this is not a fool proof way of identifying if the state transfer is complete or not, it gives a way to the end user to express their intention of waiting for sometime before the `Pod` is considered `Available` and it is ready to serve requests. 
+
+Another case, where `minReadySeconds` helps is when using `LoadBalancer` `Services` with cloud providers. Since `minReadySeconds` adds latency after a `Pod` is `Ready`, it provides buffer time to prevent killing pods in rotation before new pods show up. Imagine a load balancer in unhappy path taking 10-15s to propagate. If you have 2 replicas then, you'd kill the second replica only after the first one is up but in reality, first replica cannot be seen because it is not yet ready to serve requests.


Suggested change

Another case, where `minReadySeconds` helps is when using `LoadBalancer` `Services` with cloud providers. Since `minReadySeconds` adds latency after a `Pod` is `Ready`, it provides buffer time to prevent killing pods in rotation before new pods show up. Imagine a load balancer in unhappy path taking 10-15s to propagate. If you have 2 replicas then, you'd kill the second replica only after the first one is up but in reality, first replica cannot be seen because it is not yet ready to serve requests.

Another case where `minReadySeconds` helps is when using `LoadBalancer` `Services` with cloud providers. Since `minReadySeconds` adds latency after a `Pod` is `Ready`, it provides buffer time to prevent killing pods in rotation before new pods show up. Imagine a load balancer in an unhappy path, taking 10-15s to propagate. If you have 2 replicas, then you'd kill the second replica only after the first one is up. But in reality, the first replica cannot be seen because it is not yet ready to serve requests.

chrisnegus · 2021-08-23T14:23:04Z

content/en/blog/_posts/2021-08-16-minreadysecond-statefulsets.md

+
+After successfully starting `kube-apiserver` and `kube-controller-manager`, you will see `AvailableReplicas` in the status and `minReadySeconds` of spec (with a default value of 0). 
+
+Specify a value for `minReadySeconds` for any StatefulSet and you can check if `Pods` are available or not by checking `AvailableReplicas` field using:


Suggested change

Specify a value for `minReadySeconds` for any StatefulSet and you can check if `Pods` are available or not by checking `AvailableReplicas` field using:

Specify a value for `minReadySeconds` for any StatefulSet and you can check if `Pods` are available or not by checking the `AvailableReplicas` field using:

sftim · 2021-08-24T17:24:46Z

I pushed a commit to update the publication date.

@ravisantoshgudimetla / @soltysh once you're happy with the wording - and “right now, actually” is a valid answer - please use the /lgtm command to record that you are. If you're not able to LGTM (“looks good to me”), or aren't comfortable running that command, reply to say so and SIG Docs can sort something out.

If we get this merged and folks want to fix minor grammar or spelling nits, those can go in after the merge (and either before or after publication).

If there's no LGTM before the publication date and we miss it by more than say 18 hours, we should leave the hold in place and pick a new date.

sftim · 2021-08-24T17:25:17Z

/label tide/merge-method-squash

sftim · 2021-08-24T17:26:00Z

Oh, I should also add: if this gets LGTM before publication is due on the 27th, please do unhold it.

ravisantoshgudimetla

Thank you @jlbutler @sftim @chrisnegus for the reviews

/lgtm

k8s-ci-robot · 2021-08-24T17:38:46Z

LGTM label has been added.

Git tree hash: 52babcf11efe5ec31d963a0d4a849d363c378797

k8s-ci-robot · 2021-08-24T17:38:47Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ravisantoshgudimetla, sftim

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~content/en/blog/OWNERS~~ [sftim]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

sftim · 2021-08-24T17:51:13Z

/hold cancel
🎉

* 1.22 feature blog for minReadySeconds in StatefulSets * Address reviewer's comments * Update content/en/blog/_posts/2021-08-16-minreadysecond-statefulsets.md Co-authored-by: Simon Pasquier <[email protected]> * Bump article publication date Co-authored-by: ravisantoshgudimetla <[email protected]> Co-authored-by: Ravi Gudimetla <[email protected]> Co-authored-by: Simon Pasquier <[email protected]> Co-authored-by: Tim Bannister <[email protected]>

k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. area/blog Issues or PRs related to the Kubernetes Blog subproject labels Jul 17, 2021

k8s-ci-robot requested review from kbarnard10 and sftim July 17, 2021 20:39

k8s-ci-robot added language/en Issues or PRs related to English language sig/docs Categorizes an issue or PR as relevant to SIG Docs. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Jul 17, 2021

1.22 feature blog for minReadySeconds in StatefulSets

99483c6

k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Jul 28, 2021

jlbutler changed the title ~~[WIP] 1.22 feature blog for minReadySeconds in StatefulSets~~ 1.22 feature blog for minReadySeconds in StatefulSets Jul 28, 2021

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 28, 2021

sftim reviewed Jul 28, 2021

View reviewed changes

sftim reviewed Aug 2, 2021

View reviewed changes

sftim reviewed Aug 4, 2021

View reviewed changes

Address reviewer's comments

a9a4af8

ravisantoshgudimetla reviewed Aug 16, 2021

View reviewed changes

ravisantoshgudimetla reviewed Aug 17, 2021

View reviewed changes

simonpasquier reviewed Aug 18, 2021

View reviewed changes

content/en/blog/_posts/2021-08-16-minreadysecond-statefulsets.md Outdated Show resolved Hide resolved

Update content/en/blog/_posts/2021-08-16-minreadysecond-statefulsets.md

16c55ed

Co-authored-by: Simon Pasquier <[email protected]>

k8s-ci-robot added do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Aug 19, 2021

chrisnegus reviewed Aug 23, 2021

View reviewed changes

Bump article publication date

bd68cc9

k8s-ci-robot added the tide/merge-method-squash Denotes a PR that should be squashed by tide when it merges. label Aug 24, 2021

ravisantoshgudimetla approved these changes Aug 24, 2021

View reviewed changes

k8s-ci-robot assigned ravisantoshgudimetla Aug 24, 2021

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 24, 2021

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Aug 24, 2021

k8s-ci-robot merged commit 9924e7a into kubernetes:main Aug 24, 2021

chrisnegus mentioned this pull request Nov 10, 2021

Add minReadySeconds to Statefulsets kubernetes/enhancements#2599

Closed

12 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

1.22 feature blog for minReadySeconds in StatefulSets #28992

1.22 feature blog for minReadySeconds in StatefulSets #28992

jlbutler commented Jul 17, 2021 •

edited

Loading

netlify bot commented Jul 17, 2021 •

edited

Loading

sftim Jul 28, 2021

ravisantoshgudimetla Jul 29, 2021

sftim Jul 29, 2021

sftim Jul 28, 2021

ravisantoshgudimetla Jul 29, 2021

ravisantoshgudimetla Jul 30, 2021

jlbutler Aug 2, 2021

sftim left a comment

sftim Aug 2, 2021

sftim Aug 2, 2021

sftim Aug 2, 2021

sftim Aug 4, 2021

jlbutler commented Aug 9, 2021

sftim commented Aug 15, 2021

ravisantoshgudimetla commented Aug 15, 2021

ravisantoshgudimetla left a comment

ravisantoshgudimetla left a comment

ravisantoshgudimetla commented Aug 17, 2021

sftim commented Aug 19, 2021

sftim commented Aug 19, 2021

ravisantoshgudimetla commented Aug 19, 2021

chrisnegus left a comment

chrisnegus Aug 23, 2021

chrisnegus Aug 23, 2021

chrisnegus Aug 23, 2021

sftim commented Aug 24, 2021

sftim commented Aug 24, 2021

sftim commented Aug 24, 2021

ravisantoshgudimetla left a comment

k8s-ci-robot commented Aug 24, 2021

k8s-ci-robot commented Aug 24, 2021

sftim commented Aug 24, 2021


		## What problems does this solve?

		Today, once a `StatefulSet` `Pod` is in the `Ready` state it is considered `Available` to receive traffic. For some of the `StatefulSet` workloads, it may not be the case. For example, a workload like Prometheus with multiple instances of Alertmanager, it should be considered `Available` only when Alertmanager's state transfer is complete, not when the `Pod` is in `Ready` state. Since `minReadySeconds` adds buffer, the state transfer may be complete before the `Pod` becomes `Available`. While this is not a fool proof way of identifying if the state transfer is complete or not, it gives a way to the end user to express their intention of waiting for sometime before the `Pod` is considered `Available` and it is ready to serve requests.


		## How do I get involved?

		Please reach out to us on [#sig-apps](https://kubernetes.slack.com/archives/C18NZM5K9) channel on slack, sig-apps mailing [email protected]

	Please reach out to us on [#sig-apps](https://kubernetes.slack.com/archives/C18NZM5K9) channel on slack, sig-apps mailing list-[email protected]
	Please reach out to us in the [#sig-apps](https://kubernetes.slack.com/archives/C18NZM5K9) channel on Slack (visit https://slack.k8s.io/ for an invitation if you need one), or on the SIG Apps mailing list: [email protected]

		The `StatefulSet` controller watches for both `StatefulSets` and the `Pods` associated with them. When the feature gate associated with this feature is enabled, the `StatefulSet` controller identifies how long a particular `Pod` associated with a `StatefulSet` has been in the `Running` state.

		If this value is greater than or equal to the time specified by the end user in `.spec.minReadySeconds` field, the `StatefulSet` controller updates a field called `AvailableReplicas` in the `StatefulSet`'s status subresource to include this `Pod`. The `AvailableReplicas` in `StatefulSet`'s status is a int32 field which tracks the number of pods that are `Available`.


		## What problems does this solve?

		Prior to Kubernetes 1.22 release, once a `StatefulSet` `Pod` is in the `Ready` state it is considered `Available` to receive traffic. For some of the `StatefulSet` workloads, it may not be the case. For example, a workload like Prometheus with multiple instances of Alertmanager, it should be considered `Available` only when Alertmanager's state transfer is complete, not when the `Pod` is in `Ready` state. Since `minReadySeconds` adds buffer, the state transfer may be complete before the `Pod` becomes `Available`. While this is not a fool proof way of identifying if the state transfer is complete or not, it gives a way to the end user to express their intention of waiting for sometime before the `Pod` is considered `Available` and it is ready to serve requests.


		Prior to Kubernetes 1.22 release, once a `StatefulSet` `Pod` is in the `Ready` state it is considered `Available` to receive traffic. For some of the `StatefulSet` workloads, it may not be the case. For example, a workload like Prometheus with multiple instances of Alertmanager, it should be considered `Available` only when Alertmanager's state transfer is complete, not when the `Pod` is in `Ready` state. Since `minReadySeconds` adds buffer, the state transfer may be complete before the `Pod` becomes `Available`. While this is not a fool proof way of identifying if the state transfer is complete or not, it gives a way to the end user to express their intention of waiting for sometime before the `Pod` is considered `Available` and it is ready to serve requests.

		Another case, where `minReadySeconds` helps is when using `LoadBalancer` `Services` with cloud providers. Since `minReadySeconds` adds latency after a `Pod` is `Ready`, it provides buffer time to prevent killing pods in rotation before new pods show up. Imagine a load balancer in unhappy path taking 10-15s to propagate. If you have 2 replicas then, you'd kill the second replica only after the first one is up but in reality, first replica cannot be seen because it is not yet ready to serve requests.


		After successfully starting `kube-apiserver` and `kube-controller-manager`, you will see `AvailableReplicas` in the status and `minReadySeconds` of spec (with a default value of 0).

		Specify a value for `minReadySeconds` for any StatefulSet and you can check if `Pods` are available or not by checking `AvailableReplicas` field using:

1.22 feature blog for minReadySeconds in StatefulSets #28992

1.22 feature blog for minReadySeconds in StatefulSets #28992

Conversation

jlbutler commented Jul 17, 2021 • edited Loading

netlify bot commented Jul 17, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sftim left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jlbutler commented Aug 9, 2021

sftim commented Aug 15, 2021

ravisantoshgudimetla commented Aug 15, 2021

ravisantoshgudimetla left a comment

Choose a reason for hiding this comment

ravisantoshgudimetla left a comment

Choose a reason for hiding this comment

ravisantoshgudimetla commented Aug 17, 2021

sftim commented Aug 19, 2021

sftim commented Aug 19, 2021

ravisantoshgudimetla commented Aug 19, 2021

chrisnegus left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sftim commented Aug 24, 2021

sftim commented Aug 24, 2021

sftim commented Aug 24, 2021

ravisantoshgudimetla left a comment

Choose a reason for hiding this comment

k8s-ci-robot commented Aug 24, 2021

k8s-ci-robot commented Aug 24, 2021

sftim commented Aug 24, 2021

jlbutler commented Jul 17, 2021 •

edited

Loading

netlify bot commented Jul 17, 2021 •

edited

Loading