Serve watch without resourceVersion from cache and introduce a WatchFromStorageWithoutResourceVersion feature gate to allow serving watch from storage #123935

serathius · 2024-03-14T14:31:52Z

/kind bug

With discovery of etcd watch correctness under the extreme cases of watch congestion (#123072 (comment)), #123532 is no longer a sufficient mitigation.

Even with watch cache protected, direct watches to etcd will still suffer loss of events and cause etcd memory blot. Rolling forward with improvements to etcd watch like etcd-io/etcd#17555 might be more riskly as will cause watch compaction affect new code paths.

Serving watches directly from etcd was reintroduced in 1.27 as a simple correctness fix #115096, however we didn't anticipate such big impact on stability and correctness of watch.

I propose to revert the #115096 and leave a WatchFromStorageWithoutResourceVersion feature flag to get the old behavior. At least for as long we implement ConsistentWatchFromCache as part of KEP-2340

kube-apiserver: fixes a 1.27+ regression in watch stability by serving watch requests without a resourceVersion from the watch cache by default, as in <1.27 (disabling the change in #115096 by default). This mitigates the impact of an etcd watch bug (https://github.com/etcd-io/etcd/pull/17555). If the 1.27 change in #115096 to serve these requests from underlying storage is still desired despite the impact on watch stability, it can be re-enabled with a `WatchFromStorageWithoutResourceVersion` feature gate.

staging/src/k8s.io/apiserver/pkg/features/kube_features.go

staging/src/k8s.io/apiserver/pkg/storage/cacher/cacher_whitebox_test.go

dims · 2024-03-14T14:55:56Z

@serathius @liggitt we want this in 1.30 and backports all the way back to 1.27 right?

dims · 2024-03-14T14:56:20Z

/priority critical-urgent

dims · 2024-03-14T14:56:35Z

/sig etcd

liggitt · 2024-03-14T14:58:18Z

@serathius @liggitt we want this in 1.30 and backports all the way back to 1.27 right?

yup :-/

staging/src/k8s.io/apiserver/pkg/features/kube_features.go

serathius · 2024-03-17T12:26:06Z

/retest

serathius · 2024-03-17T14:08:31Z

Yey! Got through test issues! cc @liggitt

Note; I have updated getWatchCacheResourceVersion and it's test to make sure it passes and treated TestWatchSemantics test as authoritative to Watch behavior.

wojtek-t · 2024-03-18T08:57:15Z

staging/src/k8s.io/apiserver/pkg/storage/cacher/cacher.go

 func (c *Cacher) getWatchCacheResourceVersion(ctx context.Context, parsedWatchResourceVersion uint64, opts storage.ListOptions) (uint64, error) {
 	if len(opts.ResourceVersion) != 0 {
 		return parsedWatchResourceVersion, nil
 	}
+	// legacy case
+	if !utilfeature.DefaultFeatureGate.Enabled(features.WatchFromStorageWithoutResourceVersion) && opts.SendInitialEvents == nil && opts.ResourceVersion == "" {


Why do we add it here?

If we're already at this point, we will be serving watch from cache. So the problem we're facing [being watches on etcd overloading etcd] is not a problem here. I don't think this is needed here - I would revert changes to this function.

The refactors for WatchList have changed the default behavior of watch cache, that was not a problem was previously this response was served from etcd, however to pass the tests (WatchSemantics and conformance test) I need to update this function.

Reason is that getWatchCacheResourceVersion returns a ResourceVersion to which the watch cache must be synchronized to. Without this change getWatchCacheResourceVersion will call GetCurrentResourceVersionFromStorage which is meant for consistent read, and wait for such revision.

With my change I will return zero, which means serve the data available from cache, which is consistent with the old behavior.

OK - we will need to update tests when we will switch FG default, but that's fine.

So the tests should effectively be [if FG set, then 100, otherwise 0].
So maybe please add a short comment for those tests that when we switch FG, those should be reverted to 100.

Added the comment.

I have a hard time following what this bit does, but at least the 1.27 / 1.28 / 1.29 backports won't need this and will be a straight gating of the etcd pass-through, right?

yes - this bit will not be part of backports

wojtek-t · 2024-03-18T08:59:22Z

/hold

staging/src/k8s.io/apiserver/pkg/storage/cacher/cacher.go

…romStorageWithoutResourceVersion feature gate to allow serving watch from storage.

wojtek-t · 2024-03-18T11:00:36Z

/lgtm
/approve

/hold cancel

k8s-ci-robot · 2024-03-18T11:00:43Z

LGTM label has been added.

Git tree hash: 1ee9a24abc8f29c9648098950fee946f000bcadb

k8s-ci-robot · 2024-03-18T11:00:55Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: hzxuzhonghu, liggitt, serathius, wojtek-t

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~pkg/features/OWNERS~~ [liggitt,wojtek-t]
~~staging/src/k8s.io/apiserver/pkg/features/OWNERS~~ [liggitt,wojtek-t]
~~staging/src/k8s.io/apiserver/pkg/storage/OWNERS~~ [liggitt,wojtek-t]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

serathius · 2024-03-18T12:03:21Z

/retest

liggitt · 2024-03-19T11:54:06Z

staging/src/k8s.io/apiserver/pkg/storage/cacher/cacher_test.go

+		defer featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.WatchFromStorageWithoutResourceVersion, false)()
+		store, terminate := testSetupWithEtcdAndCreateWrapper(t)
+		t.Cleanup(terminate)
+		storagetesting.RunWatchSemantics(context.TODO(), t, store)


I think this doubling of the RunWatchSemantics test is responsible for the increased timeout rate in https://testgrid.k8s.io/sig-release-master-blocking#ci-kubernetes-unit&width=20 and #123850

…1.29 Cherry-pick of #120897 #123935 #123887 #123994: Serve watch without resourceVersion from cache and introduce a WatchFromStorageWithoutResourceVersion feature gate to allow serving watch from storage.

…1.28 Cherry-pick of #123935: Serve watch without resourceVersion from cache and introduce a WatchFromStorageWithoutResourceVersion feature gate to allow serving watch from storage.

…1.27 Cherry-pick of #123935: Serve watch without resourceVersion from cache and introduce a WatchFromStorageWithoutResourceVersion feature gate to allow serving watch from storage.

serathius force-pushed the consistent-watch-from-etcd branch from a89f73c to e74151b Compare March 14, 2024 14:32

liggitt reviewed Mar 14, 2024

View reviewed changes

staging/src/k8s.io/apiserver/pkg/features/kube_features.go Outdated Show resolved Hide resolved

liggitt reviewed Mar 14, 2024

View reviewed changes

staging/src/k8s.io/apiserver/pkg/features/kube_features.go Outdated Show resolved Hide resolved

k8s-ci-robot requested review from hzxuzhonghu and MadhavJivrajani March 14, 2024 14:37

k8s-ci-robot added area/apiserver sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. and removed do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Mar 14, 2024

liggitt reviewed Mar 14, 2024

View reviewed changes

staging/src/k8s.io/apiserver/pkg/storage/cacher/cacher_whitebox_test.go Outdated Show resolved Hide resolved

serathius force-pushed the consistent-watch-from-etcd branch from e74151b to c586bca Compare March 14, 2024 14:53

k8s-ci-robot added priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. and removed needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels Mar 14, 2024

k8s-ci-robot added the sig/etcd Categorizes an issue or PR as relevant to SIG Etcd. label Mar 14, 2024

serathius force-pushed the consistent-watch-from-etcd branch from c586bca to 206b10f Compare March 14, 2024 14:58

serathius changed the title ~~Allow escape hatch to disable direct watches to etcd~~ Introduce a ConsistentWatchFromStorage feature gate, enabled by default, that can be used to disable direct watches to etcd Mar 14, 2024

serathius force-pushed the consistent-watch-from-etcd branch from 206b10f to c13ddd4 Compare March 14, 2024 15:03

serathius changed the title ~~Introduce a ConsistentWatchFromStorage feature gate, enabled by default, that can be used to disable direct watches to etcd~~ Introduce a WatchFromCacheWithoutResourceVersion feature gate, that can be used to disable direct watches to storage. Mar 14, 2024

liggitt reviewed Mar 14, 2024

View reviewed changes

staging/src/k8s.io/apiserver/pkg/features/kube_features.go Outdated Show resolved Hide resolved

serathius force-pushed the consistent-watch-from-etcd branch from c13ddd4 to dde396d Compare March 14, 2024 15:24

serathius force-pushed the consistent-watch-from-etcd branch 2 times, most recently from 0e722fc to b692ace Compare March 17, 2024 12:27

wojtek-t reviewed Mar 18, 2024

View reviewed changes

serathius force-pushed the consistent-watch-from-etcd branch from b692ace to efb2abd Compare March 18, 2024 10:45

wojtek-t reviewed Mar 18, 2024

View reviewed changes

staging/src/k8s.io/apiserver/pkg/storage/cacher/cacher.go Outdated Show resolved Hide resolved

Serve watch without resourceVersion from cache and introduce a WatchF…

0130072

…romStorageWithoutResourceVersion feature gate to allow serving watch from storage.

serathius force-pushed the consistent-watch-from-etcd branch from efb2abd to 0130072 Compare March 18, 2024 10:55

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Mar 18, 2024

k8s-ci-robot assigned wojtek-t Mar 18, 2024

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Mar 18, 2024

k8s-ci-robot merged commit a1605fb into kubernetes:master Mar 18, 2024
15 checks passed

serathius mentioned this pull request Mar 18, 2024

Cherry-pick of #120897 #123935 #123887 #123994: Serve watch without resourceVersion from cache and introduce a WatchFromStorageWithoutResourceVersion feature gate to allow serving watch from storage. #123973

Merged

p0lyn0mial mentioned this pull request Mar 19, 2024

[Flaking Test] UT k8s.io/apiserver/pkg/storage cacher #123850

Closed

liggitt reviewed Mar 19, 2024

View reviewed changes

tooptoop4 mentioned this pull request Apr 11, 2024

Controller: invalid config map object received in config watcher. Ignored processing with CPU spike argoproj/argo-workflows#11657

Open

3 tasks

shyamjvs mentioned this pull request May 20, 2024

KEP-4578: Server Feature Gate in etcd kubernetes/enhancements#4610

Merged

Shubham82 mentioned this pull request Jul 25, 2024

Add feature gate WatchFromStorageWithoutResourceVersion for v1.30 kubernetes/website#47260

Merged

serathius mentioned this pull request Oct 18, 2024

Creating deployment received etcd error event #121285

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Serve watch without resourceVersion from cache and introduce a WatchFromStorageWithoutResourceVersion feature gate to allow serving watch from storage #123935

Serve watch without resourceVersion from cache and introduce a WatchFromStorageWithoutResourceVersion feature gate to allow serving watch from storage #123935

serathius commented Mar 14, 2024 •

edited by liggitt

Loading

dims commented Mar 14, 2024

dims commented Mar 14, 2024

dims commented Mar 14, 2024

liggitt commented Mar 14, 2024

serathius commented Mar 17, 2024

serathius commented Mar 17, 2024

wojtek-t Mar 18, 2024

serathius Mar 18, 2024

wojtek-t Mar 18, 2024

serathius Mar 18, 2024

liggitt Mar 18, 2024

wojtek-t Mar 19, 2024

wojtek-t commented Mar 18, 2024

wojtek-t commented Mar 18, 2024

k8s-ci-robot commented Mar 18, 2024

k8s-ci-robot commented Mar 18, 2024

serathius commented Mar 18, 2024

liggitt Mar 19, 2024 •

edited

Loading

Serve watch without resourceVersion from cache and introduce a WatchFromStorageWithoutResourceVersion feature gate to allow serving watch from storage #123935

Serve watch without resourceVersion from cache and introduce a WatchFromStorageWithoutResourceVersion feature gate to allow serving watch from storage #123935

Conversation

serathius commented Mar 14, 2024 • edited by liggitt Loading

dims commented Mar 14, 2024

dims commented Mar 14, 2024

dims commented Mar 14, 2024

liggitt commented Mar 14, 2024

serathius commented Mar 17, 2024

serathius commented Mar 17, 2024

wojtek-t Mar 18, 2024

Choose a reason for hiding this comment

serathius Mar 18, 2024

Choose a reason for hiding this comment

wojtek-t Mar 18, 2024

Choose a reason for hiding this comment

serathius Mar 18, 2024

Choose a reason for hiding this comment

liggitt Mar 18, 2024

Choose a reason for hiding this comment

wojtek-t Mar 19, 2024

Choose a reason for hiding this comment

wojtek-t commented Mar 18, 2024

wojtek-t commented Mar 18, 2024

k8s-ci-robot commented Mar 18, 2024

k8s-ci-robot commented Mar 18, 2024

serathius commented Mar 18, 2024

liggitt Mar 19, 2024 • edited Loading

Choose a reason for hiding this comment

serathius commented Mar 14, 2024 •

edited by liggitt

Loading

liggitt Mar 19, 2024 •

edited

Loading