From 00d1011bf1ac20ec99131c4d8d2fe4600bcf44fd Mon Sep 17 00:00:00 2001 From: Laura Lorenz Date: Tue, 17 Sep 2024 11:02:10 -0700 Subject: [PATCH 01/36] Update proposal and risk section for 1.32 Signed-off-by: Laura Lorenz --- .../4603-tune-crashloopbackoff/README.md | 208 +++++++++--------- 1 file changed, 101 insertions(+), 107 deletions(-) diff --git a/keps/sig-node/4603-tune-crashloopbackoff/README.md b/keps/sig-node/4603-tune-crashloopbackoff/README.md index 35d5c2175ec6..7a0835f0e140 100644 --- a/keps/sig-node/4603-tune-crashloopbackoff/README.md +++ b/keps/sig-node/4603-tune-crashloopbackoff/README.md @@ -307,25 +307,24 @@ The "Design Details" section below is for the real nitty-gritty. --> -<<[UNRESOLVED changing from 1.31 proposal to 1.32 proposal, -incoming in a separate PR]>> This design seeks to incorporate a two-pronged approach: -1. Change the existing initial value for the backoff curve to stack more retries - earlier for all restarts (`restartPolicy: OnFailure` and `restartPolicy: - Always`) <<[UNRESOLVED]>> -2. Provide a option to configure even faster restarts for specific Pod/Container -(particularly sidecar containers)/Node/Cluster, regardless of exit code, -reducing the max wait to 1 minute <<[/UNRESOLVED]>> - -In addition, part of the alpha period will be dedicated entirely to - systematically stress testing kubelet and API Server with different - distributions of workloads utilizing the new backoff curves. Therefore, this -KEP requires instrumentation of enhanced visibility into pod restart behavior to -enable Kubernetes benchmarking during the alpha phase. During the benchmarking -period of alpha, kubelet memory and CPU, API server latency, and pod restart -latency will be observed and analyzed to define the maximum allowable restart -rate for fully saturated nodes. +1. Change the existing initial value for the backoff curve to 1s to stack more + retries earlier, and reduce the maximum backoff for a given restart from 5 + minutes to 30 seconds, for all restarts (`restartPolicy: OnFailure` and + `restartPolicy: Always`) +2. Provide an option to cluster operators to configure an even lower maximum +backoff for all containers on a specific Node, down to 1s + +To derive these values, manual stress testing observing the behavior of kubelet, + API server, and overall cluster operations and behavior were performed, as + described below in <<[ UNRESOLVED ifttt benchmarking: link to benchmarking results ]>> +<<[/UNRESOLVED]>>. In addition, part of the alpha period will be dedicated +entirely to systematically stress testing kubelet and API Server with different +distributions of workloads utilizing the new backoff curves. During the +benchmarking period in alpha, kubelet memory and CPU, API server latency, and +pod restart latency will be observed and analyzed to further refine the maximum +allowable restart rate for fully saturated nodes. Longer term, these metrics will also supply cluster operators the data necessary to better analyze @@ -336,27 +335,49 @@ Note that proposal will NOT change: * backoff behavior for Pods transitioning from the "Success" state differently from those transitioning from a "Failed" state -- see [here in Alternatives Considered](#on-success-and-the-10-minute-recovery-threshold) -* the time Kubernetes waits before resetting the backoff counter -- see the - [here inAlternatives - Considered](#on-success-and-the-10-minute-recovery-threshold) +<<[ UNRESOLVED ifttt late recovery ]>> * the time Kubernetes waits before resetting the backoff counter -- see the + [here in Alternatives + Considered](#on-success-and-the-10-minute-recovery-threshold) <<[/UNRESOLVED]>> * the ImagePullBackoff -- out of scope, see [Design Details](#relationship-with-imagepullbackoff) * changes that address 'late recovery', or modifications to backoff behavior once the max cap has been reached -- see [Alternatives](#more-complex-heuristics) -<<[/UNRESOLVED]>> -### Existing backoff curve change: front loaded decay +### Existing backoff curve change: front loaded decay, lower maximum backoff This KEP proposes changing the existing backoff curve to load more restarts -earlier by changing the initial value of the exponential backoff. A number of -alternate initial values are modelled below, until the 5 minute cap would be -reached. This proposal suggests we start with a new initial value of 1s, and -analyze its impact on infrastructure during alpha. +earlier by changing the initial value of the exponential backoff, and to retry +periodically more often by reducing the maximum for the backoff. A number of +alternate initial values and maximum backoff values are modelled below. This +proposal suggests we start with a new initial value of 1s (changed from 10s +today) and a maximum backoff of 30 seconds (changed from 5 minutes today) based +on prior research <<[ UNRESOLVED link to benchmarking results +]>>here<<[/UNRESOLVED]>>, and further analyze its impact on infrastructure +during alpha. + +A simple change to maximum backoff would naturally come with a modification of +the backoff counter reset threshold -- as it is currently calculated based on 2x +the max cap -- so without any other modification, containers would be "rewarded" +by having their backoff counter set to 0 for running successfully for 1 minute +(instead of for 10 minutes like it is today). <<[ UNRESOLVED ifttt late recovery: +what is better?]>>As late recovery is expressly a non-goal of this KEP, as part +of this implementation the late recovery threshold for restart backoffs will be +factored out separately and remain 10 minutes. <<[/UNRESOLVED]>> + +<<[ UNRESOLVED new pic with updated modeling re: max cap values +]>>![](todayvs1sbackoff.png)<<[ /UNRESOLVED]>> + -![](todayvs1sbackoff.png) +### Node specific kubelet config for maximum backoff down to 1 second + +This KEP also proposes providing a more flexible mechanism for modifying the +maximum backoff as an opt-in. The configuration for this will be per node, by an +integer representing seconds (notably NOT resulting in subsecond granularity). +The minimum allowable value will be 1 second. This per node configuration will +be configurable only by a user with awareness of the node holistically. ### User Stories @@ -447,15 +468,13 @@ accompanying API request, if the requests become rapid enough due to fast enough churn of Pods through CrashLoopBackoff phases, the central API server could become unresponsive, effectively taking down an entire cluster. -The same risk exists for the <<[UNRESOLVED]>> per Node <<[/UNRESOLVED]>> -feature, which, while not default, is by design a more severe reduction in the -decay behavior. It can abused by <<[UNRESOLVED]>> cluster operators -<<[/UNRESOLVED]>>, and in the worst case cause nodes to fully saturate with -<<[UNRESOLVED]>> instantly <<[/UNRESOLVED]>> restarting pods that will never -recover, risking similar issues as above: taking down nodes or at least -nontrivially slowing kubelet, or increasing the API requests to store backoff -state so significantly that the central API server is unresponsive and the -cluster fails. +The same risk exists for the per Node feature, which, while not default, is by +design allowing a more severe reduction in the decay behavior. In the worst +case, it could cause nodes to fully saturate with near-instantly restarting pods + that will never recover, risking similar issues as above: taking down nodes or +at least nontrivially slowing kubelet, or increasing the API requests to store +backoff state so significantly that the central API server is unresponsive and +the cluster fails. During alpha, naturally the first line of defense is that the enhancements, even the reduced "default" baseline curve for CrashLoopBackoff, are not usable by @@ -465,41 +484,43 @@ by each risk if the cluster operator enables the new features during the alpha period. Beyond this, there are two main mitigations during alpha: conservativism in -changes to the default behavior, and <<[UNRESOLVED]>> per-node <<[/UNRESOLVED]>> -opt-in and redeployment required for the more aggressive behavior. - -The alpha changes to the default backoff curve were chosen because they are -minimal -- <<[ UNRESOLVED]>>the proposal maintains the existing rate and max -cap, and reduces the initial value to the point that only introduces 3 excess -restarts per pod, the first 2 excess in the first 10 seconds and the last excess -following in the next 30 seconds (see [Design +changes to the default behavior based on prior stress testing, and limiting any +further overrides to be opt-in per Node, and only by users with the permissions +to modify the kubelet configuration -- in other words, a cluster operator +persona. + +The alpha changes to the _default_ backoff curve were chosen because they meet +emerging use cases and user sentiment from the canonical feature request issue +and research by the author, AND because they are indicated to be safe changes +based on initial benchmarking and analysis, described below in context and +<<[UNRESOLVED ifttt benchmarking: add benchmarking link]>>here more +broadly<<[/UNRESOLVED]>>. + +<<[UNRESOLVED fix details for new proposal]>>the proposal maintains the existing +rate of 2x, reduces the initial value to the point that introduces N excess +restarts per pod, the first XYZ excess in Y time window and the last ABC excess +following in the next Z time window (see [Design Details](#front-loaded-decay-curve-methodology)). For a hypothetical node with the max 110 pods all stuck in a simultaneous CrashLoopBackoff, API requests to -change the state transition would increase at its fastest period from ~110 -requests/10s to 330 requests/10s. <<[/UNRESOLVED]>> By passing this minimal -change through the existing SIG-scalability tests, while pursuing manual and -more detailed periodic benchmarking during the alpha period, we can increase the -confidence in this change and in the possibility of reducing further in the -future. - -For the <<[UNRESOLVED]>> per node <<[/UNRESOLVED]>> case, because the change is -more significant, including lowering the max cap, there is more risk to node -stability expected. This change is of interest to be tested in the alpha period -by end users, and is why it is still included with opt-in even though the risks -are higher. That being said it is still a relatively conservative change in an -effort to minimize the unknown changes for fast feedback during alpha, while -improved benchmarking and testing occurs. <<[UNRESOLVED update with real stress -test results]>> For a hypothetical node with the max 110 pods all stuck in a -simultaneous `Rapid` CrashLoopBackoff, API requests to change the state -transition would increase from ~110 requests/10s to 440 requests/10s, and since -the max cap would be lowered, would exhibit up to 440 requests in excess every -300s (5 minutes), or an extra 1.4 requests per second once all pods reached -their max cap backoff. It also should be noted that due to the specifics of the -configuration required in the Pod manifest, being against an immutable field, -will require the Pods in question to be redeployed. This means it is unlikely -that all Pods will be in a simultaneous CrashLoopBackoff even if they are -designed to quickly crash, since they will all need to be redeployed and -rescheduled. <<[/UNRESOLVED]>> +change the state transition would increase at its fastest period from ABC +requests/10s to NNN requests/10s. <<[/UNRESOLVED]>> <<[UNRESOLVED ifttt +benchmarking: add benchmarking details]>>This has also been empirically observed +in benchmarking analysis described XYZ HERE XYZ. <<[/UNRESOLVED]>> In addition, +by passing these changes through the existing SIG-scalability tests, while +pursuing manual and more detailed periodic benchmarking during the alpha period, +we can increase the confidence in this change and in the possibility of reducing +further in the future. + +For the per node case, because the change is more significant, including +lowering the max backoff, there is more risk to node stability expected. This +change is of particular interest to be tested in the alpha period by end users, +and is why it is still included with opt-in even though the risks are higher. +<<[UNRESOLVED update with real stress test results]>> For a hypothetical node +with the max 110 pods all stuck in a simultaneous 1s maximum CrashLoopBackoff, +API requests to change the state transition would increase from ABC requests/10s +to NNN requests/10s, and since the maximum backoff would be lowered, would +exhibit up to NNN requests in excess every 300s (5 minutes), or an extra A.B +requests per second once all pods reached their max cap backoff. ## Design Details @@ -510,7 +531,7 @@ required) or even code snippets. If there's any ambiguity about HOW your proposal will be implemented, this is the place to discuss them. --> -<<[UNRESOLVED changing from 1.31 to 1.32 proposal, incoming in separate PR]>> +<<[UNRESOLVED changing from 1.31 to 1.32 proposal, incoming in further commits]>> ### Front loaded decay curve methodology As mentioned above, today the standard backoff curve is a 2x exponential decay @@ -567,7 +588,7 @@ Among these modeled initial values, we would get between 3-7 excess restarts per backoff lifetime, mostly within the first three time windows matching today's restart behavior. -<<[UNRESOLVED include stress test data now]>> <<[/UNRESOLVED]>> +<<[UNRESOLVED ifttt benchmarking: include stress test data now]>> <<[/UNRESOLVED]>> ### Rapid curve methodology @@ -690,7 +711,7 @@ container before restarting, and the number of container restarts, to articulate the rate and load of restart related API requests and the performance effects on kubelet. -<<[UNRESOLVED add initial benchmarking test data]>> <<[/UNRESOLVED]>> +<<[UNRESOLVED ifttt benchmarking: link initial benchmarking test data]>> <<[/UNRESOLVED]>> ### Relationship with Job API podFailurePolicy and backoffLimit @@ -809,9 +830,11 @@ This can inform certain test coverage improvements that we want to do before extending the production code to implement this enhancement. --> +<<[UNRESOLVED whats up with this]>> - `kubelet/kuberuntime/kuberuntime_manager_test`: **could not find a successful coverage run on [prow](https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/ci-kubernetes-coverage-unit/1800947623675301888)** +<<[/UNRESOLVED]>> ##### Integration tests @@ -848,7 +871,7 @@ We expect no non-infra related flakes in the last month as a GA graduation crite - k8s.io/kubernetes/test/e2e/node/kubelet_perf: for a given percentage of heterogenity between "Succeeded" terminating pods, and crashing pods whose -`restartPolicy: Always``, +`restartPolicy: Always` or `restartPolicy: OnFailure`, * what is the load and rate of Pod restart related API requests to the API server? * what are the performance (memory, CPU, and pod start latency) effects on the @@ -860,7 +883,7 @@ heterogenity between "Succeeded" terminating pods, and crashing pods whose #### Alpha - Changes to existing backoff curve implemented behind a feature flag - 1. Front-loaded decay curve for all workloads with `restartPolicy: Always` + 1. Front-loaded decay curve for all workloads with `restartPolicy: Always` or `restartPolicy: OnFailure` <<[UNRESOLVED]>> 2. Front-loaded, low max cap backoff curve for nodes workloads with XYZ config @@ -870,7 +893,7 @@ heterogenity between "Succeeded" terminating pods, and crashing pods whose exit states, and runtimes - Initial e2e tests completed and enabled * Feature flag on or off - * <<[UNRESOLVED]>>node upgrade and downgrade path <<[/UNRESOLVED]>> + * <<[UNRESOLVED in 1.31 review, suggested not necessary]>>node upgrade and downgrade path <<[/UNRESOLVED]>> - Fix https://github.com/kubernetes/kubernetes/issues/123602 if this blocks the implementation, otherwise beta criteria - Low confidence in the specific numbers/decay rate @@ -981,7 +1004,7 @@ To use the enhancement, the alpha feature gate is turned on. In the future when made, and the default behavior of the baseline backoff curve would -- by design -- be changed. -For `EnableRapidCrashLoopBackoffDecay`: +For `EnableKubeletCrashLoopBackoffMax`: For an existing cluster, no changes are required to configuration, invocations or API objects to make an upgrade. @@ -1002,7 +1025,7 @@ they've turned on the `ReduceDefaultCrashLoopBackoffDecay` feature gate). <<[/UNRESOLVED]>> Or, the entire cluster can be restarted with the -`EnableRapidCrashLoopBackoffDecay` feature gate turned off. In this case, any +`EnableKubeletCrashLoopBackoffMax` feature gate turned off. In this case, any <<[UNRESOLVED]>>Node configured with a different backoff curve will instead use the default backoff curve. Again, since the cluster was restarted and Pods were redeployed, they will not maintain prior state and will start at the beginning @@ -1035,7 +1058,7 @@ coordination must b e done between the control plane and the nodes. ## Production Readiness Review Questionnaire <<[UNRESOLVED removed when changing from 1.31 proposal to 1.32 proposal, -incoming in a separate PR]>> <<[/UNRESOLVED]>> +incoming in further commits]>> <<[/UNRESOLVED]>> ## Implementation History @@ -1094,35 +1117,6 @@ and insight can help us improve late recovery later on (see also the related discussion in Alternatives [here](#more-complex-heuristics) and [here](#late-recovery)). -CrashLoopBackoff behavior has been stable and untouched for most of the -Kubernetes lifetime. It could be argued that it "isn't broken", that most people -are ok with it or have sufficient and architecturally well placed workarounds -using third party reaper processes or application code based solutions, and -changing it just invites high risk to the platform as a whole instead of -individual end user deployments. However, per the [Motivation](#motivation) -section, there are emerging workload use cases and a long history of a vocal -minority in favor of changes to this behavior, so trying to change it now is -timely. Obviously we could still decide not to graduate the change out of alpha -if the risks are determined to be too high or the feedback is not positive. - -Though the issue is highly upvoted, on an analysis of the comments presented in -the canonical tracking issue -[Kubernetes#57291](https://github.com/kubernetes/kubernetes/issues/57291), 22 -unique commenters were requesting a constant or instant backoff for `Succeeded` -Pods, 19 for earlier recovery tries, and 6 for better late recovery behavior; -the latter is arguably even more highly requested when also considering related -issue [Kubernetes#50375](https://github.com/kubernetes/kubernetes/issues/50375). -Though an early version of this KEP also addressed the `Success` case, in its -current version this KEP really only addresses the early recovery case, which by -our quantitative data is actually the least requested option. That being said, -other use cases described in [User Stories](#user-stories) that don't have -quantitative counts are also driving forces on why we should address the early -recovery cases now. On top of that, compared to the late recovery cases, early -recovery is more approachable and easily modelable and improving benchmarking -and insight can help us improve late recovery later on (see also the related -discussion in Alternatives [here](#more-complex-heuristics) and -[here](#late-recovery)). - ## Alternatives -<<[UNRESOLVED changing from 1.31 to 1.32 proposal, incoming in further commits]>> - ### Front loaded decay curve methodology As mentioned above, today the standard backoff curve is a 2x exponential decay starting at 10s and capping at 5 minutes, resulting in a composite of the @@ -571,26 +569,38 @@ behavior, we modeled the change in the starting value of the decay from 10s to !["A graph showing the decay curves for different initial values"](differentinitialvalues.png "Alternate CrashLoopBackoff initial values") -For today's decay rate, the first restart is within the -first 10s, the second within the first 30s, the third within the first 70s. -Using those same time windows to compare alternate initial values, for example -changing the initial rate to 1s, we would instead have 3 restarts in the first -time window, 1 restart within the time window, and two more restarts within the -third time window. As seen below, this type of change gives us more restarts -earlier, but even at 250ms or 25ms initial values, each approach a similar rate -of restarts after the third time window. +For today's decay rate, the first restart is within the first 10s, the second +within the first 30s, the third within the first 70s. Using those same time +windows to compare alternate initial values, for example changing the initial +rate to 1s, we would instead have 3 restarts in the first time window, 1 restart +within the second time window, and two more restarts within the third time +window. As seen below, this type of change gives us more restarts earlier, but +even at 250ms or 25ms initial values, each approach a similar rate of restarts +after the third time window. +<<[UNRESOLVED ifttt front loaded images: remake this graph with axes too]>> ![A graph showing different exponential backoff decays for initial values of 10s, 1s, 250ms and 25ms](initialvaluesandnumberofrestarts.png "Changes to decay with different initial values") +TODO: remake graph! +<<[/UNRESOLVED]>> Among these modeled initial values, we would get between 3-7 excess restarts per backoff lifetime, mostly within the first three time windows matching today's restart behavior. -<<[UNRESOLVED ifttt benchmarking: include stress test data now]>> <<[/UNRESOLVED]>> +The above modeled values converge like this because of the harshly increasing +rate caused by the exponential growth, but also because they grow to the same +maximum value -- 5 minutes. By layering on alternate models for maximum backoff, +we observe more restarts for longer: + +<<[UNRESOLVED ifttt front loaded images: add graph here with proper axes]>> +TODO: new graph! <<[/UNRESOLVED]>> + +<<[UNRESOLVED ifttt benchmarking: include stress test data now]>>TODO: stress +test data paragraph<<[/UNRESOLVED]>> -### Rapid curve methodology +### Per node config For some users in [Kubernetes#57291](https://github.com/kubernetes/kubernetes/issues/57291), any @@ -604,18 +614,55 @@ configure (by container, node, or exit code) the backoff to close to 0 seconds. This KEP considers it out of scope to implement fully user-customizable behavior, and too risky without full and complete benchmarking to node stability to allow legitimately crashing workloads to have a backoff of 0, but it is in -scope for the first alpha to provide users a way to <<[UNRESOLVED]>> opt nodes -in <<[/UNRESOLVED]>> to a even faster restart behavior. - -The finalization of the initial and max cap can only be done after benchmarking. -But as a conservative first estimate for alpha in line with maximums discussed -on [Kubernetes#57291](https://github.com/kubernetes/kubernetes/issues/57291), -the initial curve is selected at <<[UNRESOLVED]>>initial=250ms / cap=1 minute, -<<[/UNRESOLVED]>> but during benchmarking this will be modelled against kubelet -capacity, potentially targeting something closer to an initial value near 0s, -and a cap of 10-30s. To further restrict the blast radius of this change before -full and complete benchmarking is worked up, this is gated by a separate alpha -feature gate and is opted in to per <<[UNRESOLVED]>> node <<[/UNRESOLVED]>>. +scope for the first alpha to provide users a way to opt nodes in to a even +faster restart behavior. + +As a first estimate for alpha in line with maximums discussed on +[Kubernetes#57291](https://github.com/kubernetes/kubernetes/issues/57291) and +the benchmarking analysis <<[UNRESOLVED ifttt benchmarking: include stress test +data now]>>here<<[/UNRESOLVED]>>, the initial curve is selected at initial=1s / +max=1 minute. During the alpha period this will be further investigated against +kubelet capacity, potentially targeting something closer to an initial value +near 0s, and a cap of 10-30s. To further restrict the blast radius of this +change before full and complete benchmarking is worked up, this is gated by a +separate alpha feature gate and is opted in to per node. + +<<[UNRESOLVED more details of per node config re: kubelet config api]>> TODO: +API <<[/UNRESOLVED]>> + +### Refactor of recovery threshold + +A simple change +to maximum backoff would naturally come with a modification of the backoff +counter reset threshold -- as it is currently calculated based on 2x the maximum +backoff. Without any other modification, as a result of this KEP, default +containers would be "rewarded" by having their backoff counter set back to 0 for +running successfully for 2\*1 minute=2 minutes (instead of for 2\*5minutes=10 +minutes like it is today); containers on nodes with an override could be +rewarded for running successfully for as low as 2 seconds if they are configured +with the minimum allowable backoff of 1s. + +From a technical persepctive, granularity of the associated worker polling loops +governing restart behavior is between 1 and 10 seconds, so a reset value under +10 seconds is effectively meaningless (until and unless those loops increase in +speed or we move to evented PLEG). From a user perspective, it does not seem +that there is any particular end user value in artificially preserving the +current 10 minute recovery threshold as part of this implementation, since it +was an arbitrary value in the first place. However, since late recovery as a +category of problem space is expressly a non-goal of this KEP, and in the +interest of reducing the number of changed variables during the alpha period to +better observe the ones previously enumerated, this proposal intends to maintain +that 10 minute recovery threshold anyways. + +Forecasting that the recovery threshold for CrashLoopBackOff may be better +served by being configurable in the future, or at least separated in the code +from all other uses of `client_go.Backoff` for whatever future enhancements +address the late recovery problem space, the mechanism proposed here is to +redefine `client_go.Backoff` to accept alternate functions for +[`client_go.Backoff.hasExpired`](https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/client-go/util/flowcontrol/backoff.go#L178), +and configure the `client_go.Backoff` object created for use by the kube runtime +manager for container restart bacoff with a function that compares to a flat +rate of 300 seconds. ### Kubelet overhead analysis From 978d99d2d0563a9261d9caa7d406468a75cf64f3 Mon Sep 17 00:00:00 2001 From: Laura Lorenz Date: Fri, 20 Sep 2024 14:14:32 -0700 Subject: [PATCH 03/36] Cleanup TODO and lingering unresolved tag Signed-off-by: Laura Lorenz --- keps/sig-node/4603-tune-crashloopbackoff/README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/keps/sig-node/4603-tune-crashloopbackoff/README.md b/keps/sig-node/4603-tune-crashloopbackoff/README.md index f11143ab3789..fa908f34ea9b 100644 --- a/keps/sig-node/4603-tune-crashloopbackoff/README.md +++ b/keps/sig-node/4603-tune-crashloopbackoff/README.md @@ -758,7 +758,8 @@ container before restarting, and the number of container restarts, to articulate the rate and load of restart related API requests and the performance effects on kubelet. -<<[UNRESOLVED ifttt benchmarking: link initial benchmarking test data]>> <<[/UNRESOLVED]>> +<<[UNRESOLVED ifttt benchmarking: link initial benchmarking test data]>>TODO: +link benchmarking data<<[/UNRESOLVED]>> ### Relationship with Job API podFailurePolicy and backoffLimit @@ -814,7 +815,6 @@ completely different pattern, as unlike with CrashLoopBackoff the types of errors with ImagePullBackoff are less variable and better interpretable by the infrastructure as recovereable or non-recoverable (i.e. 404s). -<<[/UNRESOLVED]>> ### Test Plan -<<[UNRESOLVED whats up with this]>> -- `kubelet/kuberuntime/kuberuntime_manager_test`: **could not find a successful + +- <<[UNRESOLVED whats up with this]>> + `kubelet/kuberuntime/kuberuntime_manager_test`: **could not find a successful coverage run on [prow](https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/ci-kubernetes-coverage-unit/1800947623675301888)** -<<[/UNRESOLVED]>> + <<[/UNRESOLVED]>> ##### Integration tests @@ -1021,11 +1027,11 @@ We expect no non-infra related flakes in the last month as a GA graduation crite - k8s.io/kubernetes/test/e2e/node/kubelet_perf: for a given percentage of heterogenity between "Succeeded" terminating pods, and crashing pods whose `restartPolicy: Always` or `restartPolicy: OnFailure`, - * what is the load and rate of Pod restart related API requests to the API + - what is the load and rate of Pod restart related API requests to the API server? - * what are the performance (memory, CPU, and pod start latency) effects on the - kubelet component? Considering the effects of different plugins (e.g. CSI, - CNI) + - what are the performance (memory, CPU, and pod start latency) effects on the + kubelet component? + - With different plugins (e.g. CSI, CNI) ### Graduation Criteria From c3d9e2c95a3c9580c5adf6eea64757619094410d Mon Sep 17 00:00:00 2001 From: Laura Lorenz Date: Tue, 24 Sep 2024 15:44:53 -0700 Subject: [PATCH 12/36] Add some new unresolved tags from IRL comments Signed-off-by: Laura Lorenz --- .../4603-tune-crashloopbackoff/README.md | 30 +++++++++++-------- 1 file changed, 18 insertions(+), 12 deletions(-) diff --git a/keps/sig-node/4603-tune-crashloopbackoff/README.md b/keps/sig-node/4603-tune-crashloopbackoff/README.md index f94d1805019d..cf24c69fb717 100644 --- a/keps/sig-node/4603-tune-crashloopbackoff/README.md +++ b/keps/sig-node/4603-tune-crashloopbackoff/README.md @@ -665,11 +665,12 @@ based config and 2) configuration following the API specification of the `kubelet.config.k8s.io/v1beta1 KubeletConfiguration` Kind, which is passed to kubelet as a config file or, beta as of Kubernetes 1.30, a config directory ([ref](https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/)). -Since this is a per-node configuration that likely will be set on a subset of -nodes, or potentially even differently per node, it's important that it can be -manipulated with a command-line flag, as `KubeletConfiguration` is intended to -be shared between nodes, and so lifecycle tooling that configures nodes can -expose it more transparently. +<<[UNRESOLVED consider KubeletConfiguration again, and/or motivate per node or +per node pool cases better]>> Since this is a per-node configuration that likely +will be set on a subset of nodes, or potentially even differently per node, it's +important that it can be manipulated with a command-line flag, as +`KubeletConfiguration` is intended to be shared between nodes, and so lifecycle +tooling that configures nodes can expose it more transparently. Exposed command-line configuration for Kubelet are defined by [`struct KubeletFlags`](https://github.com/kubernetes/kubernetes/blob/release-1.31/cmd/kubelet/app/options/options.go#L54) @@ -678,7 +679,7 @@ NewKubeletCommand`](https://github.com/kubernetes/kubernetes/blob/release-1.31/c To expose the configuration of this feature, a new field will be added to the `KubeletFlags` struct called `NodeMaxCrashLoopBackOff` of `int32` type that will be validated <<[UNRESOLVED ifttt conflict resolution]>>at runtime, or otherwise -dropped, <<[/UNRESOLVED]>>as a value over `1` and under 300s. +dropped, <<[/UNRESOLVED]>>as a value over `1` and under 300s. <<[/UNRESOLVED]>> ### Refactor of recovery threshold @@ -1209,6 +1210,11 @@ initial value 1s, depending on whether they've turned on the #### Conflict resolution +<<[UNRESOLVED]>> + +TODO: consider making one feature gate dependent on the other to minimize the matrix multiplication, and/or to depend fully on kubelet validation (which may mean crashing kubelets/nodes on misconfiguration) instead of catching internally + + If on a given node at a given time, the per-node configured maximum backoff is lower than the initial value, the initial value for that node will instead be set to the configured maximum. For example, if @@ -1219,14 +1225,14 @@ configured to 1s. In other words, operator-invoked configuration will have precedence over the default if it is faster. If on a given node at a given time, the per-node configured maximum backoff is -higher than the 300s, it will be dropped in favor of the default. In other -words, operator-invoked configuration will not have precedence over the default -if it is longer than 300s. +lower than 1 second or higher than the 300s, <<[UNRESOLVED]>>validation will +fail and the kubelet will crash. <<[/UNRESOLVED]>> If on a given node at a given time, the per-node configured maximum backoff is -higher than the current initial value, but still lower than 300s, it will be -honored. In other words, operator-invoked configuration will have precedence -over the default, even if it is slower, as long as it does not go over 300s. +higher than the current initial value, but within validation limits as it is +lower than 300s, it will be honored. In other words, operator-invoked +configuration will have precedence over the default, even if it is slower, as +long as it is valid. <<[/UNRESOLVED]>> <<[UNRESOLVED]>>TODO: a table describing the conflict resolution would probably help <<[/UNRESOLVED]>> From b879c7f45fe1297e71a03f23ec3c6f38ccdaba07 Mon Sep 17 00:00:00 2001 From: Laura Lorenz Date: Mon, 30 Sep 2024 10:09:45 -0700 Subject: [PATCH 13/36] Clean up some unresolved's to undraft Signed-off-by: Laura Lorenz --- .../4603-tune-crashloopbackoff/README.md | 37 +++++++++---------- 1 file changed, 18 insertions(+), 19 deletions(-) diff --git a/keps/sig-node/4603-tune-crashloopbackoff/README.md b/keps/sig-node/4603-tune-crashloopbackoff/README.md index cf24c69fb717..2b06a4c06044 100644 --- a/keps/sig-node/4603-tune-crashloopbackoff/README.md +++ b/keps/sig-node/4603-tune-crashloopbackoff/README.md @@ -533,6 +533,11 @@ every 300s (5 minutes), or an extra A.B requests per second once all pods reached their max cap backoff. <<[/UNRESOLVED]>> <<[UNRESOLVED ifttt benchmarking: add benchmarking details for this case too]>> <<[/UNRESOLVED]>> +For both of these changes, by passing these changes through the existing +SIG-scalability tests, while pursuing manual and more detailed periodic +benchmarking during the alpha period, we can increase the confidence in the +changes and explore the possibility of reducing the values further in the future. + ## Design Details -For the default backoff curve, no coordination must be done between the control -plane and the nodes; all behavior changes are local to the kubelet component and -its start up configuration. - -For the per-node case, since it is local to each kubelet and the -restart logic is within the responsibility of a node's local kubelet, no -coordination must be done between the control plane and the nodes. +For both the default backoff curve and the per-node, no coordination must be +done between the control plane and the nodes; all behavior changes are local to +the kubelet component and its start up configuration. An n-3 kube-proxy, n-1 +kube-controller-manager, or n-1 kube-scheduler without this feature available is +not affected when this feature is used, since it does not depend on +kube-controller-manager or kube-scheduler. Code paths that will be touched are +exclusively in kubelet component. -<<[UNRESOLVED confirm the following have been answered fully]>> -- How does an n-3 kubelet or kube-proxy without this feature available behave when this feature is used? -- How does an n-1 kube-controller-manager or kube-scheduler without this feature available behave when this feature is used? -- Will any other components on the node change? For example, changes to CSI, - CRI or CNI may require updating that component before the kubelet. -<<[/UNRESOLVED]>> +An n-3 kubelet without this feature available will behave like normal, with the +original CrashLoopBackOff behavior. It cannot ingest flags for per node config +and if one is specified on start up, kubelet flag validation will fail. ## Production Readiness Review Questionnaire <<[UNRESOLVED removed when changing from 1.31 proposal to 1.32 proposal, -incoming in further commits]>> <<[/UNRESOLVED]>> +incoming in separate PR]>> <<[/UNRESOLVED]>> ## Implementation History From 97cb4772fd21c5bd0229b6a8acd8d1a01bb89297 Mon Sep 17 00:00:00 2001 From: Laura Lorenz Date: Mon, 30 Sep 2024 14:28:51 -0700 Subject: [PATCH 20/36] Remove benchmarking ifftt, can add later Signed-off-by: Laura Lorenz --- .../4603-tune-crashloopbackoff/README.md | 63 +++++++------------ 1 file changed, 24 insertions(+), 39 deletions(-) diff --git a/keps/sig-node/4603-tune-crashloopbackoff/README.md b/keps/sig-node/4603-tune-crashloopbackoff/README.md index 534c014236f1..4275d8f39aa4 100644 --- a/keps/sig-node/4603-tune-crashloopbackoff/README.md +++ b/keps/sig-node/4603-tune-crashloopbackoff/README.md @@ -330,14 +330,13 @@ This design seeks to incorporate a two-pronged approach: backoff for all containers on a specific Node, down to 1s To derive these values, manual stress testing observing the behavior of kubelet, - API server, and overall cluster operations and behavior were performed, as - described below in <<[ UNRESOLVED ifttt benchmarking: link to benchmarking results ]>> -<<[/UNRESOLVED]>>. In addition, part of the alpha period will be dedicated -entirely to systematically stress testing kubelet and API Server with different -distributions of workloads utilizing the new backoff curves. During the -benchmarking period in alpha, kubelet memory and CPU, API server latency, and -pod restart latency will be observed and analyzed to further refine the maximum -allowable restart rate for fully saturated nodes. + API server, and overall cluster operations and behavior were performed. In +addition, part of the alpha period will be dedicated entirely to systematically +stress testing kubelet and API Server with different distributions of workloads +utilizing the new backoff curves. During the benchmarking period in alpha, +kubelet memory and CPU, API server latency, and pod restart latency will be +observed and analyzed to further refine the maximum allowable restart rate for +fully saturated nodes. Longer term, these metrics will also supply cluster operators the data necessary to better analyze @@ -365,10 +364,8 @@ earlier by changing the initial value of the exponential backoff, and to retry periodically more often by reducing the maximum for the backoff. A number of alternate initial values and maximum backoff values are modelled below. This proposal suggests we start with a new initial value of 1s (changed from 10s -today) and a maximum backoff of 30 seconds (changed from 5 minutes today) based -on prior research <<[ UNRESOLVED link to benchmarking results -]>>here<<[/UNRESOLVED]>>, and further analyze its impact on infrastructure -during alpha. +today) and a maximum backoff of 1 minute (changed from 5 minutes today) based on +prior research, and further analyze its impact on infrastructure during alpha. ![A graph showing the change in elapsed time for observed restarts with the CrashLoopBackOffBehavior of today vs the proposed new @@ -547,10 +544,6 @@ behavior, so each crashing pod would be adding an excess of 25 pod state transition AIP requests, for a total of 2750 excess API requests for a node fully saturated with crashing pods. -<<[UNRESOLVED ifttt benchmarking: add benchmarking details]>>For more thorough -methodology and results from the manual benchmarking tests, see the -[Benchmarking results section](#). <<[/UNRESOLVED]>> - For the per node case, because the change could be more significant, with nearly trivial (when compared to pod startup metrics) max allowable backoffs of 1s, there is more risk to node stability expected. This change is of particular @@ -653,18 +646,14 @@ the first time window, 1 excess in the second time window, 3 excess in the fourth time window and 4 excess in the fifth time window for a total of 10 excess restarts compared to today's behavior. -<<[UNRESOLVED ifttt benchmarking: include stress test data now]>>TODO: stress -test data paragraph<<[/UNRESOLVED]>> - As a first estimate for alpha in line with maximums discussed on -[Kubernetes#57291](https://github.com/kubernetes/kubernetes/issues/57291) and -the benchmarking analysis <<[UNRESOLVED ifttt benchmarking: include stress test -data now]>>here<<[/UNRESOLVED]>>, the initial curve is selected at initial=1s / -max=1 minute. During the alpha period this will be further investigated against -kubelet capacity, potentially targeting something closer to an initial value -near 0s, and a cap of 10-30s. To further restrict the blast radius of this -change before full and complete benchmarking is worked up, this is gated by its -own feature gate, `ReduceDefaultCrashLoopBackoffDecay`. +[Kubernetes#57291](https://github.com/kubernetes/kubernetes/issues/57291), the +initial curve is selected at initial=1s / max=1 minute. During the alpha period +this will be further investigated against kubelet capacity, potentially +targeting something closer to an initial value near 0s, and a cap of 10-30s. To +further restrict the blast radius of this change before full and complete +benchmarking is worked up, this is gated by its own feature gate, +`ReduceDefaultCrashLoopBackoffDecay`. ### Per node config @@ -695,15 +684,14 @@ manifest permissions in their namespace but not for other namespaces in the same cluster and which might be dependent on the same kubelet. For 1.32, we were looking to address the second issue by moving the configuration somewhere we could better guarantee a cluster operator type persona would have exclusive -access to. In addition, <<[UNRESOLVED ifttt benchmarking: linkies]>>initial -benchmarking indicated that even in the unlikely case of mass pathologically -crashing and instantly restarting pods across an entire node, cluster operations -proceeded with acceptable latency, disk, cpu and memory. Worker polling loops, -context timeouts, the interaction between various other backoffs, as well as API -server rate limiting made up the gap to the stability of the -system.<<[/UNRESOLVED]>> Therefore, to simplify both the implementation and the -API surface, this 1.32 proposal puts forth that the opt-in will be configured per -node via kubelet configuration. +access to. In addition, initial manual stress testing and benchmarking indicated +that even in the unlikely case of mass pathologically crashing and instantly +restarting pods across an entire node, cluster operations proceeded with +acceptable latency, disk, cpu and memory. Worker polling loops, context +timeouts, the interaction between various other backoffs, as well as API server +rate limiting made up the gap to the stability of the system.Therefore, to +simplify both the implementation and the API surface, this 1.32 proposal puts +forth that the opt-in will be configured per node via kubelet configuration. Kubelet configuration is governed by two main input points, 1) command-line flag based config and 2) configuration following the API specification of the @@ -890,9 +878,6 @@ container before restarting, and the number of container restarts, to articulate the observed rate and load of restart related API requests and the performance effects on kubelet. -<<[UNRESOLVED ifttt benchmarking: link initial benchmarking test data]>>TODO: -link benchmarking data<<[/UNRESOLVED]>> - ### Relationship with Job API podFailurePolicy and backoffLimit Job API provides its own API surface for describing alterntive restart From d81363db8541a6c21f71d3a4604d0d3caa00f1e9 Mon Sep 17 00:00:00 2001 From: lauralorenz Date: Mon, 30 Sep 2024 21:33:31 +0000 Subject: [PATCH 21/36] Update toc Signed-off-by: lauralorenz --- .../sig-node/4603-tune-crashloopbackoff/README.md | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-) diff --git a/keps/sig-node/4603-tune-crashloopbackoff/README.md b/keps/sig-node/4603-tune-crashloopbackoff/README.md index 4275d8f39aa4..6efcc421cdfc 100644 --- a/keps/sig-node/4603-tune-crashloopbackoff/README.md +++ b/keps/sig-node/4603-tune-crashloopbackoff/README.md @@ -83,7 +83,9 @@ tags, and then generate with `hack/update-toc.sh`. - [Goals](#goals) - [Non-Goals](#non-goals) - [Proposal](#proposal) - - [Existing backoff curve change: front loaded decay](#existing-backoff-curve-change-front-loaded-decay) + - [Existing backoff curve change: front loaded decay, lower maximum backoff](#existing-backoff-curve-change-front-loaded-decay-lower-maximum-backoff) + - [Node specific kubelet config for maximum backoff down to 1 second](#node-specific-kubelet-config-for-maximum-backoff-down-to-1-second) + - [Refactor and flat rate to 10 minutes for the backoff counter reset threshold](#refactor-and-flat-rate-to-10-minutes-for-the-backoff-counter-reset-threshold) - [User Stories](#user-stories) - [Task isolation](#task-isolation) - [Fast restart on failure](#fast-restart-on-failure) @@ -92,11 +94,14 @@ tags, and then generate with `hack/update-toc.sh`. - [Risks and Mitigations](#risks-and-mitigations) - [Design Details](#design-details) - [Front loaded decay curve methodology](#front-loaded-decay-curve-methodology) - - [Rapid curve methodology](#rapid-curve-methodology) + - [Per node config](#per-node-config) + - [Refactor of recovery threshold](#refactor-of-recovery-threshold) + - [Conflict resolution](#conflict-resolution) - [Kubelet overhead analysis](#kubelet-overhead-analysis) - [Benchmarking](#benchmarking) - [Relationship with Job API podFailurePolicy and backoffLimit](#relationship-with-job-api-podfailurepolicy-and-backofflimit) - [Relationship with ImagePullBackOff](#relationship-with-imagepullbackoff) + - [Relationship with k/k#123602](#relationship-with-kk123602) - [Test Plan](#test-plan) - [Prerequisite testing updates](#prerequisite-testing-updates) - [Unit tests](#unit-tests) @@ -107,6 +112,7 @@ tags, and then generate with `hack/update-toc.sh`. - [Beta](#beta) - [GA](#ga) - [Upgrade / Downgrade Strategy](#upgrade--downgrade-strategy) + - [Conflict resolution](#conflict-resolution-1) - [Version Skew Strategy](#version-skew-strategy) - [Production Readiness Review Questionnaire](#production-readiness-review-questionnaire) - [Implementation History](#implementation-history) @@ -122,6 +128,11 @@ tags, and then generate with `hack/update-toc.sh`. - [Front loaded decay with interval](#front-loaded-decay-with-interval) - [Late recovery](#late-recovery) - [More complex heuristics](#more-complex-heuristics) +- [Appendix A](#appendix-a) + - [Kubelet SyncPod](#kubelet-syncpod) + - [Runtime SyncPod](#runtime-syncpod) + - [Kubelet SyncTerminatingPod + runtime killPod](#kubelet-syncterminatingpod--runtime-killpod) + - [Kubelet SyncTerminatedPod](#kubelet-syncterminatedpod) - [Infrastructure Needed (Optional)](#infrastructure-needed-optional) From 40073eb41d44800a2bf5c9581982c110d3cae096 Mon Sep 17 00:00:00 2001 From: Laura Lorenz Date: Mon, 30 Sep 2024 14:34:18 -0700 Subject: [PATCH 22/36] spelling Signed-off-by: Laura Lorenz --- keps/sig-node/4603-tune-crashloopbackoff/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/keps/sig-node/4603-tune-crashloopbackoff/README.md b/keps/sig-node/4603-tune-crashloopbackoff/README.md index 6efcc421cdfc..e45c5650211d 100644 --- a/keps/sig-node/4603-tune-crashloopbackoff/README.md +++ b/keps/sig-node/4603-tune-crashloopbackoff/README.md @@ -1013,7 +1013,7 @@ increase confidence in ongoing node stability given heterogeneous backoff timers and timeouts. Some stress/benchmark testing will still be developed as part of the -implementatino of this enhancement, including the kubelet_perf tests indicated +implementation of this enhancement, including the kubelet_perf tests indicated in the e2e section below. Some of the benefit of pursuing this change in alpha is to also have the From 5997b34bd3df00ce8794dca66a2b9f2c8a6f70c9 Mon Sep 17 00:00:00 2001 From: Laura Lorenz Date: Tue, 1 Oct 2024 10:13:51 -0700 Subject: [PATCH 23/36] update some grad criteria and version skew info Signed-off-by: Laura Lorenz --- .../4603-tune-crashloopbackoff/README.md | 90 ++++++++++--------- 1 file changed, 48 insertions(+), 42 deletions(-) diff --git a/keps/sig-node/4603-tune-crashloopbackoff/README.md b/keps/sig-node/4603-tune-crashloopbackoff/README.md index e45c5650211d..9e9fbc1e4b0b 100644 --- a/keps/sig-node/4603-tune-crashloopbackoff/README.md +++ b/keps/sig-node/4603-tune-crashloopbackoff/README.md @@ -560,17 +560,19 @@ trivial (when compared to pod startup metrics) max allowable backoffs of 1s, there is more risk to node stability expected. This change is of particular interest to be tested in the alpha period by end users, and is why it is still included, but only by opt-in. In this case, for a hypothetical node with the -default maximum 110 pods all stuck in a simultaneous 1s maximum -CrashLoopBackoff, as above, at its most efficient this would result in a new -restart for each pod every second, and therefore the API requests to change the -state transition would be expected to increase from ~550 requests/10s to 5500 -requests/10s, or 10x. In addition, since the maximum backoff would be lowered, -an ideal pod would continue to restart more often than today's behavior, adding -305 excess restarts within the first 5 minutes and 310 excess restarts every 5 -minutes after that; each crashing pod would be contributing an excess of ~1550 -pod state transition API requests, and fully saturated node with a full 110 -crashing pods would be adding 170,500 new pod transition API requests every five -minutes, which is an an excess of ~568 requests/10s. +default maximum 110 pods each with one crashing container all stuck in a +simultaneous 1s maximum CrashLoopBackoff, as above, at its most efficient this +would result in a new restart for each pod every second, and therefore the API +requests to change the state transition would be expected to increase from ~550 +requests/10s to 5500 requests/10s, or 10x. In addition, since the maximum +backoff would be lowered, an ideal pod would continue to restart more often than +today's behavior, adding 305 excess restarts within the first 5 minutes and 310 +excess restarts every 5 minutes after that; each crashing pod would be +contributing an excess of ~1550 pod state transition API requests, and fully +saturated node with a full 110 crashing pods would be adding 170,500 new pod +transition API requests every five minutes, which is an an excess of ~568 +requests/10s. <<[!UNRESOLVED kubernetes default for the kubelet client rate +limit and how this changes by machine size]>> <<[UNRESOLVED]>> ## Design Details @@ -1094,41 +1096,42 @@ https://storage.googleapis.com/k8s-triage/index.html We expect no non-infra related flakes in the last month as a GA graduation criteria. --> +- Crashlooping container that restarts some number of times (ex 10 times), + timestamp the logs and read it back in the test, and expect the diff in those + time stamps to be minimum the backoff, with a healthy timeout - k8s.io/kubernetes/test/e2e/node/kubelet_perf: for a given percentage of heterogenity between "Succeeded" terminating pods, and crashing pods whose `restartPolicy: Always` or `restartPolicy: OnFailure`, - - what is the load and rate of Pod restart related API requests to the API - server? - - what are the performance (memory, CPU, and pod start latency) effects on the - kubelet component? - - With different plugins (e.g. CSI, CNI) + - what is the load and rate of Pod restart related API requests to the API + server? + - what are the performance (memory, CPU, and pod start latency) effects on the + kubelet component? + - With different plugins (e.g. CSI, CNI) ### Graduation Criteria #### Alpha - New `int32 NodeMaxCrashLoopBackOff` field in `KubeletFlags` struct, validated - to a minimum of 1 and a maximum of 300, naturally ignored by older - kubelets/on downgrade and explicitly dropped by current and future kubelets - when new `EnableKubeletCrashLoopBackoffMax` feature flag disabled -- New `ReduceDefaultCrashLoopBackoffDecay` feature flag which, when enabled, changes - CrashLoopBackOff behavior (and ONLY the CrashLoopBackOff behavior) to a - 2x decay curve starting at 1s initial value, 1 minute maximum backoff - for all workloads, therefore affecting workload configured with - `restartPolicy: Always` or `restartPolicy: OnFailure` + to a minimum of 1 and a maximum of 300, used when + `EnableKubeletCrashLoopBackoffMax` feature flag enabled, to customize + CrashLoopBackOff per node +- New `ReduceDefaultCrashLoopBackoffDecay` feature flag which, when enabled, + changes CrashLoopBackOff behavior (and ONLY the CrashLoopBackOff behavior) to + a 2x decay curve starting at 1s initial value, 1 minute maximum backoff for + all workloads, therefore affecting workload configured with `restartPolicy: + Always` or `restartPolicy: OnFailure` - Requires a refactor of image pull backoff from container restart backoff object - Maintain current 10 minute recovery threshold by refactoring backoff counter reset threshold and explicitly implementing container restart backoff behavior at the current 10 minute recovery threshold -- Metrics implemented to expose pod and container restart policy statistics, - exit states, and runtimes - Initial e2e tests completed and enabled * Feature flag on or off - Test coverage of proper requeue behavior; see https://github.com/kubernetes/kubernetes/issues/123602 -- Actually fix https://github.com/kubernetes/kubernetes/issues/123602 if this blocks the - implementation, otherwise beta criteria +- Actually fix https://github.com/kubernetes/kubernetes/issues/123602 if this + blocks the implementation, otherwise beta criteria - Low confidence in the specific numbers/decay rate @@ -1138,6 +1141,7 @@ heterogenity between "Succeeded" terminating pods, and crashing pods whose - High confidence in the specific numbers/decay rate - Benchmark restart load methodology and analysis published and discussed with SIG-Node +- Discuss PLEG polling loops and its effect on specific decay rates - Additional e2e and benchmark tests, as identified during alpha period, are in Testgrid and linked in KEP @@ -1237,13 +1241,13 @@ To use the enhancement, the alpha feature gate is turned on. In the future when made, and the default behavior of the baseline backoff curve would -- by design -- be changed. -To stop use of this enhancement, the entire cluster must be restarted with the -`ReduceDefaultCrashLoopBackoffDecay` feature gate turned off. In this case, all -Pods will be redeployed, so they will not maintain prior state and will start at -the beginning of their backoff curve. The exact backoff curve a given Pod will -use will be either the original one with initial value 10s, or the per-node -configured maximum backoff if the `EnableKubeletCrashLoopBackoffMax` feature -gate is turned on. +To stop use of this enhancement, the entire cluster must be restarted or just +kubelet must restarts with the `ReduceDefaultCrashLoopBackoffDecay` feature gate +turned off. Since kubelet does not cache the backoff object, on kubelet restart +Pods will start at the beginning of their backoff curve. The exact backoff curve +a given Pod will use will be either the original one with initial value 10s, or +the per-node configured maximum backoff if the +`EnableKubeletCrashLoopBackoffMax` feature gate is turned on. For `EnableKubeletCrashLoopBackoffMax`: @@ -1259,13 +1263,15 @@ with the `NodeMaxCrashLoopBackOff` kubelet command line flag set. To stop use of this enhancement, there are two options. On a per-node basis, nodes can be completely redeployed with -`NodeMaxCrashLoopBackOff` kubelet command line flag unset. <<[UNRESOLVED is this -true if kubelet restarts tho]>>Since the Pods must be completely redeployed, -they will lose their prior backoff counter anyways and, if they go on to be -restarted, will start from the beginning of their backoff curve (either the -original one with initial value 10s, or the new baseline with initial value 1s, -depending on whether they've turned on the `ReduceDefaultCrashLoopBackoffDecay` -feature gate). <> +`NodeMaxCrashLoopBackOff` kubelet command line flag unset. Since kubelet does +not cache the backoff object, on kubelet restart they will start from the +beginning of their backoff curve (either the original one with initial value +10s, or the new baseline with initial value 1s, depending on whether they've +turned on the `ReduceDefaultCrashLoopBackoffDecay` feature gate). + +<<[UNRESOLVED]>>Say what happens when `NodeMaxCrashLoopBackOff` exists but +`EnableKubeletCrashLoopBackoffMax is off, log a warning but drop it (because +peeps might set it and be like why it no work)<<[/UNRESOLVED]>> Or, the entire cluster can be restarted with the `EnableKubeletCrashLoopBackoffMax` feature gate turned off. In this case, any From 8b08e002e84c12f15a79a4b2a98b6706479613e4 Mon Sep 17 00:00:00 2001 From: Laura Lorenz Date: Tue, 1 Oct 2024 21:44:38 -0700 Subject: [PATCH 24/36] Specify when node is set but feature gate is off Signed-off-by: Laura Lorenz --- keps/sig-node/4603-tune-crashloopbackoff/README.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/keps/sig-node/4603-tune-crashloopbackoff/README.md b/keps/sig-node/4603-tune-crashloopbackoff/README.md index 9e9fbc1e4b0b..a35989ff8977 100644 --- a/keps/sig-node/4603-tune-crashloopbackoff/README.md +++ b/keps/sig-node/4603-tune-crashloopbackoff/README.md @@ -1269,10 +1269,6 @@ beginning of their backoff curve (either the original one with initial value 10s, or the new baseline with initial value 1s, depending on whether they've turned on the `ReduceDefaultCrashLoopBackoffDecay` feature gate). -<<[UNRESOLVED]>>Say what happens when `NodeMaxCrashLoopBackOff` exists but -`EnableKubeletCrashLoopBackoffMax is off, log a warning but drop it (because -peeps might set it and be like why it no work)<<[/UNRESOLVED]>> - Or, the entire cluster can be restarted with the `EnableKubeletCrashLoopBackoffMax` feature gate turned off. In this case, any Node configured with a different backoff curve will instead use @@ -1304,6 +1300,11 @@ lower than 300s, it will be honored. In other words, operator-invoked configuration will have precedence over the default, even if it is slower, as long as it is valid. +If `NodeMaxCrashLoopBackOff` exists but `EnableKubeletCrashLoopBackoffMax` is +off, kubelet will log a warning but will not honor the +`NodeMaxCrashLoopBackOff`. In other words, operator-invoked per node +configuration will not be honored if the overall feature gate is turned off. + scenario | ReduceDefaultCrashLoopBackoffDecay | EnableKubeletCrashLoopBackoffMax | Effective initial value ---------|---------|----------|--------- _today's behavior_ | disabled | disabled | 10s From 6f13d3bd18e4aa900679bc9d4e8c9d444b2018a4 Mon Sep 17 00:00:00 2001 From: Laura Lorenz Date: Wed, 2 Oct 2024 11:26:49 -0700 Subject: [PATCH 25/36] Change from kubelet flags to KubeletConfiguration for per node config Signed-off-by: Laura Lorenz --- .../4603-tune-crashloopbackoff/README.md | 89 ++++++++++++------- 1 file changed, 59 insertions(+), 30 deletions(-) diff --git a/keps/sig-node/4603-tune-crashloopbackoff/README.md b/keps/sig-node/4603-tune-crashloopbackoff/README.md index a35989ff8977..07c607c0dfaa 100644 --- a/keps/sig-node/4603-tune-crashloopbackoff/README.md +++ b/keps/sig-node/4603-tune-crashloopbackoff/README.md @@ -584,7 +584,7 @@ required) or even code snippets. If there's any ambiguity about HOW your proposal will be implemented, this is the place to discuss them. --> -### Front loaded decay curve methodology +### Front loaded decay curve, modified maximum backoff methodology As mentioned above, today the standard backoff curve is a 2x exponential decay starting at 10s and capping at 5 minutes, resulting in a composite of the standard hockey-stick exponential decay graph followed by a linear rise until @@ -702,7 +702,7 @@ that even in the unlikely case of mass pathologically crashing and instantly restarting pods across an entire node, cluster operations proceeded with acceptable latency, disk, cpu and memory. Worker polling loops, context timeouts, the interaction between various other backoffs, as well as API server -rate limiting made up the gap to the stability of the system.Therefore, to +rate limiting made up the gap to the stability of the system. Therefore, to simplify both the implementation and the API surface, this 1.32 proposal puts forth that the opt-in will be configured per node via kubelet configuration. @@ -715,17 +715,13 @@ Since this is a per-node configuration that likely will be set on a subset of nodes, or potentially even differently per node, it's important that it can be manipulated per node. By default `KubeletConfiguration` is intended to be shared between nodes, but the beta feature for drop-in configuration files in a -colocated config directory cirumvent this. However, it is still the opinion of -the author it is better manipulated with a command-line flag, so lifecycle -tooling that configures nodes can expose it more transparently. - -Exposed command-line configuration for Kubelet are defined by [`struct -KubeletFlags`](https://github.com/kubernetes/kubernetes/blob/release-1.31/cmd/kubelet/app/options/options.go#L54) -and merged with configuration files in [`kubelet/server.go -NewKubeletCommand`](https://github.com/kubernetes/kubernetes/blob/release-1.31/cmd/kubelet/app/server.go#L141). -To expose the configuration of this feature, a new field will be added to the -`KubeletFlags` struct called `NodeMaxCrashLoopBackOff` of `int32` type that will -be validated in kubelet at runtime, as a value over `1` and under 300s. +colocated config directory cirumvent this. In addition, `KubeletConfiguration` +drops fields unrecognized by the current kubelet's schema, making it a good +choice to circumvent compatability issues with n-3 kubelets. While there is an +argument that this could be better manipulated with a command-line flag, so +lifecycle tooling that configures nodes can expose it more transparently, the +advantages to backwards compatability outweigh this consideration for the alpha +period and will be revisted before beta. ### Refactor of recovery threshold @@ -1112,7 +1108,7 @@ heterogenity between "Succeeded" terminating pods, and crashing pods whose #### Alpha -- New `int32 NodeMaxCrashLoopBackOff` field in `KubeletFlags` struct, validated +- New `int32 crashloopbackoff.max` field in `KubeletConfiguration` API, validated to a minimum of 1 and a maximum of 300, used when `EnableKubeletCrashLoopBackoffMax` feature flag enabled, to customize CrashLoopBackOff per node @@ -1258,12 +1254,12 @@ To make use of this enhancement, on cluster upgrade, the `EnableKubeletCrashLoopBackoffMax` feature gate must first be turned on for the cluster. Then, if any nodes need to use a different backoff curve, their kubelet must be completely redeployed either in the same upgrade or after that upgrade -with the `NodeMaxCrashLoopBackOff` kubelet command line flag set. +with the `crashloopbackoff.max` `KubeletConfiguration` set. To stop use of this enhancement, there are two options. On a per-node basis, nodes can be completely redeployed with -`NodeMaxCrashLoopBackOff` kubelet command line flag unset. Since kubelet does +`crashloopbackoff.max` `KubeletConfiguration` unset. Since kubelet does not cache the backoff object, on kubelet restart they will start from the beginning of their backoff curve (either the original one with initial value 10s, or the new baseline with initial value 1s, depending on whether they've @@ -1291,8 +1287,9 @@ configured to 1s. In other words, operator-invoked configuration will have precedence over the default if it is faster. If on a given node at a given time, the per-node configured maximum backoff is -lower than 1 second or higher than the 300s, validation will -fail and the kubelet will crash. +lower than 1 second or higher than the 300s, validation will fail and the +kubelet will crash/be unable to start, like it does with other invalid kubelet +configuration today. If on a given node at a given time, the per-node configured maximum backoff is higher than the current initial value, but within validation limits as it is @@ -1300,10 +1297,11 @@ lower than 300s, it will be honored. In other words, operator-invoked configuration will have precedence over the default, even if it is slower, as long as it is valid. -If `NodeMaxCrashLoopBackOff` exists but `EnableKubeletCrashLoopBackoffMax` is -off, kubelet will log a warning but will not honor the -`NodeMaxCrashLoopBackOff`. In other words, operator-invoked per node -configuration will not be honored if the overall feature gate is turned off. +If `crashloopbackoff.max` `KubeletConfiguration` exists but +`EnableKubeletCrashLoopBackoffMax` is off, kubelet will log a warning but will +not honor the `crashloopbackoff.max` `KubeletConfiguration`. In other words, +operator-invoked per node configuration will not be honored if the overall +feature gate is turned off. scenario | ReduceDefaultCrashLoopBackoffDecay | EnableKubeletCrashLoopBackoffMax | Effective initial value ---------|---------|----------|--------- @@ -1315,8 +1313,8 @@ _slower per node config_ | enabled | 10s | 10s " | enabled | 300s | 300s " | disabled | 11s | 11s " | disabled | 300s | 300s -_invalid per node config_ | enabled | 301s | kubelet crashes -" | disabled | 301s | kubelet crashes +_invalid per node config_ | disabled | 301s | kubelet crashes +" | enabled | 301s | kubelet crashes ### Version Skew Strategy @@ -1338,18 +1336,25 @@ For both the default backoff curve and the per-node, no coordination must be done between the control plane and the nodes; all behavior changes are local to the kubelet component and its start up configuration. An n-3 kube-proxy, n-1 kube-controller-manager, or n-1 kube-scheduler without this feature available is -not affected when this feature is used, since it does not depend on -kube-controller-manager or kube-scheduler. Code paths that will be touched are -exclusively in kubelet component. +not affected when this feature is used, nor will different CSI or CNI +implementations. Code paths that will be touched are exclusively in kubelet +component. An n-3 kubelet without this feature available will behave like normal, with the -original CrashLoopBackOff behavior. It cannot ingest flags for per node config -and if one is specified on start up, kubelet flag validation will fail. +original CrashLoopBackOff behavior. It will drop unrecognized fields in +`KubeletConfiguration` by default for per node config so if one is specified on +start up in a past kubelet version, it will not break kubelet (though the +behavior will, of course, not change). + +While the CRI is a consumer of the result of this change (as it will recieve +more requests to start containers), it does not need to be updated at all to +take advantage of this feature as the restart logic is entirely in process of +the kubelet component. ## Production Readiness Review Questionnaire <<[UNRESOLVED removed when changing from 1.31 proposal to 1.32 proposal, -incoming in separate PR]>> <<[/UNRESOLVED]>> +incoming in separate commit]>> <<[/UNRESOLVED]>> ## Implementation History @@ -1372,6 +1377,8 @@ Major milestones might include: * 06-06-2024: Removal of constant backoff for `Succeeded` Pods * 09-09-2024: Removal of `RestartPolicy: Rapid` in proposal, removal of PRR, in order to merge a provisional and address the new 1.32 design in a cleaner PR +* 09-20-2024: Rewrite for 1.32 design focused on per-node config in place of + `RestartPolicy: Rapid` ## Drawbacks @@ -1638,6 +1645,28 @@ situations, while the goal of restarting successful containers is only to get them to run again sometime and not penalize them with longer waits later when they've behaving as expected. +### Exposing per-node config as command-line flags + +Command-line configuration is more easily and transparently exposed in tooling +used to bootstrap nodes via templating. Exposed command-line configuration for +Kubelet are defined by [`struct +KubeletFlags`](https://github.com/kubernetes/kubernetes/blob/release-1.31/cmd/kubelet/app/options/options.go#L54) +and merged with configuration files in [`kubelet/server.go +NewKubeletCommand`](https://github.com/kubernetes/kubernetes/blob/release-1.31/cmd/kubelet/app/server.go#L141). +To expose configuration as a command-line flag, a new field would be added to +the `KubeletFlags` struct to be validated in kubelet at runtime. + +**Why not?**: Per comments in the code at [this +location](https://github.com/kubernetes/kubernetes/blob/release-1.31/cmd/kubelet/app/options/options.go#L52), +it seems we don't want to continue to expose configuration at this location. In +addition, since config directories for kubelet config are now in beta +([ref](https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/)), +there is a reasonable alternative to per-node configuration using the +`KubeletConfiguration` API object instead. By using the API, we can take +advantage of API machinery level lifecycle, validation and guarantees, including +that unrecognized fields will be dropped in older versions of kubelet, which is +valuable for version skew requirements we must meet back to kubelet n-3. + ### Front loaded decay with interval In an effort to anticipate API server stability ahead of the experiential data we can collect From e4c8ce485805ef0027e4bb946ddbcca07dccbdfd Mon Sep 17 00:00:00 2001 From: lauralorenz Date: Wed, 2 Oct 2024 18:32:56 +0000 Subject: [PATCH 26/36] update toc Signed-off-by: lauralorenz --- keps/sig-node/4603-tune-crashloopbackoff/README.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/keps/sig-node/4603-tune-crashloopbackoff/README.md b/keps/sig-node/4603-tune-crashloopbackoff/README.md index 07c607c0dfaa..52f8485a145b 100644 --- a/keps/sig-node/4603-tune-crashloopbackoff/README.md +++ b/keps/sig-node/4603-tune-crashloopbackoff/README.md @@ -93,7 +93,7 @@ tags, and then generate with `hack/update-toc.sh`. - [Notes/Constraints/Caveats (Optional)](#notesconstraintscaveats-optional) - [Risks and Mitigations](#risks-and-mitigations) - [Design Details](#design-details) - - [Front loaded decay curve methodology](#front-loaded-decay-curve-methodology) + - [Front loaded decay curve, modified maximum backoff methodology](#front-loaded-decay-curve-modified-maximum-backoff-methodology) - [Per node config](#per-node-config) - [Refactor of recovery threshold](#refactor-of-recovery-threshold) - [Conflict resolution](#conflict-resolution) @@ -125,6 +125,7 @@ tags, and then generate with `hack/update-toc.sh`. - [On Success and the 10 minute recovery threshold](#on-success-and-the-10-minute-recovery-threshold) - [Related: API opt-in for flat rate/quick restarts when transitioning from Succeeded phase](#related-api-opt-in-for-flat-ratequick-restarts-when-transitioning-from-succeeded-phase) - [Related: Succeeded vs Rapidly failing: who's getting the better deal?](#related-succeeded-vs-rapidly-failing-whos-getting-the-better-deal) + - [Exposing per-node config as command-line flags](#exposing-per-node-config-as-command-line-flags) - [Front loaded decay with interval](#front-loaded-decay-with-interval) - [Late recovery](#late-recovery) - [More complex heuristics](#more-complex-heuristics) From 6ab6e96192870e1102f96bf72e44319a36439817 Mon Sep 17 00:00:00 2001 From: Laura Lorenz Date: Wed, 2 Oct 2024 12:26:32 -0700 Subject: [PATCH 27/36] PRR and welcome back to soltysh as my reviewer O:) Signed-off-by: Laura Lorenz --- keps/prod-readiness/sig-node/4603.yaml | 3 + .../4603-tune-crashloopbackoff/README.md | 504 +++++++++++++++++- .../4603-tune-crashloopbackoff/kep.yaml | 11 +- 3 files changed, 501 insertions(+), 17 deletions(-) create mode 100644 keps/prod-readiness/sig-node/4603.yaml diff --git a/keps/prod-readiness/sig-node/4603.yaml b/keps/prod-readiness/sig-node/4603.yaml new file mode 100644 index 000000000000..222473fbd251 --- /dev/null +++ b/keps/prod-readiness/sig-node/4603.yaml @@ -0,0 +1,3 @@ +kep-number: 4603 +alpha: + approver: "@soltysh" diff --git a/keps/sig-node/4603-tune-crashloopbackoff/README.md b/keps/sig-node/4603-tune-crashloopbackoff/README.md index 52f8485a145b..fdf7f3141c9f 100644 --- a/keps/sig-node/4603-tune-crashloopbackoff/README.md +++ b/keps/sig-node/4603-tune-crashloopbackoff/README.md @@ -508,10 +508,10 @@ Some observations and analysis were made to quantify these risks going into alpha. In the [Kubelet Overhead Analysis](#kubelet-overhead-analysis), the code paths all restarting pods go through result in 5 obvious `/pods` API server status updates; when observed empirically in an active cluster, while we do see -an initial surges in `/pods` API traffic to ~5 QPS when deploying 110 mass -crashing pods for our tests, even with instantly crashing pods and -intantaneously restarting CrashLoopBackOff behavior, `/pods` API requests -quickly normalized to ~2 QPS due to rate limiting. In the same tests, runtime +an initial surges in `/pods` API traffic to the kubelet client side rate limit +of ~5 QPS when deploying 110 mass crashing pods for our tests, even with +instantly crashing pods and intantaneously restarting CrashLoopBackOff behavior, +`/pods` API requests quickly normalized to ~2 QPS. In the same tests, runtime CPU usage increased by x10 and the API server CPU usage increased by 2x. For both of these changes, by passing these changes through the existing @@ -553,7 +553,7 @@ change the state transition would increase at its fastest period from ~550 requests/10s to ~5500 requests/10s, or 10x. In conjunction, the lowered backoff would continuously add 5 excess restarts every five minutes compared to today's behavior, so each crashing pod would be adding an excess of 25 pod state -transition AIP requests, for a total of 2750 excess API requests for a node +transition API requests, for a total of 2750 excess API requests for a node fully saturated with crashing pods. For the per node case, because the change could be more significant, with nearly @@ -1123,8 +1123,13 @@ heterogenity between "Succeeded" terminating pods, and crashing pods whose - Maintain current 10 minute recovery threshold by refactoring backoff counter reset threshold and explicitly implementing container restart backoff behavior at the current 10 minute recovery threshold -- Initial e2e tests completed and enabled - * Feature flag on or off +- Initial e2e tests setup and enabled +- Initial unit tests covering new behavior + - Especially confirming the backoff object is set properly depending on the +feature gates set as per the [Conflict Resolution](#conflict-resolution) policy +- Test proving `KubeletConfiguration` objects will silently drop unrecognized + fields in the `config.validation_test` package + ([ref](https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/apis/config/validation/validation_test.go)). - Test coverage of proper requeue behavior; see https://github.com/kubernetes/kubernetes/issues/123602 - Actually fix https://github.com/kubernetes/kubernetes/issues/123602 if this @@ -1239,7 +1244,7 @@ made, and the default behavior of the baseline backoff curve would -- by design -- be changed. To stop use of this enhancement, the entire cluster must be restarted or just -kubelet must restarts with the `ReduceDefaultCrashLoopBackoffDecay` feature gate +kubelet must restart with the `ReduceDefaultCrashLoopBackoffDecay` feature gate turned off. Since kubelet does not cache the backoff object, on kubelet restart Pods will start at the beginning of their backoff curve. The exact backoff curve a given Pod will use will be either the original one with initial value 10s, or @@ -1354,8 +1359,486 @@ the kubelet component. ## Production Readiness Review Questionnaire -<<[UNRESOLVED removed when changing from 1.31 proposal to 1.32 proposal, -incoming in separate commit]>> <<[/UNRESOLVED]>> + + +### Feature Enablement and Rollback + + + +###### How can this feature be enabled / disabled in a live cluster? + + + +- [x] Feature gate (also fill in values in `kep.yaml`) + - Feature gate name: `ReduceDefaultCrashLoopBackoffDecay` + - Components depending on the feature gate: `kubelet` + - Feature gate name: `EnableKubeletCrashLoopBackoffMax` + - Components depending on the feature gate: `kubelet` + +###### Does enabling the feature change any default behavior? + + + +Yes, `ReduceDefaultCrashLoopBackoffDecay` changes the default backoff curve for +exiting Pods (including sidecar containers) when `restartPolicy` is either `OnFailure` +or `Always`. + +Since we currently only have anecdotal benchmarking, the alpha will implement +more conservative modeled initial value and maximum backoff, though less +conservative from the 1.31 proposal as initial manual data indicated faster +maximum backoff caps were safe. The new values for all CrashLoopBackOff behavior +for all workloads will be intial value=1s, max=60s. (See [this +section](#front-loaded-decay-curve-modified-maximum-backoff-methodology) for +more rationale.) + +###### Can the feature be disabled once it has been enabled (i.e. can we roll back the enablement)? + + + +Yes, disable is supported. + +For `ReduceDefaultCrashLoopBackoffDecay`, if this is disabled, once kubelet is +restarted it will initialize the default backoff to the prior initial value of +10s, and all restart delays thereafter will be calculated against this equation. +Since changing this configuration will at minimum require a restart of kubelet +to take effect, restart delays will begin at the beginning of their backoff +curve since the backoff is not cached between kubelet restarts. + +For `EnableKubeletCrashLoopBackoffMax`, similarly, if this is disabled, once +kubelet is restarted it will initialize the default backoff based on the global +default -- which will itself depend on whether +`ReduceDefaultCrashLoopBackoffDecay` is independently turned on or not. For a +complete workup of the various conflicts possible and how they will be handled, +see the [Conflict Resolution](#conflict-resolution) section in Design Details. + +###### What happens if we reenable the feature if it was previously rolled back? + +Both features can also be reenabled. + +For `ReduceDefaultCrashLoopBackoffDecay`, if this is reenabled, once kubelet is +restarted it will initialize the default backoff again to the new initial value +of 1s and maximum backoff to 1 minute, and all restart delays thereafter will be +calculated against this equation. + +For `EnableRapidCrashLoopBackoffDecay`, if this is disabled, once kubelet is +restarted it will -- again, as above -- initialize the default backoff based on +the global default -- which will itself depend on whether +`ReduceDefaultCrashLoopBackoffDecay` is independently turned on or not. For a +complete workup of the various conflicts possible and how they will be handled, +see the [Conflict Resolution](#conflict-resolution) section in Design Details.. + +###### Are there any tests for feature enablement/disablement? + + + +At minimum, unit tests will be included confirming the backoff object is set +properly depending on the feature gates set as per the [Conflict +Resolution](#conflict-resolution) policy. + +In this version of the proposal, there are no API schema changes or conversions +necessary. However it is worth nothing tere is one addition to an API object, +which is a new field in the `KubeletConfiguration` Kind. Based on manual tests +by the author, adding an unknown field to `KubeletConfiguration` is safe and the +unknown config field is dropped before addition to the +`kube-system/kubelet-config` object which is its final destination (for example, +in the case of n-3 kubelets facing a configuration introduced by this KEP). This +is not currently tested as far as I can tell in the tests for +`KubeletConfiguration` (in either the most likely location, in +[validation_test](https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/apis/config/validation/validation_test.go), +nor other tests in the [config +package](https://github.com/kubernetes/kubernetes/tree/005f184ab631e52195ed6d129969ff3914d51c98/pkg/kubelet/apis/config)) +and discussions with other contributors indicate that while little in core +kubernetes does strict parsing, it's not well tested. At minimum as part of this +implementation a test covering this for `KubeletConfgiuration` objects will be +included in the `config.validation_test` package. + +### Rollout, Upgrade and Rollback Planning + + + +###### How can a rollout or rollback fail? Can it impact already running workloads? + + + +<<[UNRESOLVED]>> Fill out when targeting beta to a release. <<[/UNRESOLVED]>> + +###### What specific metrics should inform a rollback? + + + +This biggest bottleneck expected will be kubelet, as it is expected to get more +restart requests and have to trigger all the overhead discussed in [Design +Details](#kubelet-overhead-analysis) more often. Cluster operators should be +closely watching these existing metrics: + +* Kubelet component CPU and memory +* `kubelet_http_inflight_requests` +* `kubelet_http_requests_duration_seconds` +* `kubelet_http_requests_total` +* `kubelet_pod_worker_duration_seconds` +* `kubelet_runtime_operations_duration_seconds` + +Most important to the perception of the end user is Kubelet's actual ability to +create pods, which we measure in the latency of a pod actually starting compared +to its creation timestamp. The following existing metrics are for all pods, not +just ones that are restarting, but at a certain saturation of restarting pods +this metric would be expected to become slower and must be watched to determine +rollback: +* `kubelet_pod_start_duration_seconds` +* `kubelet_pod_start_sli_duration_seconds` + + +###### Were upgrade and rollback tested? Was the upgrade->downgrade->upgrade path tested? + + + +<<[UNRESOLVED]>> Fill out when targeting beta to a release. <<[/UNRESOLVED]>> + +###### Is the rollout accompanied by any deprecations and/or removals of features, APIs, fields of API types, flags, etc.? + + + +<<[UNRESOLVED]>> Fill out when targeting beta to a release. <<[/UNRESOLVED]>> + +### Monitoring Requirements + + + +###### How can an operator determine if the feature is in use by workloads? + + + +<<[UNRESOLVED]>> Fill out when targeting beta to a release. <<[/UNRESOLVED]>> + +###### How can someone using this feature know that it is working for their instance? + + + +<<[UNRESOLVED]>> Fill out when targeting beta to a release. +- [ ] Events + - Event Reason: +- [ ] API .status + - Condition name: + - Other field: +- [ ] Other (treat as last resort) + - Details: + + <<[/UNRESOLVED]>> + +###### What are the reasonable SLOs (Service Level Objectives) for the enhancement? + + + +<<[UNRESOLVED]>> Fill out when targeting beta to a release. <<[/UNRESOLVED]>> + +###### What are the SLIs (Service Level Indicators) an operator can use to determine the health of the service? + + + +<<[UNRESOLVED]>> Fill out when targeting beta to a release. + +- [ ] Metrics + - Metric name: + - [Optional] Aggregation method: + - Components exposing the metric: +- [ ] Other (treat as last resort) + - Details: + +<<[/UNRESOLVED]>> + +###### Are there any missing metrics that would be useful to have to improve observability of this feature? + + + +<<[UNRESOLVED]>> Fill out when targeting beta to a release. <<[/UNRESOLVED]>> + +### Dependencies + + + +###### Does this feature depend on any specific services running in the cluster? + + + +<<[UNRESOLVED]>> Fill out when targeting beta to a release. <<[/UNRESOLVED]>> + +### Scalability + + + +###### Will enabling / using this feature result in any new API calls? + + + +It will not result in NEW API calls but it will result in MORE API calls. See +the [Risks and Mitigations](#risks-and-mitigations) section for the +back-of-the-napkin math on the increase in especially /pods API endpoint calls, +which initial benchmarking showed an aggressive case (110 instantly restarting +single-contaier pods) reaching 5 QPS before slowing down to 2 QPS. + +###### Will enabling / using this feature result in introducing new API types? + + + +No, this KEP will not result in any new API types. + +###### Will enabling / using this feature result in any new calls to the cloud provider? + + + +No, this KEP will not result in any new calls to the cloud provider. + +###### Will enabling / using this feature result in increasing size or count of the existing API objects? + + + +No, this KEP will not result in increasing size or count of the existing API objects. + +###### Will enabling / using this feature result in increasing time taken by any operations covered by existing SLIs/SLOs? + + + +Maybe! As containers will be restarting more, this may affect "Startup latency +of schedulable stateless pods", "Startup latency of schedule stateful pods". +This is directly the type of SLI impact that a) the split between the default +behavior change and the per node opt in is trying to mitigate, and b) one of the +targets of the benchmarking period during alpha. + +###### Will enabling / using this feature result in non-negligible increase of resource usage (CPU, RAM, disk, IO, ...) in any components? + + + +Yes! We expect more CPU usage of kubelet as it processes more restarts. In +initial manual benchmarking tests, CPU usage of kubelet increased 2x on nodes +saturated with 110 instantly crashing single-container pods. During the alpha +benchmarking period, we will be quantifying that amount in fully and partially +saturated nodes with both the new default backoff curve and the minimum per node +backoff curve. + +###### Can enabling / using this feature result in resource exhaustion of some node resources (PIDs, sockets, inodes, etc.)? + + + +Based on the initial benchmarking, no, which was based on manual benchmarking +tests on nodes saturated with 110 instantly crashing single-container pods. +However, more "normal" cases (with lower percentage of crashing pods) and even +more pathological cases (with higher container-density Pods, sidecars, n/w +traffic, and large image downloads) will be tested further during this alpha +period to better articulate the risk against the most aggressive restart +characteristics. + +### Troubleshooting + + + +###### How does this feature react if the API server and/or etcd is unavailable? + +<<[UNRESOLVED]>> Fill out when targeting beta to a release. <<[/UNRESOLVED]>> + +###### What are other known failure modes? + + + +###### What steps should be taken if SLOs are not being met to determine the problem? + +<<[UNRESOLVED]>> Fill out when targeting beta to a release. <<[/UNRESOLVED]>> ## Implementation History @@ -1380,6 +1863,7 @@ Major milestones might include: order to merge a provisional and address the new 1.32 design in a cleaner PR * 09-20-2024: Rewrite for 1.32 design focused on per-node config in place of `RestartPolicy: Rapid` +* 10-02-2024: PRR added for 1.32 design ## Drawbacks diff --git a/keps/sig-node/4603-tune-crashloopbackoff/kep.yaml b/keps/sig-node/4603-tune-crashloopbackoff/kep.yaml index 719fe630b37a..0d89c86a4f58 100644 --- a/keps/sig-node/4603-tune-crashloopbackoff/kep.yaml +++ b/keps/sig-node/4603-tune-crashloopbackoff/kep.yaml @@ -10,9 +10,8 @@ status: provisional creation-date: 2024-04-29 reviewers: - "@tallclair" - - "@pehunt" approvers: - - "SergeyKanzhelev" + - "dchen1107" # see-also: # - "/keps/sig-aaa/1234-we-heard-you-like-keps" @@ -39,12 +38,10 @@ milestone: feature-gates: - name: ReduceDefaultCrashLoopBackoffDecay components: - - kube-apiserver - kubelet - # - name: EnableRapidCrashLoopBackoffDecay - # components: - # - kube-apiserver - # - kubelet + - name: EnableKubeletCrashLoopBackoffMax + components: + - kubelet disable-supported: true # The following PRR answers are required at beta release From fb7ed3fab87e624a640ec81d367c873706808aa7 Mon Sep 17 00:00:00 2001 From: lauralorenz Date: Wed, 2 Oct 2024 19:30:03 +0000 Subject: [PATCH 28/36] Spelling errors and TOC Signed-off-by: lauralorenz --- keps/sig-node/4603-tune-crashloopbackoff/README.md | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/keps/sig-node/4603-tune-crashloopbackoff/README.md b/keps/sig-node/4603-tune-crashloopbackoff/README.md index fdf7f3141c9f..56cfa401432e 100644 --- a/keps/sig-node/4603-tune-crashloopbackoff/README.md +++ b/keps/sig-node/4603-tune-crashloopbackoff/README.md @@ -115,6 +115,12 @@ tags, and then generate with `hack/update-toc.sh`. - [Conflict resolution](#conflict-resolution-1) - [Version Skew Strategy](#version-skew-strategy) - [Production Readiness Review Questionnaire](#production-readiness-review-questionnaire) + - [Feature Enablement and Rollback](#feature-enablement-and-rollback) + - [Rollout, Upgrade and Rollback Planning](#rollout-upgrade-and-rollback-planning) + - [Monitoring Requirements](#monitoring-requirements) + - [Dependencies](#dependencies) + - [Scalability](#scalability) + - [Troubleshooting](#troubleshooting) - [Implementation History](#implementation-history) - [Drawbacks](#drawbacks) - [Alternatives](#alternatives) @@ -718,10 +724,10 @@ manipulated per node. By default `KubeletConfiguration` is intended to be shared between nodes, but the beta feature for drop-in configuration files in a colocated config directory cirumvent this. In addition, `KubeletConfiguration` drops fields unrecognized by the current kubelet's schema, making it a good -choice to circumvent compatability issues with n-3 kubelets. While there is an +choice to circumvent compatibility issues with n-3 kubelets. While there is an argument that this could be better manipulated with a command-line flag, so lifecycle tooling that configures nodes can expose it more transparently, the -advantages to backwards compatability outweigh this consideration for the alpha +advantages to backwards compatibility outweigh this consideration for the alpha period and will be revisted before beta. ### Refactor of recovery threshold @@ -1352,7 +1358,7 @@ original CrashLoopBackOff behavior. It will drop unrecognized fields in start up in a past kubelet version, it will not break kubelet (though the behavior will, of course, not change). -While the CRI is a consumer of the result of this change (as it will recieve +While the CRI is a consumer of the result of this change (as it will receive more requests to start containers), it does not need to be updated at all to take advantage of this feature as the restart logic is entirely in process of the kubelet component. @@ -1420,7 +1426,7 @@ Since we currently only have anecdotal benchmarking, the alpha will implement more conservative modeled initial value and maximum backoff, though less conservative from the 1.31 proposal as initial manual data indicated faster maximum backoff caps were safe. The new values for all CrashLoopBackOff behavior -for all workloads will be intial value=1s, max=60s. (See [this +for all workloads will be initial value=1s, max=60s. (See [this section](#front-loaded-decay-curve-modified-maximum-backoff-methodology) for more rationale.) From f721bd307b3cbf8d5627b3c3c7e067d57e26e42d Mon Sep 17 00:00:00 2001 From: Laura Lorenz Date: Wed, 2 Oct 2024 12:31:47 -0700 Subject: [PATCH 29/36] implementable, come at me bots Signed-off-by: Laura Lorenz --- keps/sig-node/4603-tune-crashloopbackoff/kep.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/keps/sig-node/4603-tune-crashloopbackoff/kep.yaml b/keps/sig-node/4603-tune-crashloopbackoff/kep.yaml index 0d89c86a4f58..a0048b37426f 100644 --- a/keps/sig-node/4603-tune-crashloopbackoff/kep.yaml +++ b/keps/sig-node/4603-tune-crashloopbackoff/kep.yaml @@ -6,7 +6,7 @@ owning-sig: sig-node # participating-sigs: # - sig-aaa # - sig-bbb -status: provisional +status: implementable creation-date: 2024-04-29 reviewers: - "@tallclair" From 53356d7cd05df6bfac95a06eb96bf3e963e7c0a7 Mon Sep 17 00:00:00 2001 From: Laura Lorenz Date: Wed, 2 Oct 2024 12:35:04 -0700 Subject: [PATCH 30/36] Fix incorrect purple line in graphs Signed-off-by: Laura Lorenz --- .../restarts-vs-elapsed-all.png | Bin 20140 -> 19500 bytes .../restarts-vs-elapsed-new-default.png | Bin 17002 -> 16381 bytes 2 files changed, 0 insertions(+), 0 deletions(-) diff --git a/keps/sig-node/4603-tune-crashloopbackoff/restarts-vs-elapsed-all.png b/keps/sig-node/4603-tune-crashloopbackoff/restarts-vs-elapsed-all.png index 90aa3d6e50c6c52224d27126083f16bf2fc0b224..3850370af91cd8ea95777296247b1183c4a202f5 100644 GIT binary patch literal 19500 zcmc$`Wmr`0`z}0yh=PEMh=PQHG>8&PBP!iPr&7`(9W#WAil9=uo0m+Ep*mP18*?EsfBg znmfo&&#iXuLr-oE?T@qhml8zn4~zWNCwj_Vy9h%&F~fMJ2l=i$JX}GMJSOdl+PGd$w7``qwqT&8nsKPjGyyBmuuC z;<-MOrgt-xhPh<6IdmLBz}pND70`w6MmlC@Wwq-tOJfGLo$9JrSw$Sa@dzY{I(v4N zIZsQiNG~EC$0z(Z)P2t2$}BrBA09fxaS3L3R$yzC4)~)TCsf(A7{8cQ^59IMWeU|~E|uT+$XAhilmBY{^zk7sS+2_3QOcgF+9 zsy%Yo_V$nFKh@lwy>$E8ok4!9SH{}sJ7M^pw#}_d*LkJx;j-OK)r460I+$q&4q;bi z(H2p4xW7N3KK`Ib!h+R*J17A@(?D`~>W<%smDgKNpGP3f`Zm%o=jm;^_NK@>`Jl1c z83t?nR=bj4&FjRyHp_<1;FHWf!NI{=R{n9Yvyr*eohs92Vq#8M*1iL1>i&?CGy4X| zw-vwkKb2>!@K=%m(17^A^VzZ+DT~ZqGFCxHTdf1Wa%AQ2Foo$vaJDX{O>0uAkqE3cgu)`X@?V zEA|q(FN`&w&0}@2qkC9zfHoYKc(5_fV3<%9k<2}gf)lik!RmUHzcl7ZC` zDlPDh-=3h&mpMNxkjZAop(ibU;{tX~Ip*(rMN@mdrRh&_t~Xfq;uzTS@viF}o|b0~ z5SkUAr;6oTd(e({wi4MApeN@ltjF3l?NYw*+?rw*_}cAvZg+s0L0Vd2)`uiteDpBg zYisJfRS+lCH&EDP6)I|`ml1ZA&rMI3ih`76JNm5z3*xIBL2 zjE^CzU<|6OSJ#LnrSy)@7|z+_|kNwRhDy1L&lJnaG$g2Bp?jE22~0 z%|ityYn=sVsHxN4uc%6YwDG$F6=6xM&gi2OUqp2$RfOhIsB3?kBEOlQ^NQuGj}In? z15ve>*hh*b3sDPSg-v32*Q3Px*Xnp?(1NKjaYypg?)CZn@1|^Ua-oK=&rFEHhv5CZ5Wb5X@{AK!CHa#=1NHsCL-YuX|x7lsm~G!ZicOxcya~eRSzgcI@F^S3Lixk^|f0-Gvk{HlKmpIZT;x%jQSN50M!ncAHFiOMGbKSQvJu zwM!&Sp4f}sXi@CdniZghLenYq8|hkbF`{mN+BG`XkH-6TJbR^|;|?|yyUysjWk)Vj47SPn$eLe+daR|T zq9~W|oNIMw*N+907r}9#?mmz6SL@h~1h$cs?Qp=A_4G3N25j`7X*Viz9Zk_Niw&x- zy`KF_edH6j3D2=57bZb}J9|6}Q<=pFN+)rnevB)!b28vr&jwE?rCil0IZmeVfei$R zHQu)hI9r9y|Fw9MemPIB(OGP?Hq;zv#mvxgN4?yq16c>c$mlpwS$KF&NqG%Lgh(^Pf5#q}&H$JBR;NII+LgrV!TB?@7bSUZGeaP~6 zjuKfCm`8nrno)`_I#ORE+46Ar&B6-Bt8h_fdg&TQFK<5hM$$B-xq4NYLmgC<=+b6Jpt#2={c2G~R9!WM-j?MG96>7U;p^h{yi%o? zH6@Qnwoj+du6ds&M4XD^!nK*j?5LUyk2eK-P6%56?}WuEHz2GZ{{?2i8l z`TcqJfRHlaQ)yAb$+=sRI(9=fbjw^5k(QDOr%&}fHTRR{i&86~pWV`y;gBPjU}WtQ zpu0hf+Iqq*d+*|eWEM@$qb z8=>ijtvx;e2J=auY>(HWfqg%dcAElvRmKxeM%S03nllGwX3&lX$rR4wta+Z z<&XKVyKC}o_6q)8=?CTaevYoDR>l4nVbGvEjxB;&^%e}QqF;bUPX1OfPqGe#K}X7o)>H56@^S%jnfeY z9|vg_2{YI)z0&2awcNbC?hDX2n^e=M`K}p}I~7rC23FeP$o#|9?>C_31@qq_N~$u| z_f?*?sjc;J@9jG=$e?lFo zZ)Nk2u1BHr-H$;IhT==azMKuJ9x@m@-}yHHfq?7?^O5e_UBm^uRrmHGg<**>FDMvE=bdev1>CyWvD!iF+~|GyM%?PH)UILTw&+Gdfah>4&jmtnuv@c5aC!-zSt z)Z#a()^Ceoo-|YCL1m0-ycEG+E#JN;_3y-$ z;2fyp-UCd!KmeCjr>;-z`_5mPFP65z#nSJG{WV_B(zxilHkFFh6zq>M{u0lPvE$PF zSEq)kMJHU`o;`hv|L#w~O;>=TFuHS~Y?myKr9!0(?SAZ%*fhQ4Kf($%Iis&&zccsq zJc&+8n%ij3@+ZBgUj+((CSC5C?1=2NQrcPo2)>-p2zloZf-=r&sfkL!og)P#*S{~) z%^2s?rjfd`F#%zip`9o@oq`ii6sMK9tF<%LJiCR5E2H+m30i2yu1%v* z!nXY_>8|sw2G5^=7k|4FyO&vNp#CkENy@(x=MTrvvl_X&y84^u^hoJ%VusBs;Dkf? zr=>Sb>AqVFCkV9)f#*DW1!eG)VR*T-|2)&dDQ>H6Z@^I$eR_7f&0M+(S8CBlgX>U- zy_m)?isq-Ju+5f|_?P1^u%$$mvMQu2H!4bLcfe9u;CR!l9IMQ*xE4poU_WjbjyW+xw zOSNA?!7^wfi97DkP-W87&N_T(+5Xx4!DN)Wlx>|a#;uTWP=OlNLt%CN=EFrN+$)?W zCj>F+;<4Gt+i^Eup%SH&Rz0c~mgV~DG3%oP$v%tMMe|W&$=L^Hg$VDt^E_N!#bEcV z=)7{=P=u2$lZKS}%)o^hJyX}|WB0D(fzdn;l`hHsUb)TEVZul=t@I6q-jGcpRrX|! zSKD6Go>txA?x}j__}6>n&#_ZHmC$etwn@){jY9ZGGVC-oFgpP4dMM4*px4y{A>CW- zRzCNmpFO7MSqW>_mj9CY4v(rykf{e+XUC`dJ}qowIL{!;-A7Z^@Sgff^XZ;6VW94` zkbo6P5VF;(d^xmO7wuiI_7y5c*Lb?xhKn_Fa)mb^?`}B5#Z8s77Q<#Nf5gY6EGlmm z2jC70bY)I)D8l%Z-&WlS<>P+lF6j6dm53^U`nvbJVw59@J`>5VyuO&=>^VSbewA1M zvl{^;X3X@bhYCg0N#3S9joHRq7M`p)4AEuyqZdDPuGL)RTq~tTq3*#=y~qDG8_M+x z$MHp*%t@!}6ID#L?LfxP#u1M4jy7@5c22t?Szt3uzpPWN&V>m?90ok}-G7!~eLE zC;rHFct6dWlUX?cp|yb`m}+%X4J9}vOPdVUstnU*;LWq>$?we)ic6$$+Xqc}$2XZ> zIO!xOv!p6@DY#jXrQPv$85y~IN3ziB=#9_R!0z2+uAez07L<#U zlhOO!N$1E@i)Z$ln2nc*CZt-w*{+qw&(aoL&wmbw<|PZoCvNEV#O$~x+Xykr>If>fma5Yhq?CR2nJ zciiO+s!Vg{=PSqrC%AUq5H*b~^-#6P+T|gVqra2VgRgq8QfLuFuivn02`eXhB5S?? z`;624XmSh2fXI}(!YtbCq8 z=N|nV5#hY(D($%x?pdLw|d5tal0WG5F4bIhI!BV5mj|5^ate8u&brC5qT zMqO(gg4Y3;VV4dH)fTzb~WP;H=k^<&dmpLLR&x`bC~)MBo2MkF&;613@fRo?Az(raA$ zWI%e97Pd`hRPAo-pB!HV6{+kq2g$|BkR!_y_^11L$No18=1_EAMuP}iaZMC;98PtH zhnIp?sR&VRq*draO1qTlx1pZtYck}X(IyzR61aWB0xVwB4 zJWl-Q9vLeh$A8lPNXvd3<`T4?HL8X3lg5qq;0GrZmwwsHHhd!s6|W{N zqAF!OQR&(*zQS#{WmswpzsIz%T3XN0c4}vMoOQ>a)9O|OH>+?+?p@8K4K(3iB^{l@ zoB#dv^#4GD>(oH(R|m5X@|^>^qzK17yx*v=Br zb;TK~31c6xKe!)H0fEfqdXfxTeKoOe4yCO)!5xmj>XEdWKH2n|--{-svYv24v1yCl z2}S^MI1U$>02i{aSu3ddf9}=QS5$nT!yXKbz_X8 zPKkXIfVej+rzrx=m%W}@fQLU8IqvkT`>hQk0BPbi>4E&q>G)z+N59;bpI8f_pf#MV z_IU8m0EbA1N_(X2Z_7t~R7b#1j{!NX&%mpiAkrN#Y=0Hugy8O{j@)lzm(T zg1}LZW1tQcYDp`NFG4VTt&a%inxSC+)v_UHJ&(4Z9~zcJbILm{lhv75Y`Wv0-uj|& z=W_N3_N3-iEj}_{VQOM7;lHNzv|N8F@s*l?wPMs=B<#fZ(6Gx4cD!)Cn+yWk28*uT zEHvTP#a9qgqwaMr$5x{e{ih>}h=;;@>_QbTF}>{EGVd)qZHr!^}ysJ65za}+PDWch5*aw19Z7rAR8VK)A7&3Q4xg?Zl?e9 zD#%|m%ZZMRyv)PRT>>VAE!{m!97fjth#QYy06xDsf?2$}%w_g5O2L$kDAwq-0##tX zF{Xc{*a8RxK}6e!UZ>+w%u%88=i>Zv_O8rHy#|f?{^%*p?+*p^<0dj0QK*FOW4<}w7fP^M0Rp_&~ zI8-0t2k`&ba;M3$&FZy+f1bky;2{yE&e)9Gw|utL3mg!sk%DnAHQB3^B@C}!xvw^sr7K9Y=E2i+vp9VDrDJyk--?V zH9hUIkp26=yR=Jo?OAHZN?QFZ!nxKtKlc2YM4ie2xlZ)r_fN3AD~m-ba(=S1K{YqK z1)^vbm$tK@T@Yo}5Fjx)KSowoy*T(xj@xN&kAR3`Sr1xXv@6+L5RwibMbWm((6Ge0M`mJ`DpRe|rp`pRZ& zAs5&~55TY+<{OqPr3WT=P8L0v%QdM%);!RdH_~wjE2oeq)wP8p)Ke7*X@QtJesT+K zSWfhBnT3RV&x5H7_CGe7XQjr!Ci=}2yWKw>{FSR?<+&h;bRbM_|23<26@-)|i?JiH zPK1fY-%AQFv&hKxlEl4`!uCT)rS?~d-l6dvK=CjNJy7uM+YCSaYrv1M@*6J)=c*-% zje=EQ?J4;3*S)IvOsWUTbt#KLp!(BOQ2ftDUbA`$7UPvm9uMCWhhyo8Sw$D1VuEuY zy;=k>k@)L^F+kWf_{wi$?1I^8W5ty||LZ$cU{l;JjJYb~?^}J4W`0+1iAY(`ry^--K4r|SbuT!de=oD=um&oVp1^>=m7u(Put{1)G;oUvQ( z{r=a&&-J$}2I_!Na-R;Evot8w+~uBr-UUjDqIq=jWirv-+5|VXvh5CZPZTXuoy>4~^M6Ih_7P zp{L|K#Ma4?Niw7=I%!Enl@(q&7KyFB-a9`THbB*n2_pqzq|6STb2RT9y`LzHiaZn7 zWgAFLqVeZ+EIeAT+l-Ji7@N~d9kH=q0}<`9h2e@%@T_%Xu|->TAl1Ux0pbSz5x^8O zK610HZOs8HsX(yXEfD@v&JqR7f8qqOn51=m=I+L5p3PMRrte9#RuEW>r--xy(7Se( zz)-ZU08z1Xsta<_D+|+`hL5O=HY?|8L`n1BqG?;}wK5NL3+u;t&O;z>s-lUifXSSi zCYq`Q_O3j5wC%UI=J=0n4gK!meN8`sZAYvWi!us9gLD9d01?y*cIgRcCP}Pa(cp8n z<~5!MG5@2Y?j-SHC)9~2iW6+erSM-zkXD(`6eT}qqD&JiS_T}g zEs%Tbx&}7B>zVqb>s2$%$b{eaR|}}x`69J6md-4Via6iAbXoP zK*pHP}1)NGwcYMz5+3A+0HP5`sHNJZ#=|q062JP#iVC~$*XQ* z&x28$aWFg^`=Pw|Jp*J}<)7SuGM)>x{%3o@)`W!#*?vmBcd}~sF#cwL3R^huJf^gS z##~4G9XjEf80s~+1sEd9^_LLvdOwH2;B=6LLAvRl+f$Q1**~%m_@e$V@cgX2S}A2W z$jP9eEr+Z1*~8yzmwT_N@EUu#}x8JAA7B{ zD_3a^RqKR#{C>35k_iZuIZ_#e8qa)_e}O6+!&L#(^|RDEF2e518sint2(W&<0avu% zuu~QB*R0;$dTvrp03yo1@5fW(0=Mj)wm}|!%eY*T77XHe9iv`fVvB)4xA32~PZ=uE2 zu0^q<%wN^=;lqb@&@fa?6nzCKAu{2@UYo{igV|94SAD!b@09fz9XkL~o_|Vi7RgFD zY7+s}izw&V8wS(mm@|-+#XjbP`PT4PK{2GflO3(MopVBgLrk9@R5*fGi_LPe_kh_M z!q{j@--STz?qkScOG#eFE;(Ow`;J>p&>kyr_Tb4 zH0p?u^P-^Sm^FxILqI|G4v*QqJlOk`k?AtpOpPoL08t~5*4F!#1))DX%@R}0 zS>+zq!LKSLu{jEu+VpW})Fg5DVp8|UK%q3>a|t>y6U%3We~}gRzliN+fUaoRL=j-) z^#L_zS0I7hR-h)_k(ER9W-Xn|QNFhO5eQm(EuenY)#d{1lh8I39X(1fLZR)ckx%8C&(rSAU#dK ziUT1<&v#A%;1m;cU_<7hUj!2dv0V^qs9w#Gl}+loK;yl z0z%c5!t!__+eoJG%A)JpC(vY3PX4e_bwV9YU zV+Se?q*d2d)U}JmWhde+ix01DvG~0nffx#i41XSIrokR;tda5vQ`z;}dpJ?J@%;n0 z{=Bsv0ZGIDa0=;|PdH2y!^{!gtwXK$dnlTR=QneWH-2 zV)Gv#tOrx^7g4bK-}|l$5_MJ{pfW&GS|H_-l(pQPs2Vh!jRpvfoH)*%JR$uotR8v! zU%}u|2L8JDgl=1#_ z<&93BLY1)#qd*(7pRTVXVl2Z7Cr3cb=n<JNtscq#(v= zPz*yL^rgW6)`klaRC$0cS#M>8=2B?H5c96qK#Colzy9dOU(U%a54VMyULb?miFoKE zfbvpHS0h$NAvXunF;OD}aHTfN3M;$>%ibs_ z{Eky&5}?q2N2J-ws-E(Kt?#dIg6X>eU?W)U2TEHa~`CFpuH^8QK6?cOhs;JlO3L7$@>b^X7G!A)|wEXa8o9 zI=Y?jV8KUBzT;P3Hi^0|^3v4589lbp1vwG?VsW)_gVeP!T8A!kR@LLx?2QbWd?m>Yi5Q@+#GD0 zBrzJSJu?gIzqQjubFx6?3@OXO1&B^Ge=KNKT@2`NuQ2Tvr)4>0(KyyG~4(%24n)upFe-X1R=>FeA3-jDSAuIR6V?U z!RJ5uFyi9UEyDTjQh}Af&>@U zdT)>{u31=5X&_VmFpN~;Ca4s}kh$S7V*hT6CUswmS|rHWpW#CLO+a>@CwsdpC`xR~ zEZ3Egzu2w@7KdDmQ6Ea{Uw?{SGciTMPS!z)+Xs~`_W)q9CQX859RXE$^HUWGH(Mr& zZto56y>3yWy_uK-oHIGe0SWBPWK(K{UepH|w1_=o2sA~h1?KiePy&~@5E&~$$fGA98KGY%M}K!c8=|&OG(vhF-J_q7L{hL= z8q^@jwSpAMwACWzvui2tzWmwjLIW3(fzcsS_5WCy#^TCqZfKFdb7)jO>;;WwCkn$hNP)lOm zYMIJZXEmzxEjPO$!vb<*Ah$UH1pFy~t8u`j@z0QcMu4ZWeac=@R}GYXHS_tVoWnTM z5zNOY;~BX=k=ZjWw~J9C@wn;;s#SEhc3ZrNqum4RU+phhAb*0N86!lUlh9{W@ap^WtAlb+*_#2$u`Q5%Hgu!&a^)A2)%LQ~1ZZ|5I* zZ>OO1jYL7^eUV(U|Jf(xd9mWPFQScGG#UdpMy!!OIiqv@)&s1yA*qm7{Iu1@rE^F3WVjo*_4OVBhJ zL~LOAr#f}^(dCP=%fBDDq%DvAhj7&z6Y||(_SQ&Gvz9f1U>6k9a*{JpqCJ%2{N;k5 z@{4bxMjW>Moxjo>d$Zkiq0HpRL>O+hvWskojytwE504))=#He-Q#rOqKXOVMc*Zc= z9y&CN6W~)HF?|~OdsZ{M7_daeptemsa{rgN-7LpS82?k<2C6tt*H5C8DO7`Ig`PK^ zK|R#4y1^wle(-+u1BvykQ-_1bp6;_|jze{@7u#3V5fA*9?w1Ny#Hsp?ILuD}5=lN1 z9YKwn`JthyqcA(X;?By3=HwsG)L1ZF8 zNjSW?=putg!QpygXprcb9=YA|I)BUghPWltISIj3>XkoQU2MHKEtIf}vQC9vv@cOU zm;6{!6((0GoZzCK$EBefEPk~${#bVa*L{6*B$f+tFDjoX25`(t0se+%G z4nw{-S~x~iQc|v`ev)!)z6Uvexpb>EdFdxMlq zO)NE68yi_*ZItm-EO!_x%nkg}xZi~zFjd8%ZK)-Mn_M4A3$9-0!C!+Qw+hw7#^+zR z)!ok-l}8COxmsY^loXq9XC-#pR|(&Ku_k+q7`-6E8@4SNP*cX*kq(ya9)egVK$>`; zqN2?%GS(gEP#MD7W*09utau^$a{tn=yC-7~24 zHE>lu{X0W&(EA|m*^w{$h`XW_jbYYwTj3ZV?pKu|g)LZ(Y=IaoG}{^do4t4}`#a$> zQwZgPJcwe~i6zFGT>28}+XTq8-VZkG6Q-Yb@%R}o zh3YVMuf3Im`q7OSUdJm=(vz6b3BZ;H3W3fgd%z?X6Duhw-ONm5$RxEuwerPxonKCh zm4P&=bQ@V})YCNAhr$pOYmc#wdF7thj&_`lRG=`a&lBPvweFFC&g za%+$paZ!RnLvdbD6(N=MWk{1h;qK4)&1aJi$l_jy$%m!naKVJ#&!fw%@t7)3LEtFC z@ORVpe|GohY0g&5((<7qP^UTdr=il+GDg zcj4l?{X6s4rrFP`!dCsCt;hK}Rfgym!!&&FnyIpErsRN0YNQfN1Ib&d&eQej@eqtk z!_P2=@$Yg_TDCe91WaV2(cYR>(p+TjZmB6EoIQryLu}IF2xA?!DVlJoMYJ=(u%%~~Pix20AZq}osG z)r;9HZWL=H3_7&VTeISKpO||X=G|2e$otD{WL+Ap1+7hD!xNd-C>J_&pr#Qu4zn}D zAK!}iDlF!{jqmpBaF`>syyaQ$V*~4+ImPfT9P3ruK^xx}@?-;N)MUzuBx1rt*B9C5+zLQX zDg{R~wB-1AoasBW~0 z#+pgWypk-MmgH(6w{k$X`x1gh@c{rjuOn7viHOy4ti_;2jC<4^IrD^VP!Nh6T*J3C za3xZnPC2xD{h~}joU0#0oNJnry!5f#rjqPQ7#LYhie^9N9Ha1wEhjl)6fWC;k&5i> zg?FFKh|wyZB)78Juls`yTx3F)sq_$TeXtp0ZPKH3Avrj0mIe#&c9__(9ym2eR%++P zu0hlVn}{fZO<-f9YJmFSkyYmvCcu6X#o3$R`Yr6H0uD`vhwVBd*^vO{9_@lO>-MQo zlaC?i?POQ?(8v1;glZ5!gqe;{O6*g8VS)#o*T|v#ML8Bo->gyBWW(1=KxIzpCyHyj z`GyN(srLK(+oh*Xio)8gu%qwY24x{W0e!wbyp>Xg`ZJD`BT}mZtvs(88}4 zuu~?d{9vK|q#qUkvam)R+#eTfUhn-KwKiFIOXPJ^K|04?U?+e_GtpWc?%^K_mT*Lp zMH1FZn-=Xpj97&4c5o2&{j>PI#}SBkeqQ_{Mv}bXiD_mD)HT2J+8(`c)P9J;YOM?2 zk1`_q$IDVlo|AGi9WtpZC81-(pb=?W>IwvP$ z-GpxK%%*q|+F`)5^eMB&Nn-2iGMIy!I#~C?{N&LZRf}fkJFQy6txuTZFh^}I zDIMw{zR)%5&KCY!lp>6}P(1a4Ps!blh(3CB{Iq$|3!i9!po-EfTqFU}OOZ%IEJ~v9 z6Sbbn>`ZG3>35VI!X&GjLg1}d*I8xUXg2+A}>c z{{l*vUYzbufw0Sf?){-q?})^zaUJzPI9-07EGZnvcVxeA9P_K*qqwa&%SUyIAAMQX z@1d^6pWjczEt;bQ}EkAyL*%CdC=6q1uS875#A6i$y*qIt8gYs{Uh65 z<@0{&bBWfLKms*ZH&3g9Too>qaOk!^k>|Ln?{Z6jY<)vNL#U-HSu3&dpdeh5Q-k6~ zgXoKm=*C8xOg1e{6bt_{1wX6g=WCT8ajq|B?*EBzX<*MH$)QoK@LZK#3BYcdv^7gV zV{G^(9cveI@Tn?Nik9%8pFxMfJDUb)L!1V=%fAWkkO+v?1f9(?r`nER&Z3P$qX?D% z6$sZwEt#~2cG}oTc5*|?$}8Sn5N)=MK$Rea@@r;buO&hj4?fkt*3{ z+U1E^yJ)pUJ*lJPjy*M(WK~h5_GjNg9%KYUJoz(g;<(W1z`KMOzf3+t!?ia3+Uk_q0Qo_LvIsIYvmR2EMD?vI2ID zQE^aOL+9?B>f$9<_&r#G+4M9)Fu2|Qh+y{MO&%?s8hx+$Q>=f1~=-e-z_oC{1H!D_{-J!Zv zTeA7NOT;$7uyCkw<755tXVrSYay}G6+9W7Iz}URhNDVpnh0dWz{*7l**7l$Gn)BC= zIC_5!Usf-pgyEyydSjO}y-v9VT{J0p;BC1|s|RoT2UZ<2>LpHbG&zTOp}p*u`D}p? z@Co!%JFPKNMfrm9LBZ*I!mW8FsB@x^$uKqDZxQ5^+?0F6-~CearS=wg*(bPEr}ar~LxylzI#vTCm1H3?yp93ut6Bqo<8b!xD(QEHUt#J~+)WQl)jeB4 zd>=jv3X31bH&{90S{j~qd%|XZq*+TzyC|aDl&sK9rnLA0z7LT5TA-0S=Qc*0`>1Pt zXs?_`#C7h8QSpm6RFQ+vje;VB?+ABsDFBU*TyeEM=_14`Zje=Jt^c8>VFfaEL)~wS zE8PZ<|&Xl?@ zV@fxU#yN0b*8LZEU*C^O#u{77i-#gQcB}NaoTXi^d&$3;`4Q0#rb*L#K1f-1m!^7E ze}jUb0-1!rg}v(a!J`DbjoKy9JcJlzFW5j9H1Qpm#mhM2gH{Gx(iv}iAa%pPXkEO> zXHe2Y{zPeYBtP(}?O>KBf1oB?YT&c%;b}3W=>zA={pY!D8zV*@#xC1Cw^8M@>XJ6M zlkvQg*tz$|pA4SoN++ujaM7xS6FHH!FJ|NG()iXzDQsZCZq4-g5BbwJU5V3Dy2f8U z4j4M=3VK>X|4`W}?Kt)qrg-q+>^Q!K8oBUf>~QWU|Ar&{zDu9foik?;>D06Y`|OqQ zPZOUtGVTw2MJ%*aR#~$Xli%9r{h>ynB`WJNjWrqcft(h1Z-)l(!p*K}_EKuvPmkjS)#2@>7 zU6jInqnA8DarB4qyJ*G_ud*18t^V#`=@WNo)*il_X3`^4Cb=QN8$*Md^>D-VZ!YnN zqjW#N%9C4_S%PND6%sE!6xlcGN+yw5lOD6fG=BWF5vk$+a?=WLJZx=mD-O3Ds8SjRF%hS^nrIu@2m z3K&NVZTPJypC^&2E}MJL8k>A7R|-roCZg6yxAtoGQAfs(KW1`tXtULO>~ta=dE7(T z5aXw?T?^wlt>4XZ^saK0p!agHH>#<(OQ^ouqE$i4(6WAQss|a)?QXkt^q40u;8u5> zIqqt{+e&Y>4hPv+_nwQ7V84*{82RzhQ*k85*%4m_=;buxE){RSXN{)3HdJ|$BxWeo%UAh#d?%0KJs+g0Wv=o<0-0V3EVbcPK zaQ=K^2ScXdXBSAap-Sd72`T^cX34Q5*ZGbr!gGB#o*^n-__h=$`WJ)R_@cm`SE`D# zX~3jNMOAC8v}*-3=j~n+NXlKdhR#>Y)2o)sr~8#_IMBD;_$z1yfb$xlePrPAEAV?I@D!QnZo%Is<@eB^L{{u3_|<>dFw9uF{RWUr4wb-T6Q2ae1OZ;t5L96fAX5ze)5h_} z49t-&0>VS~z~*-uVCW5rN;UHvMh4I{2B!5t%$<1Y|LWIQpd)_|_kE1^dIV|r0#y+A zy1f}L9uf0pmsc-W7mXeL@B+?L9r1x5p~63O9?F3ag=7`&COdT180$pRZV58rK8Jab zqR$g<%1~!SS%S47^8fAwHC|gt@vYi{-%>b-n)wKEB6Ge|0(w-DU=7EoPbQ~Fz3TUv ziKmeKnPnqa1~UI1wgWw|u}Prsd_H}3GK{mYI&;RKK@lTI}oBA^Am9yDhXX;{r1haAV4a^t(ieiv(? z>ZK-uLvshjFL>1U$mASq4Y{h{ojg5i%K&1#Q}xPSJtW{N(ud+U{pJfPvbZ7%!z5Xk z+;`VNy9Ti=S0!wEp-O-JxFzl~6NuRu9|oK`y+CftW2;Xrl6Ig$k+|orOK;SXb_YyC zFFzZ}gFd``(2Sej`KE%wlz$T2jBl6+5^WhM;YVw>>eq(8d{jFXsN0VWXBHo-2dzu3 zdnYHhpsBrXrK2_|SKB0Z%YwSGWserQ1U1>%;2`7}KW6e?~=s>F``Vqj-9nj+;7K zBeDDNhu{IRJClTuV3lS;ND;g8O+N$imJX2_g#`x*w{{0FfbGjWs@?9;RW-?ZuUV9~ zC`5R|I%Vo@T8#-kUZdK}B(q6$eVcoI?--YQSO}4YN*OCJ6pBkEt+=gpu6SHbyCLSz zxqwr_n0AV7cBGB75=LHEtZkW9Z-t{+QZf`y%_vfyZL8PjJms=qJ|j%#u{kjc+E#15 zO}!Hk$+xl z;ywFjCq5gUtC#7I$evObX_Nfy1jk$LPo6^pzZlcn8wha5V<5uh)BY)E_tasB({NM( z;6sStj`727RG4J0$N)_&A5J(eh-F{?%+k9C$c~!f)#bb~4q=W-a9qSzeSdbLKVO=6 z5{qAHP>1c(4ZHf!wY)zc0OSMO93Tx9AY!n0gq^1(i2IWK+r7$tf!M|A2O9HCSUF;F z6H8JI2nJL6oWsa&S6Z+x@avc!U$CQ-r;^(ZWa*2U|LhTE{jm)7A_6~1yL2$_Tmkwq z48V~x$JwSgJO6E!K*r|$y3e!Mt@Jq9-?ZSs`MPh$z$Jd`s@L{i>yyo+bhtqA#jbrs5=ne)?S3j3^P6-D;>+Xq!;d3st_S_lL}|NPk#bqIt~ z7XqPhIdu|zGBfX927xHMJbxmi>0!J!e){{x`2^Cd)PU3H$Eea;pM<{rb=LGo+!Gx$ z!wZp1u(bPnr5Jn4R(NQWh~5QGTc*_K{jQX6Qjr;-k8U>t>dWHjlwlAXD73B14!t+pfz2*f|jED;+xC>xbFeio zi^|xE`r2}vf!3?utM|R}Yn23^FPrF{;bJpUtDf(wp~o|YPT~mMKvJH|Vvp$Sxn~Bo zZu!XJJ|8}vM<~Y5h3aM=Z|GdoV;_M?-^z^v?lBOfu|MmK~%@)R5JmPRV*lq6jISHS`A{gg+q8D!9 z=yv=b-HO!SpvL}ET6oRcr3{;HCv<_4c6i80+F2+I7q*uA_el!g-~hPSKE^5^jvcCa z*>V!8U|itNqh*AiB8r>GDE{Z2_Lm3R2u8)Ga@F&lF#_&8-g_%isM*y0J3o7Wwc{Kq zO1`jAVE?4ZirNm|@y0D@u7x9x(;n?~35j~`F8x40X1FS8EgQ^G4kaD+uItvi3CG2( zdMYR=7!GXBhC|n{>L78!XD-jIbY>{UkNJ*NIwkFLxL}9!9jk{j1q0yvCFURS_yZ(q zyfDJGevi(6b-38EzHDNIe?+QjU$@BQO|43$<9Nro${_k#=R$ANhCoSyQEiPy4MH4a zKl^b=D*%7QsoICXxVaHy>*q0o=rKP~xjhOWDX|E_?(F`)8D1sZ-13fI(b}lPvNq%V zpf8DVFg|(Tto5rwflYUksDplg+uou@4KaJ&eRoOiATH5k=Ct>AJD0=p;g;hOk$@k^ z4(tzLQL~)*7l+43TE2;z0c|L|EB8$NYmxed!T!9qU5o99F;lPHSBGB;AdgE%6XoRO z=2s2{eHm}*Oh&~vG`NMb3MKdP8Qa|K9 zfkYq<>e^Q*wY1hK6iIhzuM(l}JJ_7+rWE&3-j7iZpG}z#@zmtSgOt+>ZDH)=uN{PZ`<6Ow zHYVy*z% zcZ3Kud~-$weKV?c`-QdGVGTH1t_Xob?@U`zJaQ}P?TmM7h*f2I|G1ncFIywCW*%oz zW8HV-fY*AuIml6eXHQWwxpYfd1!)38Z)GJX{i#*hVef^mfmy6h{K|@s)21N>_0{eH zfm^Ul<%F>kb7ZsNIb*K1vwOM9$n{-1em(YB5xeMkzxgN~`|dcAg7q3WcC`>e$c-+z96XO0hhmODX7H-V z!qw%y>fNY~9n88l?4+;7uf-`>6}U;#cH^}8acpm`k&cZ?G0V8jnn@UOo8^F%vZ&n| zu#n9-?9%&aalsBxJ^c($z~lezds>}6tC-RY4erfWM9ZCrj8}C=OUufHrJxC$zWwzy z)wqJLg}wCiKFBN)g$;Qe9n^X57@pe2QmSR` z`=e51X)4D!I<7tVnF6~)L1^x)yDVh+)>i-6`FYLamt<~TBo7r%D~&yBAqaVPk26jB zhvjO=Hu`z@Oq{lhM~--too1UZuqk#h6ILn43fgCQ-RMZ87z`{fdr> z4O#u|mTXov1r#J=_AT$ETPOnIh)GH3sT_8Q8Eli?2{5@N47bZnL(;mQe|w*BKJS_6=!1AT6N&gT zwoavefioV~=bBNXmzm7;{Nf0q24jzQZH5OXsBc!YOL`Q=4>7b9vTgp&PZ>H* zct`jn+9k^uIK9``Mk~1nsZ&)!q6ro*8{3$=RCehtB74cLq(;180~+4&EMiY)6H5&* zzlvLA5Wi0|o0ZjGR}%3EJyz{9F1gpPpv^UWmApz<57nK)#V4arf?ou{LRP-!OPxD^ zMLqVxE7RkKZ1d94QjkwJ1TcRWJDRO6Z1g3kK%?a7pN0Bq_d-c^@k1~r8mrvf-iC-o zXo9U@jtvqK1`|N}AoTm|W1`3@*Nce%vg*wh-|LyPr^JO4$M3l88O3&E6*)aVxe_Ee zM($<08+86vIq-^fzPwoSIM zBM1VG$;7T$p>aWW*1V2snJqN;S>a3?E0%ZI69ft#9OoUJR^iaS`qrtgWHpRU$8Eh;9k(`tB$b^~@bZoXQWA%G;i}Y|_G)@26oF#9Y{SY_G48K!~DOf&)2OdC|!kNlG-Oi1-|GQ;! z1DKw|Rc}4aQA2WK1!8!vJ#zeb^A9ZuMIm2ek>}{}rU${g!-9b~w zg2*8@*3s|wPC(xTK^fn4nu17plasUW&*YQxXzR&#sX{aCtc>*^fpZDx(rRp}Y5nqp zlDG~n3L8_{Q%{;}hG?n^)a{&EIvWDq6h@p@o{5?+)cA%N@x1n6H4|Nk&XSE@vCnBR ziQAyzq!&mz(sv@z`+c!%WAgKX`}aNp0XYia`|G0OF3)ycpJ@^VFKPz1m`=CtWd_Pp zF~K((%}bSuF6nVeAThNSvK^C76uxSHdzu@2JAP3mGY*7s#SSmtSlg{ZAS-jN@Sq9W z$uAjZ_`h!G_Im9`P0dPf))qC2hVSqdOT$Mw(ZVKJuj`IbY$hbz#(~P$X~iWIeu_Bx zj~6Spt!nJK_Z417NXAXQ`0VV7&96M4C{OkKXs?;`!4A*e!&Egjj@zVrsfWFh$~86O zbu2#FZ?Q{z8LV7SVbLo^SDC+4XV2T7qoXTm%ETY>x&4hwBOMYkq8oqld(dV6rWlXI zR|K2-m?lZmOk4YZU+fjuGYVC`$uKdDBtjeoDq3j*40BLYvg0dLZ$5h?TjarA5R@6c z9MLOM#A8EW9m^e@y4oGO%HK3{)0|@pvE6o?Xrb}r7YpgEG`x3+mz)2l+4BS!+j}Z& zr%+x$V3p#f0(y7;NcujcL7QU7Yb)8*n-EtCg6hy)gg~s~W6~h&PEJ+TFiPV>jerlX zvNcwr6_M!5lM`=S=1_l?em*$^?!ZWWhn=!tpvAmwZTAwZ@cp}&csr9|UtUxeo0k@} zuP1fme+XC+a$IZ`Tva_S-fO>CNR=ziXYj(FjE&AK0=oxd|!`|Ja7WHv1%(0=5jzUJE z06R4|^#k9lC^5~zuea%}VjJF>V`FXoGIn;6`Drw^zxqY{?upr!YdrF0cyni7(gbx& zW|Ef>p@;OV2lNAKeW+;)5m6~g5U|Zm4)k>vqEeks z(zcH>^=Ts|1oHk81sw!(&Gf`wKmp&MD-EE=gHkK9UtAWvwzvHZMM>hNBE9{zrZ5izN9J zTmT(^k=xqHQHUV3Ux5<>Kgz|dsrA#T!@XgL9S}$FYEST7>5Ngn0k3i+Df`S#D@l$a z8W9!>IvYOYx(C-)IeBwp@^kz=wf2ZnV)Y>-d4(!3t_GR_Bkv8a07!R;gPi#_1*{k z?xcq<3&VVsW2T{OVikqHYj#6`QizcZCUMdE`sE$t#?`Maq6}Uz#Kgj#zE({tYip|P z38kO~;VYW8`1RTg3p8#U6AmQ8Bx%iq6f9_G0-* z`s<37d>W6!S5C;OJM1Jo>XxX^Lygntmz4Uv%56~{t>GLSU}cHk>(6oe0i)~^Ze_fM zb^U8Um$>?6KUURKdR8@S8FI6OH$Ihr$)Zh zrstWVn{`s7l}?o$q{Fs-RG}a0pLIlnq56tzmX@2u>j7=<_T5@>J0#L7w*)g*QqeHzCQ12@F~}2-7ooLnk^DwAx}y~i zvj^Fqv}7+FV3t3wSl#g8V|=vx8NavM8zbs4QWgEUzEvjh5kK0y0(F}K_^!;_BxpkS?Av9(gGmI^dt(o} z-$=#yGe&E70vkLCSo6$IbEkU!eg{z{(;((v&^_OECVte+rDq3q%R}hNRtka>kBO~O z`8eU=PSu1^>sIo9P5T9r!)~ z;!rg#x+dQCl%aZ7%&M0bY}e zi#>&8CZrA(S|A!KLNfwW!>URUB){6#lFl)^0S3kC7r}D72Z}J9dL5&h?*F9^n6-sf z%c_WNSWwX^yE_2F<=7;PiTc|nNmp*e_wcCsc#&2`XCEOx zUh_7HWP|!F+bu?Qf(8|!3odK|0z~!yQ2iCr|yGrkfIgTwkVug}9uDy9N>79hUD`|uwExTgRXK*Mt znY~T-o^Zv+uT&^IpVBWHDlm%QyyRaP2BkW+giaDq#L4i09BkiP^BgQKnhidtHF=8q z-6V0pa?4^xwhwSzwBSw~lvq4QK$)h=eia`}mg1bY09J-0rBF#3Ibl~x4I6}@pT2G- zsa`orrx6BA_Ky>^E+rN^lsZp8Mysbw>8zMsRfSz(=TTcUQ0t7qnbz-**lmfwT&}RF zS!{QWxAV>wz%=;c$7a(vdpG-IZAeE+q~#1MoT&zH07Kw% zh>#BgzKyyZh(qP>ofa4%2;c`Rtm4|6Vx_doWyAsYw*2j71}f<#YjB(ShV; zC-7yDj=Tw8%&c8g$*eH~ba(WAm;O5eOy{vx0z<44d9mPxq5%+(Iw!ZJ?9=$|nT!$nOigV`DdCz$W*nkdBy7kllk z=Z8R06k?rR28 ztdoTMdf8CE!6>`?*q1&*0Ow#$)OkfEC4v&OpM+)xR3x|FK02C+^ub)5d1gr6( zW+@VW(Gn^)u_RuFcYT+_K+j{SlqiW=Z@{k~mT2ee>(#og*#qr#C_{Cd)?<~xqptmnyu`bM??V`MUbUim+O&i~I* zxN6$hs3^VX&p!bC)e8G6M()}qw>|(3hP@S3=Y+>By$&)qvNgC2s$Zug9|(}Sg!F_u2X9br7`p_}?mLfvd4`Zf zY`dQwb-=(dII!)LEi5Whv5cqz`_3vl-3W|#P29=Kd zhD^^D~ouBM&!sG0jzv zUub14F5junn(is0|1U$3NDra8ndRu{S3tP*N;whTTjtV z4lPZK^q{;QpmE zaLxF?vGL=!O5$p>7U)ITgIqwI(SJks>l;G8xLuDbeKO~-8oEYy?Pu0mR6)7Q3F4R) zR4(CHIqhG^!vL9nX4)2T9dKzd`O+M7(C6Scgs%?wNgZ#>B3|YHb^53Z*JX8zZk8~1uo@XmYZ+MFT_&@$7=KP z@VuU08F!zAv$#C9B}dceArQ-X^5Iwc4at!HZSI-B_D}$6v_x#u2P6fxw5J#{Gv;{~ zK)z4J2w*J$Axi4GZVZPjyKjwM%*IMU6(!v-x)z3GhrkVoRK;`gb`tc>wUkfyF zXu-06LunYUIM4od$)McINxxChmt4}3rG6dfD(pi99y4E%lP_ikx;btjLDgtTQb45r zPBXS7r+zL6m>kI}_<6t|RrFF8{gD3!xhOeSdqN<|IKZR%LHcvem(f&KrWyhW&52#y z)j88W$LxFS9}m!BUCW4f+|H$h_}iAfp6@uEchYoCkU@Sq5!-3g$iv464?cx5g^$=| zVgQh`Iv987Z~~2V&&Ou^%U%J~Edd~9xV@o^iWwjqU^@nToL1txt|c4}3RPN7fU_JC8a6fpB$_lO`em z9q7Ea);%tJ&06GGyCy0|nfDIG*xI0;c|ayDxu+qJ*GgR9mA>7(J~3&#J=boCyDblX zEBr$Cp5N26MZ)8trR^*NcG9e7HS{FCR?$j-dI*S{uEKy*^THBumUO>*a+xccsp zfm5}!mZs($!?~Hc4x7gQSX!`KZk3?7Z?pK_B@C5sfCd2MuR+vQ7vXSIohtRr{qb6N zyq5EMaCaj>V;z6{h8DhRy=+x=Uu@Ts*Bm@?$+{|tTO;FVu*n^Mezz9EWH0dBY_r;l zF;EX`-aHj~64-O+7I?R_>>}pYjjGhVB9`}(-S@!y)Iaw(rULJ)HS%67cT7}ftOs1L zE)F=7peW!q{O~mAbj3;gi8@SSfGu!1rYj6})9={EM8ban^s}Bb4t{A%ZgTVY=Q#s} zQ*<~rk>&T0UKb^2rDEO1m&9+LO`+1lS!B{gX$K04#8;3VT;lJWJtbq22bjTeOVG65 zB5K=KJ#bt72BSx1O;HZ@rz?(M|LyESdxl+n&<_wgTxkOE6kX8#9iPhjr-Ix}vw%Rp zzZhXzJ|Re*3jhn|BNOj2&u^*^+_xT!Gx#hfxVHvd9Rl%V zPc>5H!jOLCCI1#-Cu$sgo%)>I(}Ujg{enEN@$BtI^TDr|D?nk!?`A@c9F6&J(dm+5 zX05${bAOr5z-qLC;|BqWzr|`#Mr!`6h9JPD()`)8xcBF78WyPs?*rlENb7Sj`EI!V zH9O?-yuf%?NzLtk&HRHZn$<7jgt`5msgA>hJ+ffI=Z*TLBzcxKo-rL+t#a~KtM{4Z{ufY6Ee%w$P@~uF zoMb3}jfGjjcooE56)R{3lIvDsfMm2((4a{><%V4v-H5C-aKP2x>&$pzpL(kI*BaGz#ABsOb2W3mG`&} z>IJgmc_M)~JhA9ad?e~T)p&SahW^ppIPUvFPqIJ-Eg(f8bnodMQb3YLHFUoIjq~)* z4u|m?RAB(?t<2~0X6oGL*@1^F4V%=wpxDC*D?0oAiULO7-zI;i#lB(n_Vgp~riI|dc@uhKH@ zHv0e?@e~2?@d#ALo9LjY(+c$Gs_Wif$&pv40&`67`_jjN1ZfN8)G)@TX0fgQe)4X)$ucew_nF%y`-3Lhpaz(fkOP+aV5ATbqKAI$GVH8lb)Ka# z`f^oJ6alnfBjPtI>G-P}m7xSz37|@>0_C^(FERqARs%u}8+b-AHJn`r{(<6FCO{qj zkJIRcRmi31gD-%d!dI?QkUu?>Z&=;$wZFO?Kt@bskW|)Z7dQWw!A2mekjI0zv~#b^ zzN>P)mvvR1F)m*~bhLa776B#0ogc;a7(w#X7o9e+W;rz-^C3v}?2qh4mlR6H!nnRE z!@ITVx9sL<$RlgdS3yJp2+HJC&U)YU?HZ=Et?*rBo~zqkLu!i`i3~45hE7-Z|6huV zQOs$GzfU@Em@5#8sdH0PZy40LX2YmaKkNhb+BB3;LMYd&uX+VlwZ#}`y%Udec z^Pt6OYJ076((O7}rcat;9oUb%6Sg9jU6(2uWr^SLFYf9x5z;W%ohlkQpH{_0o?W1Bqds&376$`f zfANRwTGOZ#a=f5^UtdfGZCUeACDIHhr`!TwnOs!uMuQ!IO`Eyb8SC&Nx>a6_Ziwf^ zcAG|82AT=y7WxW{PG+$}P?+|??s|imc~C99f&;j5hQ(?fl^gSy<-L?6xBf6anU@@w8S+Jt8 z)D!vID-|Vv1TQ4(c2`6}u`>$0V;OvaQ{2%|rqoUlhw}h1`Ta$_s6z>Gff^H3v6(#h zpE!vv+M~58{?e;fZInYL6U6@GsDU48jGF-MS@3G-IV%;C)d}y7dUPlsC8V)E`K*XV z$Hht&^{N$gVUh#63cr&SSz|1A7MWv3Jof_y+;+Gqg-zque(^_n}yKJaMT8=%{hj>+R`uTtH1tKucX2J!S`arR*W#~G>rW-iBi1U=R zSVS%K2~~?Q1d>|hzF~}=L?WvFMa){yAX#lc`^Ipk%7m-(v3~P>xv!{T( z9|3T4ya?JZ)RkGfyBUEl6&YilnoxabyHjtbrsy(POi(Cy|67NiGU%-Cm3 z&V6XA{mLJQ<^FUxj71FRqcfOPIXb(^hWfR!>H+;qM+f(@SMpIY6cA;GAg;Z9GJ2)c z1^}=!TqdxGLhpJ}J`yu=>`u=622 zli|t(6VTBt1WZ$U-TIg()GS?yiYafxXGh7P-m4r&FY=u-Dj_fJ zt-2)Pze}WTes}39CMV$^h6UI38bO);!Y+Q1O;wgp%{De?MR4*RJ)8YM043YJZ;uj? zbXKU4sue9W3k%m^+Qp}_f*uX<9QQ7`W7446tLE)Uy)wpd1z?^(&RnElZcFB(F0%3fr?mo+R}aVt{y7@&)xXh( zTQTis%T4!7r>=Vi(wk9zVvhKE@q7vx2E-V9&NI1Zsj@RM6$}pM9#u!*HOa3ujtXQv=YDHvRr>R+W%@Ros`NKLj#$i$ zgY>g<8}WSlH^>Dx6|mzOmd@^h-7K}Vj&3T*MJc193^Ik*UH-^cvr3O<$n#ZL{8=D2 zoSlg2NhkI&%IEK)J_a2|1?JL_3#17C!`qrsjfDnHhEDg` zL3zG*o~%xTN)K)adb^7~tS8(Gg|XY&vlQ41mhl-lMLG>oVL@AS6+o}D5dNCqKsH%i zFoL7BD!@1>K#$28{fAntZ3?q`+`4NJj?8s*#q_AHZhjsehP+{sOemT>!-N76B+uv-@b+Z7GAi#abM z;=Fs5D|H^2oUXQ<1HuNXkQS14ovcWb#ki&%(2^1b#p-~-aL9Zt!d!s!>0A9xfv|Z= zW|_fl_zmPFf7KMFd+{PrR>DIbQAyLMb94@5D7mC4Oy@XFHdNRQ+(xoatxno%@2V90 zm%lDTmTiJkMFHSps8Kp7{H(%v{9akIGy9MSq9Og#JR5}D%mq(%+&-(`gQYLgBW%`y=civwe~&iq{m;IHP7ZJ`Ywb);wq@Hto!v40xMep0ij z|D_E3)xnpKlLcD|!XOxlR>X*r4}cJqcL7#x4>~mfti!E!+T4;$L|=R=H-tZIvF~LC z?+v6qfBnZ?>_LSb!)fx2Vi&1YnO5vXrtE-0F^KJ$A!8u5Wl_6cw@fg<_{mDdrvFxJ z+2dQx^;ZEI5srOpM@4Z~p0bNOf3Go@-MbHO{K4FNJdr|kme+a9#J0SG6HcYym5Y^9^@ zVJ1V%Wx5#c83AhYuih4XO?h=PPXpAQ{*RE(E?>+OL-Mg zx)MP7k5K;KFOzlvq`_Z6oc1AGor0WD{#NfHP_7K8D3zm?G^83PrZT(sKVx~xTPK+z zix59?=?cRYVe8L-8S=@wuLWX<`}1J`VjYgtpJixM=K&_53vg~nl)re0s7?P*kjSjM z5Hz<*^EYRFa0b~oH#KlAq~eA59etZm4;=; zIV4~Ox1-kA!oU1Z7j%00ule5Xvg@F=;#Rf9?~0EwcdgbX%(FsVxCG ziY0WKYEJGagiS1+juGqC;^kSXXS&Ak^Xg_c8#+o+S#%Y5kr6*yf|cL&Yy2^qGqI8xBnm<};kr!+y9=`nvn* z-t3&3ar|+&P=P4TAo23P{-1QkpgVCWmLI05F~{*iLTH zkuF#rHLrLXfJsH0rR9d#VDg+IXJ0N~_%O|VXPh;TzOLE+xVOu)QpjMU*B(n_sIqAe z+BDLv-utV?B%KM+(rl7d#pZ@3c2b0QbO|c1d>ddL58B+h|=8_cGsi9E%BJRP{MP2 z^W80IIK6Lj^xeJU$1lMV!z?CO4<|qr)}50W@Ij0efVhm}_OOK79(na-X&lBMS+h2ni+t2MrMPh;1Nm!V#t7}#=dO3-pp&`CF`ebcXU zPGM7@26J{LV1jNEaJ25%`C1@-l%^}j)G(}N8SOiS!5_G{updv@@A6st3VbN%KLYMz z1a;WopSPSPw%G3{bwwYDjEWz4=#F5a;%8;S%2mlE5cJFLvSbAGgN8`_Bku|}FNz?G zfs3Vj4ra4Y+P@Hrrx~=*+>6JTVQNWxTh273&zBOL(^Y1}gC!0VFwG2>*fyat3Lek^ zOm0<&@gy8&D)xV3mk>=!^HK$9+ks82Buag#YcKp;osu!>)+32xW>kjn{s>Q^+oGkh3D# z9MN$2m;g|3*4WEBH@r1gYgUZOxeo4ln-GrqoZDTK6h+>%MJuzIFR^1Qv(pSurrzllY^D;HbCI77w0gQ1t*odLyMcv%+!Q zHiVh44}__z0DAFs~dW9VBU1_ ze0p)Z>&YbjjKS@>;7oHl-^_~*uAxxz6f2eBF6Lvtib4g;jgjGbT~`k!Jb(FlCGTrP zrgQspN@gt=5ApFI6$*RXTED0Vzg}B(wKQ%IvE7Brz2kNv^;=*D;%M+`6gOdUPocH=2PG0ddSzDRHk8uFy z**67H7f{vo5Xl<4UH%Br_(_A$W*RGo8#Ltw%SeX!o-Ga>LSQo7W~scGU_!ENJMdn- zwJyuU!)t?y5pvNWh%f4g3pRBq>eIQoq0 z+tH|cTQb3+AoZXH5xy&ff+|)mas19iTuV?(xfh#>7R&+TyPrve`afhubPA2v0>UXj z2Yjad67Yp1I)aIskD8xafchTweQH7K(tt~_619s%h&jV74m>K>gJv7-j z*iKEn0d|>GdI6tc0*!aH&rEi8QnhvRnbB=NHkr)WN_uC+;T^Wj80)*a+Lp3Bt{e8- zrtKAdO2ZF3``@m6u4vlT<9n|%vr^yWAG2q9{l-fe7|jXP$!hogSta%ud>8=x#rk}3 zQdK15ZNcKe^6oIUk2{KJhyHbz;P4}w2VszNcl~?SxQKo(iI|n%#mL_-C231Nh))Ug z@FJ<&z>hCOa)SA((~8ozl6H0SWyxYU8Z%(MzH)jcU_8og-z7{o5U`d zcDw6-M>Y5a+`8xwD75!q@W{%`UugWU(;&VN`?TBBD08 z_X~%isEXnles68x~Au%3^5*e?DUGRrr<1nopI7U!xPBG#qa8x!nYsDQlFy{Wz(WYg^{e7=s6Nl; z_Ucu&;6lk8Z67VQ>g|b2iQ&BX(bmq@(UFm3n3Q9uL3x=-zL`Z!B%gy>ZFI%XuZ@v) zq_LH^xS^FrLP4o-LSbo(T=(iTG~JqYxNfmAi=~^maRYNuK~IwS{e11a4zAFutflU) zz>vpD9EKMe2vezNmuX_?FPskyQRwFt7CJZ~eJL)w& zh7U1{?JF71m{7CdAlbj|#_4@gs^!$q7}GVsP`GZ$Uv4!Q-OQ$8iF$GjKZZ)WrT&ZCM%`CjN+0;4_deURwJI8()s zs-N0QaW)|mO6BZp4RdZ42BXHdxe{T;o~LYTo1f~m8YxDr?B2v| z&2gTF=hW}kG={m-!XoD{6i-b0IbU`$Fm+aIvjm$@<@(r}TVX;*hdq}tRf52NU`%y- z)uzjp>rUb&_4-qNjv}vFPywoIO5-jsbT5J7RrarE3Hd0~ALmb`f|1}V=^v?K6@qNi<1(18~cS-+v|bh!+q$LYUfSj2Et6112OXM(f%vXCCqFmN5u>c z$s_f6V^);lelVIhNeb^KwiG(7%N~r-APEV4I09mHTi(x0vg>$I`w>pafY{jU0LLOK z`BaSAcpZ=q@mC2I4pPSZQr;=N-Xlq;`1OO|N^ic>BQ+wU;wLuoPF#c8J+C(l7f0S8 zjhrRTMLzY866bWrrte1)&y$N%Wwpgcj1v4Qrfz|ZAnm6&VfCKR6X@6V;_aBLsASik8>51DFYD~TX`5m> z@D-z&=tqc(Zf_8N!#fTJhJ(FP5iG{#M+nb}g-1Ji-A6k@1zoo<4@`z@Bz~aLSa6 z<)^6RVZ!SjG?6Gex-|yp>Y$q}&KD^E6ys1>QD&47Mq1nc=CC$4RN>}eS4;D=qo=75 z6Lwv$0@3lI#0fX*X7BLoXezad3dDRw>vZ@;c?p_?+IOq}6}G0kE^rcZiGsHftTX7a z#|ENRKY;N+)8Cgmz`TWgF1U-> zrJ2tpIV(O+Sm6nVJ-RvZQ4^=TbLg`-ju~06@xbiw!Yh`0k}E_Yiba z)L3EC+RmV+&YG?E;x7gC4HnZ1#x*ko`HQtBX7v@oR9cC}+jAn2Zyt3#vtHk8A`7LzBi9+2iP>}%NXL7h%bi=N?Sgg^h0LMx>ei zLul&v@HpL(vc@gGG}hmPCnOHS9=~Xk(Vt zZ=kkohb0zs3N%EN`5ZoVclWrjr}xO-r@F;wx&%%XM2PT|&HajT(1_<Yik!8KeC&st7fZbgFvn^QEH#m zFSC9LdhC^e;OGOkuofn?sX+9Ae}j1 zlRUItd%WGr{deMMu!uY<9|kqjOoGD`33SfsGKPBs1O*@vjdc* z7|E#@ZEnCj_wRrfyNdj`@b)p4lMNltuvti_irT~1qLvGCq;b$e@O?j%?@a6 zl<%@4Y}?A>hu{Gi*|qd_kSlJ`^8J)E0gW}3ysi4|kdZ>wB5E9?z3(UTT&k|sFi)2M z>V-SEj)NiP-`paAj8NLiU)!a$F;d>?h$Og4fWL0D22Meh0_CF(3%L6qprY9&9nW#C zfg^Ak;LHU#@siA!<$=smEgd5ZtM9Mgk+sSpz)TO#-X`@TVL5FFLGG(HgqaY55%Lg} z`y`k;3bIYyoDSv~i9sG)NPu$)_J^C(TSr9fBp8nz2T}u=B$KV3k2bIeb9wVC8D0e+ z*JGa7gB=`!qm8V{<96heMyCsfKChH(mW@X~uaqKIaP#q%V+)ZJDA4hCscw65P`$-L zlf@T7LIaJYYzO@1p1?J5a8!x3Ux_RSU8hkb0!f~*z3Q{s+qj&%Qrp_3T$WwRXj;ef{^m{@-!d ziJ)w8dYZ0qT)M*5?3Ee2)hFlG{>d%A9$S7BxKYbod|8S0pBXRh%>O-R*y{S}zZJvQ zrn0)(=J|d8@dXE2%Yd7H%YVPPxEQqX_2~4wz-20V&+YQ=SZHf&cb}ggVqH+0^u5^$ z=;qUxZ0jPH^@}gtuo1Y9rM~RYU*ImuLlxS6QRQ~ucb-2w{b%Xn^Pw02GQE5?xBOn^ z$I=+!AxMW_^DL>-O+2}++UVuAuA|mLEb%wCh;x*Y5nWN=ho>ay~z;g7;4Ag1F+o*~8~wuj}7^eym<* z!8VV7@$3u2K9ujta$TA6f<=}MG?zWM_?+d(%7@2yKAV-j1~|L|I?Ss4hQ{5#d-JEI z^vh3s`}kzK-DgR0hnl+Dtq)!l%v`bWmow|1T@PRSyq~`A-p}Xr<@09(=iWn4U7z`C zYs%)k%s}7o@3)ga_<3FA=4E$2mCTp^U)XS~$=@!Z`=|AOU}`@OTz$-JUGSh`@4sKK z*Ih1I0GuPZ23q5PN~pwx|HU`ZaZkCWC1I~8^Qi-u|LynMy_o;L{@uszORIs$2z+=f z|Nlm*%zWS>Ax`pZ48Ar4bJ*PlV8<7 z5}x0o&u#Cv>iVR!W&BH`<^S8gn*a69=JUJ0?*4wS`n0Y;H1+>@!n}3H{?Ebob3ikz zQYIM*pve^Aw8^pQz+(68zF$|@?{aqKvgdMaod563a^TK}+VW`GwX;n!A4QpNx$L(c z*{KVR-vehlo<(I`IVAky^=qGd4SvEXd+qn}`aka19b^8k1QyQE!Wjda2eb_STYpneF!XYHMgy=qVDNPHb6Mw< G&;$U_t*}Y} diff --git a/keps/sig-node/4603-tune-crashloopbackoff/restarts-vs-elapsed-new-default.png b/keps/sig-node/4603-tune-crashloopbackoff/restarts-vs-elapsed-new-default.png index 0fea89b9d3a1e5345ed08e27d303055aba7a1fb9..d468567898309a8401769732e3e1805d8af0f471 100644 GIT binary patch literal 16381 zcmd6OWmwehx9$Ln0uF)-yfh+$v`Ps`NOwp#3ewF00#XK|2nZ6=J#)A5C{aStR(js0wFYn zKnOf96N4+imVBxp5UFpE`Z)aaEo?ula+T{UNS+RTdHoOsyNu$8hO73qk_I%WS6C zl`h{5pq*IoIKVK-c5JDioY5(8dXd*7Av3PN>a5>NLp>5;)y-G3MSgDy~x`+IU z>2^lVR&{>9(&NXEtvf#6+`-PBo5MBJU3|9R;3T=ZxU6@Vd()U8!E%Z%4#BLrVvS}4 zivYd8GzrIaVb*}s=UUT;Cx^}F$ga8OL;>3=vlmaA`LVaJ~dLF}a>Dd>%>b|SF35zAQ zvoUYTXhIpDJP0~#%*o5^UN38Q85>kfc`(R8wl9{Lm}q0Xay97amG7*`>~mS0{xs*4 zqJ{RT4_$WAip&AlJ|xGcCz90s=1zfpCq{dRSh{^O-)n0xJc>46S?y~!f|l6vwv zWw|K}vTwg@(~2TZ)l_=zSk}rdFGh zvhq9h=`#pu4#ghYO{O1Duw4yFS z*wXy28aj>@-$PsYE>TU^d*-Ex`)E{DRP+Q$q(w*1AB+~w?@c(CA=xJU^eR~%Z2?0B+iMLOJ_$ma#8 zjbaqH!8m?)cbcy0Wf-1IJWKraX(7%35arW*4=O$SZrM}u#GLV9p{-!m$F?$R9L5a! zc(f*ADSGR&adyZ0NP*6BU1_-c%HHGrCJkwleKE05&5~m*a`sU%q_#F(4(n&8=6waCg>! zE73~hE~8XnV||&y>S{XOO6>I?u#r;s!50|2*`f;OEvjC;$(F53LRhv({ac2-MEz#X z2QVL4pDTLpdFMAF;O&8oI2rQl_;w6>Z}c$pNuSX~@iS;ux`xCkThchG7thGHouuP~ z%oFFyith}b)cMl^DRYW-a5STsM1DIpud($kItVu&R8MnZMP}P)*0+6T?BeN^>-YCm z7^^n?+U%Og;hyVF&!uX-nre4MT!kpT*k>wFq2yuwvctv0BiQc68!7AhcC9Fz@nAg% z4o||KyeYHzjWFln=kMRNwGdj_n=60vN;FUBRkX%cg=mDfN&R#hb%A|f1JP?T zuFTLxzvjBSiQTXP6;;(^CQLU-mJcTz(oXsi$ob1 z>^AJ0KhJcm*+7^4YkfS%)+ZdxQ@(G_dI|Prd2W4m$j$5sWMoty#EQwx4+uTdkn(>m zH!deyhW@a#LgT`g<;#l=2o+}nIKyEoRq{4BL~-3}60Fc$Brz8QN5%t#t1eSjoeVx* zLTnry$x{0c%}R2+_1OVs2McB0xU=N9co-PsmLC%5Z^&I1Klm-co8GsVtM(Nf88di!V!e4@R*{==a1 z5P^uX4$-a61vTtyzlDWCo|@NSU8(%=g16St4z-_YkjqJ4rMHS~*x|7k!3(ymtUZ0Dler=_A%YN!jaU^xanWbT$Xi=5;b_@mnW8sGY9xh zpzg-I4wDtn;OQFjlQ7<>-G|0#&c}m|@i@sMai&#UXdXOJgY!y5R8wfY$AFje33dP? z^yKRWony%xIMk}ATDl~%%RNs&KA`b$|8mT4SKMtFKBpZRyti3IwJc<6&nSTP-NcG@ z<;@QlT*Djsy{m@Mca;Y!c@2>aqEFKS{Rz8Fph`IW%&$7RGR@}f^w^sQ0tpVIEYI;! zQB2lnD03-$2reX1l(V^cea2+w8Kh29eFG#VOa8xskc8bz-~(F_$Cr9l5Mrlv*-%|J?Q`p~c(6xaPM<%~bqv?|GH_$J;$Ms5;PYel_( z|K9r7w|6e(70GGClF>(RgO1*o^BNoM{?&YAlQQwS$+Lg%4E5g+JbjLJOAYv;WMK;y zz+<;pq2dL^iK)+R5cutj&)cY#Xf0 z6m;a2U3$yaV9&fsPlDsQDzW!*acSuyV;Na=Q=hsB%34O>86Ji#qgq`j41Q>`J{u0*Maduj$<742PnwdPwtXr0+MBVcEkyb~0fFI%Hp6iB;v^T z*hKLFq|6&G>@j3S0zX_p?wD@m{4?BQ{M*)hD}#Xku>vL#n_v+*DRH=Nac=x0lF^Yc zc=kZj{MGPGy=QK5agUmj_vX|nLlAC%ymz-CCx^{ki4%&?~g84c?pg)7_VQ5;20Pt)O#%S`n9&dfcEQbEk7HihZm3ys2KwrsuC+ zfoi?O$}5LrTruhOsQxBqjBqeQv4wE$UFbD&r8kaKB6OKerp>#lTXos`Vo9UfidPR) zB4bh{1EyyBIy*^@pXcht^@*CjJ})~zDO*0!3_8&l*TgHDaVrlJ+M|eXM`q%exo8Bl{*Y$gP=H}*0+i%{y z*}(_Wf)aD2Up;!S-tS<8lS4BtR(z{*IDwm~-JJfPp>5eFinylHc2-2K_bY6PtJ5SG zd7}H2Yh6%1w!=Sj=GF2J_xi;#b!{Jrr~Yj~GZ@>st3LG7s9Tl<`)m>t5_kS>rM$j% zkz6>rcvj*X&@C~J^!M>pV-)qs=~@8%lRN0R+oH!VQp44^AGa*hd~eu}i1>2OnYc@VW2Yg}}+_EKHonH$ZuQ*85jL*ipE$%Ta|DLz-^e*EcCyHt%IvF>4I zF~*A(m+0@h?pNl9-sp#BxKNe{`&IZKVlr5$O6}+DUY~f3MkfPCt!i&yKysz|WVCF? z6T#$o^n2dMdO0`2ym03^ca$@%%`J4zE$ zWXL3~GOj(Q@pN^4{dn4fAt3`q%S`5a4-Bg7;Zadvc1smOz20_J3Wj|S%<1oui5t+T zN%)UFw5Zd{#qI3wqOg!PYCHl^QBmFZKiJ5)3m_mnAp6N%(%fuDM5I!g8n#n2@F_-I z@XCr(gvKlT%e#{1G$!j*uf$HWT6_F8nDY*=GKwuS6-B%cXRU0eVSsPI1FBrvWNf|VzI^PFBBs5ziczc2Mh z;_f*BDf}HG$o7Zei{C;5um_!A-;k-J*qr@LqP{l(V*R=?rgV&p?_7=j9nT9nw#|6y z;b}j$egf%6U@;KL5qJ+Isse=6AS(1<23a&j^8d<>YkZa^MwJQ}#5&*cSDGdkq~CrF zs==oL4bnI+^(4W9AqH{pjgK0sV#?vvyg9&BDgtpQVSQjs6X0D`Aa*k*Esf8%KP^sV z?u8Df9?f)^!W6U^t@sehy}vbQ5elVemzS5n|NOV*>Uf!j92yMWiG-%S5iqn|uk|qu z&cNJP-rRpvNHa=BOvdZiHxf_m&(>E%xvB~O{EV38((cL1SygutXUYZ)4kdQnr zrjnbjADvRUDsvJ+FEV5&NEe~tXsO=ZhKu<~pqfW~iJZaitJVXt_g*Nk@%lpmrd)5Y zG7H!4Z>Ng;M6NJ^edC=cjE}c}j)yfupJekV`8u!YO7%;OZ)-~I$NIYrWIVbLThZd= z}K*TC`2?DWHtqUH_lzm%XT`Fjb@4c$H#j|h{gKUL}7@|w$ z6L|~Q@aoMsUNfuQd56W0=n9`Lb1ZCz75G(fU6tcFpU=I9Sr@>1AP^SJid|f!v3=cC zQ$Rz27zVEaNt+IO=_6)U+Tu!^9m4)R<+4Kc(l0k7lpE<$p85B8+!i}1br!%wV~q|S z_=hrPmzI_~Ki{)bs~cj?f->UAFj91#L!)Ejib*6)nKtZDo+{=Q1ClYNLk#K}?qt7O5*SdH&vFlK;@u}NNen=siVLUVpE9GXdQE0n zI~Ul1S6n75ZI^mQ=3@OZ_3aYl%|Su5Ahh0~qUr>0m}}7-`1e%FypCzdZq2k@d;0B0j7z~I9G6+v5f#d5Nn_I|!6+m|lo z>Z5#_gt5HJ(CCjhJ5N=)>v;U_wuNH=8KkZE#9>oa_UxwWfN9NW>Vk)7pD#cK%GT}1 ziWB`!${yQ++a>*A>yqLU61sutp<>r=;Q)!iq z5`MZsd{D|B1dh9@G92EPa&fb>vvc{SV1wpEK^*y^sthqtDjmZEH_L$mDb^G@1`_>&4 zw0A9~|BP{6L6?k+EPNtd|lO#|m; z`CMvbUSa)?#K5=!nUT?gi5UPe3#NNg`{LN+dMPaGQQnBQl?*-$Xu_w6BnHaL${rgN z_pq>R=|5ME8oW#BOaCMqE`oSdY@sn;lWrT$*UkpXXS*aS|BsuLI0GF6YJff5s?`(` z!+WC^w~>(%yJ-Z2=wqaXX-((R1NI8`zYD+?p0Ki;0t@MrgxxC)w<7=HOG3BCHq`Ue3Kxh*O>`MO9`^7xz6@`vAl9LpF_+eb!pua`=m z9a(+VyMVvSQaqfe*?qLV09>owGQ<=C>%_dI_;V*mz_w2bB!a;FmwTfBm_G$E-W@OC zkxKa!d!id)?L75v^Dj{sN5hm`9ENg~BcK2>(km+N>!2OhMhbvIrc0Xrv0z!~(akgo`KTLm-06Yq>z4A&JY6O<5vf(D-J*XjAWufqe#=Ss5=ZYAJ z04%$9#@u$P8-n{of31BJv_$#L1>Ape)I5=9PD440s6 z^+VdKOUe}WpNWfs4}Rv9Vf=zvPI6rQLGK!iPV(31UcG4BoxtxiZc<`g{l3KYDI1(XwOjDPzCcQF?%{Nm{lX^5f1j$=A_ucF>52j9 zS#QlEy=ji_faes;VXZ0gEAw)4KD!CI|B~apbEmwa z#TKkCU=};4yZzk&zIb>|Y6PFrr3IeY2Yvxq8x1TmIaN>4l-IC4*x7YWNsR~(w9Irl zRcUd=+)zWn8!8%DLh&}j_E#!rlAwJIi2=X_oA(r6V0x-$x+@c3vg?sg+KB7tL+NvX zpWgReRx9EF1%X$5=Hv1kbfqwDe|PsUsEb}KU_BJ0nBa>|W8-!xM0G)ZX0sF22sodV zvCEVFwNV$6&}ZH8i4|A!nZdXbJU`kvs7C%F&2WZqx}o(8n+@{V0Zmm((uJ z2A$(<`frWqs?tw<_j{7=^fchouua7Ij$ia@oyPe}fvQ=6;6cYaaJ!u2|n)cU=#Ty$=0vA`Invr{Z zFkis)Hrf=<&v4Hs%Add=to;))C{u|b8GNfg(dN?dR)>v=FBL$gY9RY1?MAvC-A5#~ z{q4GoZ)jP-rX#$y0Tu4&h1=Yu;R;_zhxDcOs~ zKpHmm+h4mqZth<)6L5%%!3(#;(ENRgLg3-{qlJ2SS{f#ldN#bo*%zB(o?3>*vIbtv zqhPlGIGCwchQ}6~=A-Xi?N4v&wm8f+At!~U+a4<@gaUN{#LQwNfy0_Jd7$yx|M}8w zvpPxF*@xGQVUMsi&eI!wVY029-xPsRACK1?;DyDQcd(=1PiO`0KGKf9uhs%#l&qrK zd3p>vS867Hcuow9RZ%Qq6}26HvqvxDl7kOhC1cJ@U2&M$48EYjUzy&_NkMq_`8a^I z;5PO>Qj1sL!Ow;9I|5Mo8LU}@w-A9Ao;Cv{x&>&YG|$FVb@vVE=pU>D9P>*9OQc@9 z3|+Yv!TQ0ptH}SvMKu)1Ksa+aBfLs`Uyk_7UlfC)5Xfncct$aRXEWFR)x7j1BN{=T zR!+7@uSpH*N->SQ8kKMQUtN{7qn;Co2{@oW8<%;H6cJY+=&^yd{oM@;PS!s@(1PK| zGs(2wNRfFV2*iZSz|J)4D^!7iBlf^J0l|AuVt^=_0l>e>XD5vZTK2e59y6A}^^#f@ z8yon;d#sK{AW0f zzjKs7fzYj?A#6}^%N;;9$drC;kmZBSRDB=DD=GF58bq{%VAykw4r(AqLD|i_s3rB3 z4o@6|U6$fOZe(w7??OV%%dqD3J@gwsaA%MMc^lEHVgfQ_)I}0Dc7krcPKrW+%73k7 z?SoLMPhtSba(Tne){ih?my~85rR$N%d6nAcK_NpWCw#1a0XLa;nyOM(h@e}FG+fnt zX^c>Elx5xiE@+u58DMPwVq1Q7AoEp~-QZV-byP#`#!UV28VN-D=>zv)ukkV#Qe|6KDB#1m)BxY$^0(lCdc z+Aa7}ZU!+vQ~e%J(~zxSY1=Q{Z5ix_4*d^)jQNrh7O=^HOaHHI<0pzh8-sa#? z_4ckoUCh@9(~Dw~P*eXnH1yaWgonwW9w&RN?A8Pj3RhToEf5cYScW&wxhK+9V2!H| z;#?Rr$cVi7WlDaP$}5r>LV}-X84x-=aYEkLdZp*!Q263t(iRNmDKL4u9N1QCJCnZ} z26<@isdi|yHdc~6dL58PZ9_x5_|ga2`spur-o?hnO`?MHfgy_yhv4-IkbEUufmAmd zQ=$-exn41;4!m;H_Kss)RML zy%rHM$BL)lNqs0A%tuS-9!ZR`17p7|=^~vP4%WjhSGES|JkB8XfNwI||xFU4Yz2D0Vm<}N=Nei=psjAW~EH?rQLoR16v_`E8RD`hlf5B{q_gU?&mLPI&s zOz@ubQV&4##&m5zhWOi?pErKp@<-{J6}KYr@mi`$t1L5H%UDciHLK&FoR8&T&!NSr z(gs+GY69OgE(&+v2X6%35aY3=ROKK#E?~s<^!3>RGc4Q}g-CDM;kZhyat(+b%IQG) za&d8S0u@|$Jw@i4Tz>HApQuVMBm&Yzd~*0DGj6<^utC5fH*T_8Mtp$c0*KF4)YM&- z{gT`kxHW@bsy8AtZm%8!cHvcF_6U3d#8OUvevw))HWfv%p&x&W=FHZ*T0!8qtAYdul>5>79fSL;#?2GzSu6;ec#Pqxcv(}(R zP~NEl$}yAjo=_y;@fy*Hx>S35FpI52n5;i(H^f5E&;sIi(JL6DFmNh z%S&D;FO#ax&C66MFBc+r9e{blS9pYRG3Dp=w6-|0|He7w!swy4SY6=q3OP(k?fhMeDajJ~Ch zLJ5$CpKw(XMVNtTFkRz}nt#G!xr#$fY|ug|#80I95Sm38#{5>5bK4UjK6r~{488~8 zL|F{Pv`SS3aq-TMhrox^mFy@y`+>@V$f=fi|NDia=v#_iYokTIL30GU_%|*_^-G;M z&b-*|wW&nYyEaDmrV(U>44oVGb#&a#RB0OpptuB5Wp5G4wFR&Yb`w?JaTvx2fX?`M z+N4sO*1Du}vOrk`K^D%!@Ge`#@o*sG! zs-sap^fC7dne--jiU9H}LF0T?FoT#om3n0LQ7ZcdEq}nb?5u$TejAS#`Eh)9<8L-0 zr?M@C!3r|`Vmlb-&q+xRD)%72nLz-cBS3{2@!$7YGb%Byk4e&(z5+xtG^`!YyW?ZfEN2(ywQUv5? zAUG~QJ_%=@x+;x&)HW$fB#p!W29d6|_D#ID4_cm*EQJ6W@9gOjY$q6z0$&rnix2mp z`RQ+35(6X@u=yjm@V6*{uvq~FNFPcp5D*qY$_EU|Js3;~3N(>RY8gX7EaJX(tL*|G zWcwYs?5>t~1|uCvBu^3bgr_|89l{W^^a&OQdjT{Cg>6|4F$78Dy~G!25OJ zCj#N)ASZr$P+ZJI{e=iBmrZI2He#paR@B7Xh%+3vyZlL84(N&_zRJV{G|zCLwIA{F z^J~J@lb3;(OH0h#4q^Fe+SCzwRgRp$rvF=%0#&&==zsFvtRCNJ3ysuTjT2R&iSPz} zTcCFElZN+7;YFEht$bo`UWO#lsN_RLS4JBB(wVhjJp6;*H9K*|iPT@RU>JCq#;dQT z$OGegET%zy$x0EZ3O&Nt3yX`@fI3G7@}}RfT2rP-Ix9Lr)ul0(M z1-^(+=nT3!>wgwk4hj1oIC1juR5cKipdZ&O&67tH3CP!+)pqP~$k`lRQ$F@^3)`t7 z5qbB)#mF}Z>wh57{ua!?5k_E-#~|$HRJA9+@0d0hWA*#R6m^;**O|I#5FQJichUVm zvT?$|56%gyu!D9`WC`whG7o(;FPV8?bfT0f-=`)6woZ{$GE$9MVMMPy?Od)dTvCh0 zGfjdg{2VJ_lo)?|-N042Kk=TF&az~XQv9jC#lZR~eATDUg3h^RNV^zl*nP;kYDWXn z7f0pwC5}vDJ{ShDXkogK9)-|K_`nH~&c{1T-aZ+vby{2=9z`9tI zd=-0hayEN6 zXs3lAJgndJWc}nqWAwXR=?59B5%W6+C+ z|6ho;AZgDK1J3yAk67{<8RD?B*lBZvTS!Q6xi7U_?`0T$LSJ|+Lq!yEN+*?Qm6kQu zLSFy;vGr^ep^o>X`OaZjm#e;O^pYkqoX|_9{4V!h`Oh76dM!h4Ug%UG6?{6=dWpHX zqVEYv7luIIU_aYfU+hsEuD%W>x)uiN_8pGewI!%;@A44!>^sxZ)pco>9Sj$pQGgZz zNd6DJ`~WhW`v7mabbS&s%Q9khF;2mIlDdI z5!S){WR@R0os+JAbt^qAmZ>tG(F+e~)Vxa-pcz(2vUQS1D|xKUxWSt@Zp>XTvgC>y zs<**napiurY`C2Fsk-(4F6%BcMWUOho7+y!;yGnLtdy!>z5W$#h{}qwKEd`|@HqoJ zDqG&8RTP4qlT#TeZVvCStm_*|=X8b3YdGDQ6w$VBW0s4NWM)X(5ME1aN%M!#w1$Ux zX(2c)toVEew7^0MI(+~B-6s4)RaF(39)ja%saeC~sc+~dEo7F2K^J2q_s*&St0(3< ztxs;eZk=#NqS6VHl{=k!@*qLFp2*CS&-&;!AbbNWE3;X2X+y)qMXw74UO68qawv)x55|sC^#i8rTl|6=+|Vt*(3?6gI{Mq;o|1*j@6y0bxkiesnz%q z#cc7c%AqesG$u9GnYt?5T{M}-(9NZb@zmJ*33GI3Dp9to_j8Y_&;0Mzwr*A|TF49I zv==KoSvhJ=$yIM3R1y&Y10ew|GirdJY2Q{8bp{Pg752mII(`K2mW~{vBg%-^L>J&x$SMPI8o zGyJI0!9dPHxVNLIQbD*p%GL=oR%xBZ_S+jjeFExfF1SLVkO@F1TH?|vMZc`7)W5pP zs(L1G?I#kQGu8?m!5Cw?!^j%WMAdl|(NP*5qFw5_@$|JA&_*Z{0F!z4KA_YbX>8M% z!kMiYMNX=NQ11mwX0LG(5P($y^kR>)vREqvA|gF9pFBWnve~9}9309?2IyO3&I^47 zx%Rn3N-PDn5-mq%7Ju?YlX~;YNPsQR_W>0!kDo0Bm4BHP<@q0;>;b^pObQRh80(Gt02;OUN)IrXG~23rua<5(xI z-~Xgin&eH^MwgXz09cWH9mHj`qOA3^K)hNq-9Pb2CE0vG4-N8Je7zn1)0XJ5yXaxs z)NKzp?~Iw&uJrJpCKI+L#LwT0&I@NZ%OiRU9|4Exq2Zi$l@cGg3q*`>A4&tn0#EC# zLHR%Cw(3h^HE(0kqtcTuHFx89#2#5`%X#C5X5Zok>X7NarEmt5+ZCR2mZ`&E6-#5q z?Wh^jQiS0V9ebD%k4t`|b4|*u8~`~W?|^HA%A5ZhVA~a*XbpUi6A&6px)eG!hQFpE zpuI^#j%pEHh4A(-?_1zsH3IG+cQBvM>rTI$!QVg&AO)4#^R%R-wXxWt3_;M)%{4lh3J)I;va|m=RAMg}{N;%UZkTJ~n4XZT zz4DX1Fcf;>CaUXM-&HwwpFf>4Ejg#ocAxlL8f)cUB6vOiO%mjPclh{(v0JO&_r$K{R_E@KN-`!6MRSNh zeB*%dR%uJXY`3%St*pjQ0-6@dQ`vtZmVwl%J!{yJE(sQUXW>mtSac|y(~R&L>4uiB zPV(<+X+};nttKbwpIwbTK5_>+#Ixa;cJzfiDE;MOi!dIH&oR=eWaj{)3pxeZSa>Zm zymADORz_xk=%VgRWQG-d^7VB_#*IS2_Q%qPNLN8`NCh|pPzf+5@< zdh2z?%a*DXz8m8Z&lRia;;6z>E0tyuiX^x<6|Ik~+nvIKi1WgZss_2`v)?Lqe#+L< z=|rt+7L{)8OXaW`KP$*n_`3jz^y%;*D{mp_8Mk^FMa%+~p$ilIYw`)F?r=tK2&kkJ z--SM<>k#~WTP~pB{glDx7;Z!GFZ%)rlymAJ^6G19Ua1U&)R=8J?eHB}~r zC5S1FXhB|+jQ#ZG8Jg!2)y-6sCkJr@d&-u`>tyXAM2SxCofJKPfkz7PW)5@uieR5-7`9qv$T4G z$}l_}CmJXh`f?TWJ8H~xNIIJyD!!fy%TXfgpQ?6z2AXi$AJ|h2Sw6dz>!w~dFm`Cnbe`4sgc;-&!N?1amBqKpl zkJ8 z_Jp>1-+7;p6H-<L`~q^QyuqNo^{@-f5t zysB#;n|IeN+FeAS_#>wocU%8g$+(-=eoL1s`DC;XVl!M3-Hlz@TY+yc0atICkk-5j#iwy;M0rPmHvR_SU2)qz0$hKOQ63S-Q zo^w-~#^ch(j62DtZ->xBV%jG&U{_9PnO)Y*Jap_`=DGHUCvU!5zvBC6p@sxW4|+mvoJSMs76(Vwzwq<>zq zys?x-@p{ftwjE)T^y>5|pYRU>4|N`6q*w~hxebN_VY=*+ zZfbDcg0k|uHqWPKGcKJbPks14-=_izzK)}0zRuUl+%pvOhki^_$$VJ{#7w+&t5Eyd z0jrn&WrPVw;!){`ZCyV-?;`IX*-@%;9F?bCmbL;u6nqpTx|zd+UAQM2Ab3(xV)!6U zkG0)fw?@_^{Mkn8@p8BKVzuqFV+-<5_`7y%RGloGH_)+}$8SHl=e$ia5jVs)41b8! z848Tyy?*9gOTsr5bLQUg;mE~%bZp(D0jK2F(4?gBqQ1@n`@&c2dHzt7?qlDEMB);P zfVFb_O;&%HeHWu1wr9gkXy(O<(tcHP~V$%UT1iEAECathM4Zfy&J5JBwf{tV!;|B@aQx=Hp*M$W7@db3z- z>>mH!&E=G>K&{6Zo{#Q6srQ|Cu98_Br+>PSizZCOP1p}pOB(TX2Qu0fS?~!sx2@9@ zh9+?LlHavKse8MHah7H71ecBu%J&&-D$#flK`2y#1BWq(n?E$Bbt#qLt|VU8r)@Rf zd7EAFbmGfLXH>^S0pwqW&-l5D@AsmekNs6PYqhhfTp2&zGt=lAX^-rxe|nHc1PKnN zLXVy9*YyOQQxCsF#@N_lM)l~vnzJx9KejzLPKB(qCHaB;1S-%_~Y4=9p>9@@z|dD4F84V%dpKUJ#uDACl~bJRo@aO@MCe84JrQuMdmAvV!HUU12{E!-?9bL z0J>j-!-J_pp!C8@|LQ?{FH;f@T6-w|e>eq+FLC}kf+<(h*f=A3Bf2Pn0U{^}E^9X| zM2c_7Qr0K63)%mg3xc1%C@q?AUmrgZb-*5JJ>5=`mThk)1<`E?uzZt%ZH+lC#kO=qp)9sgz*v`P@Ves zKzAD~4qc4dip4#YW+lV%L#1Vg`0M7|-gX!0!uz6@hZtT5*!w=2W6+rprL(w9v_%o@i zA3J2uRf#@++#}-yjV{O_{5cWO;MB7T3QWqNBs}AZFJ&GRTHw3p*v#jfa=a)T7>OIp z&=LlyU4QwL7v^+#K*@fy4c;tqe*w7}fH85Y|Grt2EskFol&81F*6zsO14S>KOXIe} zA#Q7~S!yLf>}!BVnO5^BtwMjP1N5l&Jr1_XtujQj&cQC^9nW5~AcRD@`6m|_*9;cl zc+{RU21;>Ua8YeTl%ERb9Se~O}bC6JfxZj8_^bh369ye}a zBOOcYe|-2bxp8*jIW0vGy{wYn&9u3*GaI$%4vt4nYbc3=4u%0N>@_w;2&W-afWb{j z(QmC!{iktso`J6Ym?RaB72ST&-uSX zx4vlu&tT48ubd6UI`5vPfwD3e`d@Z*G)*qZdO(v>TJ5_xsFr`({v^;Vr5efga;tV)9$LDG+d+&5Y}q@tFuf9^MKt^}@kmE8wJdlabXvC`~67 z?K8}xfgk_dn!>hmL{}eLDliJg`u6{aWrh5T)u_hpmj%T!pZ$J&zzK1W1Y(i9Lf(GU zUM_#)#b1athd-Tjf8Y4;I30j*jrczvG}t&Nkcxi$nq7KX2s{l@me-Igm9c#JKLBn0 BrP=@h literal 17002 zcmc(Gby$?$yY2u2A|Z-^Afcpmw}garNVljo(gR3?N(f3f3`is0Js_e2(m6DuGz>@$ zC2-d3@7w$AYoGYz?CYHK&lfYiYu5X$C+_Ed?)6qnLy`Cz^)(m_My#wPuMLA?8^T~% z?pN`^CqI|HYGE*0K4tlbx-ZSqbA&H-$B{>_*RaX&XJ$T2oc4LhuJkdn*Dl+52Gx+M zAAMiZcTEf#>{D82c-B@=F*BNzRNPvBIHGUN9n2Kt(jL+hM0(}(`OR(0U>8kU*{cL^ zOfGmXg|-j~A%PyEsrWv%wyag7>jJ)IQv&fH&bo{pD#qo(K7N$2V6ecy$B3~AVX#j& zS2)4{WZkF)|MYwG2^$9cP4)loi#EJc$|rt&#Xk!TGX%^PPR$UWCHiI6C8n{N9CTxc zHAx8xO2)=%M05`f1I}g^zK?y>L9?_~O50dj<-eG$>=?y{#cW+QB)C7avr+52oM^Dr zmnz^{cVoTj=R-5;P3eo>X#2p+0?jNZk=>AUvCBWJ(x`QJ9N5k13M`IWXV1tu4iBYv z?~d$HE2o4XnC^Zk&pJf4U(|V@Z_l+X|NI*1w)h#hA3NfpMI1fv zx35kmXd6Q#=`-eZbNBDBmF)Y_R!1_Qxla#b!FqVOu_sRUSFPq-z4uNxD|%IVkW9|4 zyPWUWWl0|AdIh9jUoX+h8*bJOaJ5=<5mr?ii z_;c*KpO!5b7TUU&DeC%pY;*Oay5r8Wiv|0aR z>X|{$+HkHqqOFZ(*l%ZAs>IFQhElP%T_W9o_G~s_P^$u1APeuhp-=D_VY~m05q`g8 zmnQgn+>Ac`CE7WV(0=}UcXphvdu z%eHzPb5HKoblc^{nNthcuIRicb?Ct^c;<8cV`@PiZS7a{2EE}o7}by$1A@2gZ?kc5 z>}RO=F}XQPxc>ZN-=&!)<>)hf;Ncl_5V7K`xM!4TwkkO|{%89I#&yU2EF;_eI!sB5 zn66U4+|+L4Nx+HKRGWWGhNQ24z2l5^=H_lj22DfRJaJ){x&NWwRGmYzU5O;Kk3q`0 z)BvAkJ4W7i=4&Kn%g*x9r|hm;kse{UB_$oNwV``0#c4_ls$y9>tWxw+ev?;jp}B^i zPZRzMUug;bv04hy>AIW%LN^*9%VISZneD=~2@}`OooQcY( zooYyALJ%H@9uv0;U^8 z0(26deGgE)zb0F~Ys3oL1G3=J&i z5A_2JG0CgVk77|h@vQx>2+ChWi}_lPh@hJr{n>l()ptFbRIEkYa6cp~wh)RYnMl=|#BE07L zx-u-pon}S%O?O5{rGAE zhqezGUix|*xE9rm7j^~EBkzW#^*(QKpWav<;$2mHF=0hqiy)C4XGuG{=D|6(3pr)> zwBu~!FpWD)(Tropf?{LNAqg$r*mFV(PTs8jz>C$u**no6ke@dF5P$yoH6Z*^A4UVv({%+JVR($RbnjM>TZ2^FF9jEbGF;FZZ_fwjLgt03 z`47GA>^`oNXD22nr_b}A&krC%;#t)BJ;Zy=vK%z0b^E0kAIkS2ZT)UE_vI`_O3v#BT8E>Y;Ez7q}??aZFvpZGVg;d;_GJGSYUs^xXV zho^Qx!NeDNFu?Cvg5=ex2s`r6Xac{Cid<^*KU0fN7#;9^=<#P=GmxJw9YOffE6!6r z$D8NOJye3O-;21gz*8L6{qd;YHT3*owPL44u0Y(*bTwz)RpS2o9)*u-pwnrgfR~C~ z_|YG6EUB9y)7eUwocoaX?`Z7ojXf7xR zI!-Q9d4BOWZTax>tu0S77PYHBS-{!x>-H&#U4>U)l_{)}PIZPa%I}*G@TL1dNZX;- zP)EaG4iZ$&3!mv2SWR^~ko6x$t<;42_=5vH?+U_1 z*_oIi zM9_V8pe`K%4f%IL%zAR-#_+GZX_)UgFjx^WmMXSa*gBn$4=NS;N)$X-dlg_Om=vQ0 z04vAuf^Y$H5x}N^!GZ?=3tzY{jpVBqeE6R49*hlp`z8oiuoCR4XS&XTSwKK1^g6v$ zgw(`yW7KT~#|%GqVR3OoV;`^$77mB`_$~)`#U!DdPd^^7W=bKL|ieXwi(3-<^p#)?sS3=lZY}0Lh85xj8VLB+(lQP0(1~Msnn&Pb- z$680GgO&#eLvnFhS@iH$Dfv?=kp;gR0wWJ9227hl-VWhiVQ8tTT00%BlxqgHj)Jmg zrka@#*rZQ~qw{nMHAdP4FYP6tgRHtof4NH!attf7VxjWs>x1=CMRCuq`(6wF4QiPW zrf4pwQEe>vBs9FyM>`8Us?j|Kmy|RGpUEBX5S(*!7?dXzW(A0^HRjgV>Rz0+U(Wbl zvS#^dYk<-m6$q-3NBeUCXT1AQ4>$YWCCd-%(*lwf=M`r}6++HZB^;5mA?#;JbPm|RLnUONkdTNu20%Q|TDa8mS+= ztlXNc>h$YcA1!R@gtwogovpwJE4vGq7hX;0n3-izv4sW|qv;$23jW!$!k$3G%+z3~ zB%apg-*eJ>hX2t;VdfFz*w;gjCnpgz4*5tUNuL9k&D!K808v+5&CSibQ5>|z@D2FQ z3#eA4bx9w0T(<;8e!jqfIp4!2209$3jKPxN)ed}m!9btr5}v@5-fiAelHtScxtRy% ztuL!)_VX89*M_nSyj0EsQcO5n2$Xigj{cV=!L|%MTU`VQ%ayNn8*?0U zZ*2VZ*(_^l-uLA;Z?#?dBYIFpr?WbkRe?DL)yErD(Y_)53UlPFxTL{grsVRMw`6W8 zGp`f3u3sseul}yVxOmR(zS74hX@qe-pIUP-T(fldz-$?g^T-dhSKvb&5-#(`J2aoUDe6#XnU@o z&ts`O*1lqArlI1`$zY%`QcS#cx$JD2C(tf*7?0@o54w`b(zAcI^YoIlR*{{RwQ$u!PPz9elEs0$1j{#1}YB5$8y(F&=D+ zAsNr@S*lxkko&&ZqBKA^rHZ=dB!JTS>GyutMXQmDW ztEj6B4c6HxuRcQ_&HFmtsQ{-Ygw)KFc3*Ys6tEdc*$*Mqn=b$G;gdhfhkzTEL3r9m zMr_~VEKAQ99;2GTzv*$zJ^CYr2{zx%-7|4;b0J?Xb+br`)V% zw%Pv-lUhG8_hxgVOkhyxeu(zyV0FRK+XI{O&J5~Ic)0d!$Ht(;_^dx)t~`VhzXlF= z7x7ELVWRYd*7B{gQQ=*72?-Ormp`!1To4(O_xLvhal_aPZ4vcYK`%ic`Sa5}(%0(Z z&x!RM6W6I>^w$<{v*tIg@#O>TJeO)L;eCPfO#2fu!(BdVo3Oj{|9AUOWlE?MV1rVd zsV};2q{M({s>!|3#Oq+~vE4|Xa^26#W{(p1#+b5kt?j+J7O!L>hY5LId>G6^wK>S6 zi`Gj?Uq2~B%>9vh;JHT=IL|q8$%}5r=$FO@>kIAeM6}|%Dk>_LLmwWFR6J=9-vE{C z1^Cg{@4dbG9-9-}`^irNjG$gaGm?E;PS4QLmX2Q3MWz|Pe)BOcsA*guaV9!mEb#nrdms4G_r=bz{^$Pws{;teVo=B{naw_pSW(mY{eCFWF<0Pqdt>NBXvuPK za_^|7GF3=%8LKxZwo433mJJ2V66H)LPJ_{_|=w)#L$wK?PgWu`WJSx zhOVwvCJmlwOEb^8R;#*>Pt7~dXa7)Wf z8m85MVncij(;Lu}k`+S*tn<2bLTl@T7UkKy0Q>|sLKYw^fApT392Gx93q4PY5_X!k z{X7mZmeu9Oxu=II!{5KmcP|w*eSe!z7PB?=+@OmK!b{<*5}sQwtTmgzH(R zkC!BG%{GmejgDJLcx*6n8P~=Dex^4~$YH}2I75!t>49BOcQ@~%J$P}q-Jc3A%Ci5f zz-_)bKkFUJmdnq}lgA^bW;#DTEO7@h7_I@%^c}tQ{F{1U&25Ny!)3 zCJjz#DU|trgi;|1jp$voX86aykIaI|vjX{p#T0~?`T#=0Mnw6VGHoTisyb8Bw|Q`; z*6vrqQbfcxyYXT@#3rz-D zI`}O<1&O18Zw|rDN^l7&Qd|-N~r9w^!Foz zo7?ovAPev!YFcfF8OX}v`Iwv>h8AUQa8e+YspmmHQg0lQ#6#pqQIkQ=^m}}7xmUeW zMOiuac~{ib`1`++a%>HZ>C`ChwiIqQXG;paHHZiP5r^1-Q8DUo+%!MF2StPU3?MJ= zlw;5{HU=W%P}pd(LREUv{!93cYTqNLmgdKQJ-ejd0mV&LDrn)j&^A}@^tbW&QM@(R zf+99(R-i40enMsp9K6MfiCyC~z{+NGYkvbkmxNZFDfiKP{zd!4jd4#Q&`&j z&|EkL<|-Uf)k-LnHh~2{f0h-HFOt+pnYzE9g^#N5&AJcn65_z_YOJN{H)nnDGIRmB zg2iQVUlrQz;8`85CFJs z$3G)SvgPpYhI2y1CR)6lA!cy=Cg8)9X(c=x_sgsYGVCT=U$ujcpJLDs`a$DXhlsig z_=qon%4JC~*xiYP(3*VJt4_sE)Ofx*QEo?-X2bz8in#BwRAx1 zEpqZH^&d6B8gX$qy)RBymy=AKVnNDIv!c%QUZD!80y~Z;z9XE1&=WN6PZx=Hbly-_ z!vUURdz(5BF3CZxPhB|ym>7|VM3Ap4@r!E%9mRsa60w=&OQe8z0eH9zQ8$x}Jv_8PRSb*m$UhqaRHt*%x=#-1Z2d#{RTB zr@a#~z>)vt_Hva1I@9dc)1EkHYa=|tci>GCljZHRAdRMlJo0#phs=!uJ|7fV%P2=l zG;WcPpyFJc`wfgNsI>k$Ct4DdG7I3@O3FOlbeV}r2{R1VNDati0h2k~#*s4Tz5RVO zi!F?g*l@t*2hQ~e7MKUxAPw`ic;di= zjEwdG&YKK-|4-TI|5@}Jk2o&i{8RzpLmB@OhuC?Uf)4w>6uxTb1*sN-pE54jc4MP( zqygmK-XqF7n_cbuzFpO7br5>TiMOFJ^qL`0`=9F8(~T0$Q#s$FYlMXTzi{wLQnLG$ z0B$Ve@gW}3%3rhxWyNi8Nf|6C4;CCowPA#yb}O&3aT0E0ujIYoFVasJQso9s2W*Oh z4u5xER|P%YQuAhcyzh3C;@;lgtoU4zv2<>6QBjP3nepHKmahNK($I7by>htiH@85! z%OM(atn~Vwrw*DCg-CvJIo#f``mIvr|C$k#s{@Jcm$|k$f zf*a#lFj7@nRxNfGIw zg=&k)18}@OL(1P6ntPBCY>db$sdcb@QOs407T!BwNP_o)LonY>S55g0%~f!l*A8l;;<4udp*EIlw;GLkf- zq(?~kH-qdL^zHf1Jkt8w`%n&ac=$4fo9%57iXa!tG`RJC{~p)Zr{+a~XC^u}6+zq+ z0s>t|Qb|+O>vduhIPWR-75I85ujBLYd@7uwZu|!5LZx zBru^Ia~F?RE$%fzDeHHtlVeT5xet|?Ti(p`d@o>Dg=S%2j6CZU7m%dUyNdN^{ zDUyP#uL^XVy`a=X37RPkRFKpM-qqkCUxJ&()LG!Y0C}9&!YtS$v0olae1`=jo6b*Qd$gzdU0%0#u`=PK%&}dy|W*16@zc+0~69i-3T(IimMkAb%<= zzkP7|XWy-tt3jzZiEFta-6_r{)B90GM8sAB*m;6n7)euP=tkv-ZN?&EEDp`Q}kg|LSVF#yVpc6N3whVSVljiG6v$V16y_(Mw{p4i1? zvcB#W27K#juAey2f}{}a1A|PbigB>?5`Y_?0%J9w4})En6L4Ldc|m1J(!d5ZTCR61 z=vA6_zun3A-e0kP!UD$LiQA%o`+-tWcdS5O^&}IF*~?{MOw-3Lt!u9_`4R?BhfdNb zMF@f`P`7OTx)&U88mKHXt;1aN*8OSA7-KHO%1@@9q|(A|H?Hb`%(^x8JM5ER>C0Q7 z^g29QpTH95(>h`p)0_1eehfg--OM#aP&04+JXKK#yQHL<)am^w8!Zv-WkvZ%4Fp$8 z86(lt$57dRsB%Q-w^h}()ONNl&;@rcaGGm2;=%$&^y}BJ)*BI{4u^oAQ&La}nIm%j z@tPj*^Y~dbFd}OKRxcHa-;ZGYJkZ_IfL=PX)Eqf&PXLoSAJIdJyqNsqG>`$hL2qww zPk}FQ|77$oZainmmEzWJUMPPr{Wp5P?OxkO&$7iH+`-0j&@3>!)BY?Ce!%k^U^`}z;RVbp@UPP)*kt)n zlyX4}43XA7`+Ch?eZ&Q<$<#AA*pPltWBCt1X}R^h1L{iyI4TJ42`RK5c!TFlXaOpt zAfcDyOifJ%m7wwiw@+F8OzdqH&$sVh+%Pn(b$M-eX8{BhgaK>RI-$W}4T@TZQPfV@hyFUUqAw*%(7J_kJO_ zQFW0X_d!X@u@V3#o?W zjC_^#2uK`t5OQBp6L|INOYY2qS|cNArQ1Ti0~zGyF+;Uo-wE_IDxk)=44;2HH4NE(>T1>lAv$ZKbh0r9L>w z!#3C~T@+Ya(9|xO3kD0Asd^q}gy}Lusa%eEln-P|I*2G>8=cDEHSG`lQm)JiyaCGo zCnyDf4t92avJJ<7(26q1ZF#;&+v69B?sXoppi@0#V|D<&J$*|lI1TSEFE6i%cD{OB zW}$AO9Qv3`K?7aqK&A@vBQ%xVrY;@CoddO&d9K-mP>g7*Jur|;9A1==kkI|%;p@C7 z?ExO;IIywn#G2v_?99wxuawl*3Qo0%bX0sUaud5VxJJ8%G(L|rOMMAd_APINLq7aX z2*q6PhWK1Zhm7PiCTK)#fxZpuZL12sa0XKNI?;8;tuOiHf|aeke85qhzw163^>qA6 zpmC2t<0wrVV2gBN$65bEFz(F*sTbP9m#Fe5`J0C8h+yhc0Et=u1`U%au`V^}%It`xPNr*|-LA1zSd1dV?Xg8G%PDN_5wgm#UN`3k%x9+En>AI4sW1W9+3e;=|1;W$9xa)p!A}igNRlFkb z;^MixJ|GR6AIR|oD@mPxVU@a9sA-V(v0%_)18YyUC`icVhYU0{G7^9GAKLE?4Y`C+ zwjCe+fed3N9y6t}Es;s#%1^d`SMfPs4EsCGkP8ui{J1mG^KfI=kdVBxcBTo0jCG^- z4$;_Q*@iJbtSB9H&fUkmztk(KL8&!z&{C9)nS~Q(D4StpsWyC57KWgVxL^@dx?$d@ zr1r(sNZ(hVl~kKPSD4JN0`Q!t3knAnQckX+{(K;tNZK+`&F0j(TxV zySleroeunVb3lUA=k~cm`$@i8yeUXg zZNEc!y0b10SlRC1cR_Xd>>!Vpz|biNl=GN3V0vJ;P3J{HGzUU??#lb_%s!b$mG*~A zvurKGO#oz}=aD9KzFT~w@gDs`sP01Pb2~7QXzScwg~cq@hS>wC49X{9S^JG&*|u^g zL|XcBlU900)jPz4#*_l~!F5@}^|iQT}`MWL*zO0agh> zd`?G|WR(Zr{x2{@uyES*u$t}BVqbjw%8@`dNHzn0t$Z)1dx-}Rd*YkO*7Onu$nuhg zO-)VK$#7yrzHc|_8z_SWder4mW6~8(-;j>=ZHkQH<0Z{i!!sMO zWu5{k`A@>p>~j(>AdI(H`ouXKeGctwf&@ti!la4cl!#s0qXhqYsp0%r**djUeg)9A zp{-?v6w(zQ36fDwnJ!r}nu*>8C1QOfe+25O4O!*pU;cVO((3KL&vOqZ<2Q;ci&&Jg zY130sU`$tfUBbV42Nb!)%dT>}2f+J$4%Y6)vuX<8@dsdRx->SYVc)(R8}^0OtW!(G z5`!_dGpe&soV#^(Z#WNlUW@tCta3~JZa4t?Z_(n9+Fcf9Yle!rhd;zK5g5Wq1L5l- zhnuJt!q5$25y0>BBlWi!!<9@$8RJ}4?^-z0JE4TRf!P36@bvNA?~}jq=UX&A4=5Ei z02ot`05IHd7Odes9>=UYxq|oDP;+jnPdKT>+&wIC~zX8ld2qSHWA8UqT2|-|iNMcUy#A)d=x9a=u=6r;Z0ziDpxbOj>I3p59%4$Uv{=8IC_FMJyRXpX$pyHm z7$$fjk(td<>K7?Mbf2u{MsHaM@mL26fVcMhV21N|Gcn1@tN=u9Ghv`%36KXue}mC% zuU`03D9;|TWqcpnQ03m9Tg^fV*s_GJsDury%B4!y(l1Qr(=;a2j1V8#_ofykjht)N zgJ$Kg#(|=bSwsXyga6E~(;?VJ=%l#4kYp zR)T?UL?l?{bs}CVG{|TSI(75=ns|;^K`y3VUO^6a_A`a#e4X@z0F9{sDO}9;=bJgA zaJD&T7yD?Lo$=`oPcB(IU93&8VZeOE+kefDSNKm{KOk&^42Q^3BF^(KSSlPUM|!qq z>K&iol%W0+^*Goi(Y2;K#I1!QTd+6?%pI7cAMAzuPp$8(>YPfkww0TJJTQb~R~ zw0t`#)TLaW>3lResLtz^S)>cK-Y1`v<>cjYgC}*@?@Y_(d#&6@2u=gq4-EG{@ig-m zag!DkhJch7B6pqcfvpJ6hTZ@n*n~%x+3Pg0ga6FObd~yT*^U;xpR$(K)74c1Dp!bq zl3q&^ZmxST@YbxM)%BtsM=&8?LjI93CDpLn(w9dq6hgoxB_I{^2VVH?OLHM{3jaG7 zBi@z$FD~ZOxLcB`%Zia*s+fDhElSE6^MT&?>F)V;G!?sHoC3Sqd1^g*5y*YdN!|7%HAi9jp?2M=B%HeNalff6VhH8j|+cwhKuy$BXSTRNpP`bK;lCZ~5uzI4Vc^O|jnIZnic>4+=#dLRSH<_v6Q#h(+CIwQ9X?rDRND8iwg9wkSyf)2s) zs#ezKE-2cQJXjm*3b9H7QQZT>3?%`~)jt++Is*Wtysn-7J-mS7E77E>5QubnW#61y zMF2}%<$}bsRFchIC|K||UJt~w&$CnqHg^|@C{$+Z?Sk+}d~+HIR=4yVYdXPcv!wv}sd zv4){$o07`+xy#Evin#}3n{A)y0@E-3P7Z6Z$@)TwvIL=ZXy;6HJ9YH+_3N|}!DNDh zLds(!;rync8~Im;lM;E)_cx3rm6p!Cy=qvSD^;Q4a{fHm~AcLO*m#fewReC{eic7E_Ob-;o6}g zb3_3A` zvIhH!Qln}xAQQ-@eczk<=hGYan%NE>`IVy^sgq(uEtX-dh3_ujbdS+7L|*7vp6@6~ zrQLUX_G`IP*E);#DyWf4?BwL+aR`JNmztQ>pX1$XK7{A z{+gv&0uq_HZQVBLvD6Kax7=L5D>yrPCRp$9;b>`3W0|); zWSQg#fCxcvK9uBbQV5D46sodK`d*4DBpap(+Qqd5i~~Is6`q>Qs%GS^^7iW44vScY z1@hEsbta(Odg&Iy?ZvyR5sMogvGhmVvMXGM2W8~RCo2iSdAKTwcEL#ha>FWFaOr>> zj0Ue=(`i=k2eZ3=(-NS`S9$z6_UshQbZY$HW8CiXWlVF#i6~n3N-)r0=vrbf%aJ1N zpbC{)2x_HTjLR;4B8pW3l@SLVkgW?)O;iV71_)4@8E77DAFgsu3qb5tH%O8Y3-zM; z-Ir%c5{6vW9O{;bs*vq+x{H!J7JJxorRWT|f1G#mw6CBTTDl@`mEI8Zs@`ZVl)A>5 ztyNvJaeF8Rc<4r=hAc{N*dwqV8tS5Lw(jOD?3|9Cg6Vb-pz@7=3k2($46X5RT*;!MgTHD7% zp*@NXXXi$4Hr-U{KXlx6V<$Z-f(g`ZNoP?9~BE zcdxGNbJw)p6e!bUcB&*F@X0 zYEj%s@kSNNj^ml4g`rj$KGqx57b(y`-sj>HI@?+IP=bPXJdXi(p<^Ugk8i$`#8xTy*pc~{ zd$30yT=pamD_CORZQl}}*k+0< zucJx&Au^29`Gb2FKR^GgSK386ok7hP3FR?>G-m{~mJr#{;dVP9nnq0$!A!heNVe$y zHKN{ic~@fIuzAWU%+bbvO7%A3n}QxQg>Un>X(H@7eo1lgFFVCfEwtWDGru|Yr%;A> z^oVobM~suULF#r;irq7M80lqggA(OUUd}w~B3tMn2_tsN$sH z@ex~6_r*Q)|MR^DeZh(hK!1959026tnv<{5}+MqA{(L7i63hTH1drExJnSjbHDABctPOV-Ewow7?FokEDNv@4k>)9c zHD}yi>XUT)ejKHTRG9CX3?N4byAABL7GiYQZH6;B&iw{>XEWaw@8n3DpPtv^m$FQ2 zh};=Sw?e^`;9!F?)gzMYaPnQ-bq7HvyXd%3CSrc-1VN_tp|nsY+RV|IytAKH*4Amn zVQ8&ek55!1h%WnesV*9B3A`y>RQ0WZXKSGQNBW}%Gs~8XX>rkPyLfcy6Ia<+N&Y&; zve8Mi4Zv4)$z@nP%C1T8pXr9ge3UGHrR?b2^EKyPZRiU8oMrC7K)O(4a>AnaV!U`K zKfW8`t#axaQ%tP=ryUb9utxPS- zv5Y-TIPrT*%eB6*o?kp{EZ4KH_nhu_SsqTx6>$1J&*FS()!uhLXP$a8LZZVWS!2fE z-k{2V;ZR7xZLEFa7)f-wQNEFnT6XJ-{oy9}mgrK~?r?8+|KVb`@rk9;_%CbfTG`r< zT{#YvQ%KeL?b3N`pPh}xmCh{vosMc>Pq;=xgV*ZX$VO+NaZaWH`R9{S07h^zF(rTl z!G`IQoxy%~#YxkYH2&Pz1Y25HKPUqSQ@ADNUwSd9imb~?c?J^<_0uG0U+C=Zmk9koI4_vdN z-N)&4((pgRiSsmweTO{DV4vsvRD>);*PSW}7r~3Vdg5)$XeH$V=>AJiP?Fjp`J*as6WEl54#UM$k$`0MY%2>!%Sx+}cwgbyy*F4t!^Ru5_LlXwyirIIz#pj*`J0 zNFb(pOGGWG1a1}BhyV$b%Z*9~h&sp55O?_z1YaNdY7aiu3JD?)7IvKO4j~L)#)tn} z0P+VTFo=NxEf>R3sgb~uD_}{0(r!$L5d1YQSRY7W;(;P3y=(e)zNGOY5xtZtpyVF@ z)zdL|M1a;;_EUrmm|rgcUhG`ig~YmG_O1^e4`RPkdX?yZ5Vi)*T>%nZg-;Qm!0nQJ zK)FLypu1vrc_dKGfL8)f{(ruF`|qZgM+06OlO1k~22$){zzoTw2PAfCuPZFso1^mH z241G`?VB2OMjusdifx>pYCUt+xI4W21UIP242^qe|711!+r$2UMr)TNFeYZ?Cm_I5)5v^amkHf0h9It(FQ$}$j;HN7XGZ_^>w4tN-5hY4#t4uQ zYZKl)MdDi`fOK+?N=Ue62Yji{bN*<*t-N60cWUux(`sgXbv1vgxJR*=;(eds%U|@D z_kl|$&p5ZwNtRb!P7MaIfQjG3iU#S;Pr*3Qa3^uJP&1wPa5U?p)_BGxE4Ln{FtXhx zeNjcgz|71H$uvJ?1vAbbm7=|Fs;FmmT}(Xy0?@sN_VfNXZ2nu4HtK4?wohtv1ZMMVdCcbUbJSw|n7hLmg$lV9?!%>o=UEz!LK^q(y0y$y0*p<-(we z!RQH+K{6{DFXGiJ?6XQc=cB%t^uB}s(Df6?I~9YJ$kp| z^|J{4c*StL$iNvSNHX^S=Vacsqh%PMJ^2BN{=luScwJrHZg8#rx&YtwaK0)fbcFwwAXDKGDl8TKxRQ)?&9)qBG zlDl66-GARd$@Ire*&qIMVgeWSIPb&#gGZZ$q0tSXl^->G?QjEz)_oI*$d*)}oW@a+ zjQq~M*S!mFxEmPP*@JttmK%^JFOv|-i-P~h-OBBhZ-T+;ddRRjP1sypf%_)H?YiVa z3{~5oiH0_-Oo@Lt<+Q}XVMX9vUzHDZJ9*BBWJm0?|DbIEYgC$4^f@)c54xK^b0BVi ze}4jX;PqYVdQ9K(3<=CtC2aa)FS)&2r2Ry_<(f Date: Wed, 2 Oct 2024 19:51:49 +0000 Subject: [PATCH 31/36] Add latest-milestone and alpha milestone back in Signed-off-by: lauralorenz --- keps/sig-node/4603-tune-crashloopbackoff/kep.yaml | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/keps/sig-node/4603-tune-crashloopbackoff/kep.yaml b/keps/sig-node/4603-tune-crashloopbackoff/kep.yaml index a3a4e638f3a4..1262c24b0346 100644 --- a/keps/sig-node/4603-tune-crashloopbackoff/kep.yaml +++ b/keps/sig-node/4603-tune-crashloopbackoff/kep.yaml @@ -25,11 +25,11 @@ stage: alpha # The most recent milestone for which work toward delivery of this KEP has been # done. This can be the current (upcoming) milestone, if it is being actively # worked on. -# latest-milestone: "v1.32" +latest-milestone: "v1.32" # The milestone at which this feature was, or is targeted to be, at each stage. milestone: - # alpha: "v1.32" + alpha: "v1.32" # beta: "v1.34" # stable: "v1.37" @@ -43,7 +43,6 @@ feature-gates: components: - kubelet disable-supported: true - # The following PRR answers are required at beta release # metrics: # - my_feature_metric From 58df245ff5d22f4911300ebf48d04fa35eeadbff Mon Sep 17 00:00:00 2001 From: Laura Lorenz Date: Fri, 4 Oct 2024 10:08:41 -0700 Subject: [PATCH 32/36] More accurately represent the motivation for KubeletConfiguration Signed-off-by: Laura Lorenz --- .../4603-tune-crashloopbackoff/README.md | 23 ++++++++++++++----- 1 file changed, 17 insertions(+), 6 deletions(-) diff --git a/keps/sig-node/4603-tune-crashloopbackoff/README.md b/keps/sig-node/4603-tune-crashloopbackoff/README.md index 56cfa401432e..ac3182aeadde 100644 --- a/keps/sig-node/4603-tune-crashloopbackoff/README.md +++ b/keps/sig-node/4603-tune-crashloopbackoff/README.md @@ -718,17 +718,28 @@ based config and 2) configuration following the API specification of the `kubelet.config.k8s.io/v1beta1 KubeletConfiguration` Kind, which is passed to kubelet as a config file or, beta as of Kubernetes 1.30, a config directory ([ref](https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/)). + Since this is a per-node configuration that likely will be set on a subset of nodes, or potentially even differently per node, it's important that it can be -manipulated per node. By default `KubeletConfiguration` is intended to be shared +manipulated per node. Expected use cases of this type of heterogeneity in +configuration include + +* Dedicated node pool for workloads that are expected to rapidly restart +* Config aligned with node labels/pod affinity labels for workloads that are + expected to rapidly restart +* Machine size adjusted config + +By default `KubeletConfiguration` is intended to be shared between nodes, but the beta feature for drop-in configuration files in a colocated config directory cirumvent this. In addition, `KubeletConfiguration` drops fields unrecognized by the current kubelet's schema, making it a good choice to circumvent compatibility issues with n-3 kubelets. While there is an argument that this could be better manipulated with a command-line flag, so -lifecycle tooling that configures nodes can expose it more transparently, the -advantages to backwards compatibility outweigh this consideration for the alpha -period and will be revisted before beta. +lifecycle tooling that configures nodes can expose it more transparently, that +was an acceptable design change given the introduction of `KubeletConfiguration` +in the first place. In any case, the advantages to backwards and forward +compatibility by far outweigh this consideration for the alpha period and can be +revisted before beta. ### Refactor of recovery threshold @@ -1500,7 +1511,7 @@ which is a new field in the `KubeletConfiguration` Kind. Based on manual tests by the author, adding an unknown field to `KubeletConfiguration` is safe and the unknown config field is dropped before addition to the `kube-system/kubelet-config` object which is its final destination (for example, -in the case of n-3 kubelets facing a configuration introduced by this KEP). This +in the case of n-3 kubelets facing a configuration introduced by this KEP). Ultimately this is supported by the configuratinon of a given Kind's `fieldValidation` strategy in API machinery ([ref](https://github.com/kubernetes/kubernetes/blob/release-1.31/staging/src/k8s.io/apimachinery/pkg/apis/meta/v1/types.go#L584)) which, in 1.31+, is set to "warn" by default and is only valid for API objects and it turns out is not explicitly set as `strict` for `KuberntesConfiguration` object so they ultimately bypass this ([ref](https://github.com/kubernetes/kubectl/issues/1663#issuecomment-2392453716)). This is not currently tested as far as I can tell in the tests for `KubeletConfiguration` (in either the most likely location, in [validation_test](https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/apis/config/validation/validation_test.go), @@ -1508,7 +1519,7 @@ nor other tests in the [config package](https://github.com/kubernetes/kubernetes/tree/005f184ab631e52195ed6d129969ff3914d51c98/pkg/kubelet/apis/config)) and discussions with other contributors indicate that while little in core kubernetes does strict parsing, it's not well tested. At minimum as part of this -implementation a test covering this for `KubeletConfgiuration` objects will be +implementation a test covering this for `KubeletConfiguration` objects will be included in the `config.validation_test` package. ### Rollout, Upgrade and Rollback Planning From 1a5f314ad4994a229bd7cf98e1ce9403928397fb Mon Sep 17 00:00:00 2001 From: Laura Lorenz Date: Fri, 4 Oct 2024 10:17:21 -0700 Subject: [PATCH 33/36] Explicitly add proposed fields for KubeletConfiguration Signed-off-by: Laura Lorenz --- .../4603-tune-crashloopbackoff/README.md | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/keps/sig-node/4603-tune-crashloopbackoff/README.md b/keps/sig-node/4603-tune-crashloopbackoff/README.md index ac3182aeadde..f318b1a53897 100644 --- a/keps/sig-node/4603-tune-crashloopbackoff/README.md +++ b/keps/sig-node/4603-tune-crashloopbackoff/README.md @@ -404,6 +404,15 @@ CrashLoopBackOffBehavior of today vs the proposed minimum for per node configuration](./restarts-vs-elapsed-minimum-per-node.png "Per node minimum backoff curve allowed") +While the complete information is saved for [Design Details](#per-node-config), +its expedient to see the exact config proposed here: + +``` +apiVersion: kubelet.config.k8s.io/v1beta1 +kind: KubeletConfiguration +crashloopbackoff: + max: 4 +``` ### Refactor and flat rate to 10 minutes for the backoff counter reset threshold @@ -741,6 +750,15 @@ in the first place. In any case, the advantages to backwards and forward compatibility by far outweigh this consideration for the alpha period and can be revisted before beta. +The proposed configuration explicitly looks like this: + +``` +apiVersion: kubelet.config.k8s.io/v1beta1 +kind: KubeletConfiguration +crashloopbackoff: + max: 4 +``` + ### Refactor of recovery threshold A simple change From 4b3835f05820df671f098c951179a698b526f746 Mon Sep 17 00:00:00 2001 From: Laura Lorenz Date: Fri, 4 Oct 2024 10:18:15 -0700 Subject: [PATCH 34/36] Revisit 300s validation for per node config in beta Signed-off-by: Laura Lorenz --- keps/sig-node/4603-tune-crashloopbackoff/README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/keps/sig-node/4603-tune-crashloopbackoff/README.md b/keps/sig-node/4603-tune-crashloopbackoff/README.md index f318b1a53897..734690936837 100644 --- a/keps/sig-node/4603-tune-crashloopbackoff/README.md +++ b/keps/sig-node/4603-tune-crashloopbackoff/README.md @@ -1176,6 +1176,7 @@ feature gates set as per the [Conflict Resolution](#conflict-resolution) policy - Gather feedback from developers and surveys - High confidence in the specific numbers/decay rate + - Including revisiting 300s maximum for node specific config - Benchmark restart load methodology and analysis published and discussed with SIG-Node - Discuss PLEG polling loops and its effect on specific decay rates From 1515af5705d87b3f240ae7eb88c7496f0f68bf49 Mon Sep 17 00:00:00 2001 From: Laura Lorenz Date: Fri, 4 Oct 2024 10:41:39 -0700 Subject: [PATCH 35/36] Add unresolved comments, and annotated unresolved with target stage Signed-off-by: Laura Lorenz --- .../4603-tune-crashloopbackoff/README.md | 38 ++++++++++--------- 1 file changed, 21 insertions(+), 17 deletions(-) diff --git a/keps/sig-node/4603-tune-crashloopbackoff/README.md b/keps/sig-node/4603-tune-crashloopbackoff/README.md index 734690936837..b59641b0fc0d 100644 --- a/keps/sig-node/4603-tune-crashloopbackoff/README.md +++ b/keps/sig-node/4603-tune-crashloopbackoff/README.md @@ -528,6 +528,10 @@ of ~5 QPS when deploying 110 mass crashing pods for our tests, even with instantly crashing pods and intantaneously restarting CrashLoopBackOff behavior, `/pods` API requests quickly normalized to ~2 QPS. In the same tests, runtime CPU usage increased by x10 and the API server CPU usage increased by 2x. +<<[UNRESOLVED non blocking]If you were testing this on a small cluster without a lot of +additional load, the 2x increase in apiserver cpu usage is probably not a +particularly useful metric. Might be worth mentioning the raw numbers here +instead.>> <<[/UNRESOLVED]>> For both of these changes, by passing these changes through the existing SIG-scalability tests, while pursuing manual and more detailed periodic @@ -587,8 +591,8 @@ excess restarts every 5 minutes after that; each crashing pod would be contributing an excess of ~1550 pod state transition API requests, and fully saturated node with a full 110 crashing pods would be adding 170,500 new pod transition API requests every five minutes, which is an an excess of ~568 -requests/10s. <<[!UNRESOLVED kubernetes default for the kubelet client rate -limit and how this changes by machine size]>> <<[UNRESOLVED]>> +requests/10s. <<[!UNRESOLVED non blocking: kubernetes default for the kubelet +client rate limit and how this changes by machine size]>> <<[UNRESOLVED]>> ## Design Details @@ -866,7 +870,7 @@ behaviors common to all pod restarts"](code-diagram-for-restarts.png "Kubelet and Container Runtime restart code paths") ``` - <<[UNRESOLVED answer these question from original PR]>> + <<[UNRESOLVED non blocking answer these question from original PR or make new bugs]>> >Does this [old container cleanup using containerd] include cleaning up the image filesystem? There might be room for some optimization here, if we can reuse the RO layers. to answer question: looks like it is per runtime. need to check about leasees. also part of the value of this is to restart the sandbox. ``` @@ -1089,11 +1093,9 @@ extending the production code to implement this enhancement. --> -- <<[UNRESOLVED whats up with this]>> - `kubelet/kuberuntime/kuberuntime_manager_test`: **could not find a successful +- `kubelet/kuberuntime/kuberuntime_manager_test`: **could not find a successful coverage run on [prow](https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/ci-kubernetes-coverage-unit/1800947623675301888)** - <<[/UNRESOLVED]>> ##### Integration tests @@ -1165,6 +1167,8 @@ feature gates set as per the [Conflict Resolution](#conflict-resolution) policy - Test proving `KubeletConfiguration` objects will silently drop unrecognized fields in the `config.validation_test` package ([ref](https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/apis/config/validation/validation_test.go)). + - <<[UNRESOLVED non blocking]>>Is this also the expected behavior when the feature gate + is disabled?<<[/UNRESOLVED]>> - Test coverage of proper requeue behavior; see https://github.com/kubernetes/kubernetes/issues/123602 - Actually fix https://github.com/kubernetes/kubernetes/issues/123602 if this @@ -1559,7 +1563,7 @@ rollout. Similarly, consider large clusters and how enablement/disablement will rollout across nodes. --> -<<[UNRESOLVED]>> Fill out when targeting beta to a release. <<[/UNRESOLVED]>> +<<[UNRESOLVED beta]>> Fill out when targeting beta to a release. <<[/UNRESOLVED]>> ###### What specific metrics should inform a rollback? @@ -1598,7 +1602,7 @@ Longer term, we may want to require automated upgrade/rollback tests, but we are missing a bunch of machinery and tooling and can't do that now. --> -<<[UNRESOLVED]>> Fill out when targeting beta to a release. <<[/UNRESOLVED]>> +<<[UNRESOLVED beta]>> Fill out when targeting beta to a release. <<[/UNRESOLVED]>> ###### Is the rollout accompanied by any deprecations and/or removals of features, APIs, fields of API types, flags, etc.? @@ -1606,7 +1610,7 @@ are missing a bunch of machinery and tooling and can't do that now. Even if applying deprecation policies, they may still surprise some users. --> -<<[UNRESOLVED]>> Fill out when targeting beta to a release. <<[/UNRESOLVED]>> +<<[UNRESOLVED beta]>> Fill out when targeting beta to a release. <<[/UNRESOLVED]>> ### Monitoring Requirements @@ -1625,7 +1629,7 @@ checking if there are objects with field X set) may be a last resort. Avoid logs or events for this purpose. --> -<<[UNRESOLVED]>> Fill out when targeting beta to a release. <<[/UNRESOLVED]>> +<<[UNRESOLVED beta]>> Fill out when targeting beta to a release. <<[/UNRESOLVED]>> ###### How can someone using this feature know that it is working for their instance? @@ -1638,7 +1642,7 @@ and operation of this feature. Recall that end users cannot usually observe component logs or access metrics. --> -<<[UNRESOLVED]>> Fill out when targeting beta to a release. +<<[UNRESOLVED beta]>> Fill out when targeting beta to a release. - [ ] Events - Event Reason: - [ ] API .status @@ -1666,7 +1670,7 @@ These goals will help you determine what you need to measure (SLIs) in the next question. --> -<<[UNRESOLVED]>> Fill out when targeting beta to a release. <<[/UNRESOLVED]>> +<<[UNRESOLVED beta]>> Fill out when targeting beta to a release. <<[/UNRESOLVED]>> ###### What are the SLIs (Service Level Indicators) an operator can use to determine the health of the service? @@ -1674,7 +1678,7 @@ question. Pick one more of these and delete the rest. --> -<<[UNRESOLVED]>> Fill out when targeting beta to a release. +<<[UNRESOLVED beta]>> Fill out when targeting beta to a release. - [ ] Metrics - Metric name: @@ -1692,7 +1696,7 @@ Describe the metrics themselves and the reasons why they weren't added (e.g., co implementation difficulties, etc.). --> -<<[UNRESOLVED]>> Fill out when targeting beta to a release. <<[/UNRESOLVED]>> +<<[UNRESOLVED beta]>> Fill out when targeting beta to a release. <<[/UNRESOLVED]>> ### Dependencies @@ -1717,7 +1721,7 @@ and creating new ones, as well as about cluster-level services (e.g. DNS): - Impact of its degraded performance or high-error rates on the feature: --> -<<[UNRESOLVED]>> Fill out when targeting beta to a release. <<[/UNRESOLVED]>> +<<[UNRESOLVED beta]>> Fill out when targeting beta to a release. <<[/UNRESOLVED]>> ### Scalability @@ -1855,7 +1859,7 @@ details). For now, we leave it here. ###### How does this feature react if the API server and/or etcd is unavailable? -<<[UNRESOLVED]>> Fill out when targeting beta to a release. <<[/UNRESOLVED]>> +<<[UNRESOLVED beta]>> Fill out when targeting beta to a release. <<[/UNRESOLVED]>> ###### What are other known failure modes? @@ -1874,7 +1878,7 @@ For each of them, fill in the following information by copying the below templat ###### What steps should be taken if SLOs are not being met to determine the problem? -<<[UNRESOLVED]>> Fill out when targeting beta to a release. <<[/UNRESOLVED]>> +<<[UNRESOLVED beta]>> Fill out when targeting beta to a release. <<[/UNRESOLVED]>> ## Implementation History From 220604e3694c8ba8ec582dfd88950ecefc75ccd4 Mon Sep 17 00:00:00 2001 From: lauralorenz Date: Fri, 4 Oct 2024 23:38:14 -0400 Subject: [PATCH 36/36] crashloopbackoff.max -> crashloopbackoff.maxSeconds --- keps/sig-node/4603-tune-crashloopbackoff/README.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/keps/sig-node/4603-tune-crashloopbackoff/README.md b/keps/sig-node/4603-tune-crashloopbackoff/README.md index b59641b0fc0d..557642b805ed 100644 --- a/keps/sig-node/4603-tune-crashloopbackoff/README.md +++ b/keps/sig-node/4603-tune-crashloopbackoff/README.md @@ -411,7 +411,7 @@ its expedient to see the exact config proposed here: apiVersion: kubelet.config.k8s.io/v1beta1 kind: KubeletConfiguration crashloopbackoff: - max: 4 + maxSeconds: 4 ``` ### Refactor and flat rate to 10 minutes for the backoff counter reset threshold @@ -760,7 +760,7 @@ The proposed configuration explicitly looks like this: apiVersion: kubelet.config.k8s.io/v1beta1 kind: KubeletConfiguration crashloopbackoff: - max: 4 + maxSeconds: 4 ``` ### Refactor of recovery threshold @@ -1146,7 +1146,7 @@ heterogenity between "Succeeded" terminating pods, and crashing pods whose #### Alpha -- New `int32 crashloopbackoff.max` field in `KubeletConfiguration` API, validated +- New `int32 crashloopbackoff.maxSeconds` field in `KubeletConfiguration` API, validated to a minimum of 1 and a maximum of 300, used when `EnableKubeletCrashLoopBackoffMax` feature flag enabled, to customize CrashLoopBackOff per node @@ -1300,12 +1300,12 @@ To make use of this enhancement, on cluster upgrade, the `EnableKubeletCrashLoopBackoffMax` feature gate must first be turned on for the cluster. Then, if any nodes need to use a different backoff curve, their kubelet must be completely redeployed either in the same upgrade or after that upgrade -with the `crashloopbackoff.max` `KubeletConfiguration` set. +with the `crashloopbackoff.maxSeconds` `KubeletConfiguration` set. To stop use of this enhancement, there are two options. On a per-node basis, nodes can be completely redeployed with -`crashloopbackoff.max` `KubeletConfiguration` unset. Since kubelet does +`crashloopbackoff.maxSeconds` `KubeletConfiguration` unset. Since kubelet does not cache the backoff object, on kubelet restart they will start from the beginning of their backoff curve (either the original one with initial value 10s, or the new baseline with initial value 1s, depending on whether they've @@ -1343,9 +1343,9 @@ lower than 300s, it will be honored. In other words, operator-invoked configuration will have precedence over the default, even if it is slower, as long as it is valid. -If `crashloopbackoff.max` `KubeletConfiguration` exists but +If `crashloopbackoff.maxSeconds` `KubeletConfiguration` exists but `EnableKubeletCrashLoopBackoffMax` is off, kubelet will log a warning but will -not honor the `crashloopbackoff.max` `KubeletConfiguration`. In other words, +not honor the `crashloopbackoff.maxSeconds` `KubeletConfiguration`. In other words, operator-invoked per node configuration will not be honored if the overall feature gate is turned off.