-
Notifications
You must be signed in to change notification settings - Fork 4.1k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[6.3.0] Implement failure circuit breaker (#18541)
* feat: Implement failure circuit breaker Copy of #18120: I accidentally closed #18120 during rebase and doesn't have permission to reopen. We have noticed that any problems with the remote cache have a detrimental effect on build times. On investigation we found that the interface for the circuit breaker was left unimplemented. To address this issue, implemented a failure circuit breaker, which includes three new Bazel flags: 1) experimental_circuitbreaker_strategy, 2) experimental_remote_failure_threshold, and 3) experimental_emote_failure_window. In this implementation, I have implemented failure strategy for circuit breaker and used failure count to trip the circuit. Reasoning behind using failure count instead of failure rate : To measure failure rate I also need the success count. While both the failure and success count need to be an AtomicInteger as both will be modified concurrently by multiple threads. Even though getAndIncrement is very light weight operation, at very high request it might contribute to latency. Reasoning behind using failure circuit breaker : A new instance of Retrier.CircuitBreaker is created for each build. Therefore, if the circuit breaker trips during a build, the remote cache will be disabled for that build. However, it will be enabled again for the next build as a new instance of Retrier.CircuitBreaker will be created. If needed in the future we may add cool down strategy also. e.g. failure_and_cool_down_startegy. closes #18136 Closes #18359. PiperOrigin-RevId: 536349954 Change-Id: I5e1c57d4ad0ce07ddc4808bf1f327bc5df6ce704 * remove target included in cherry-pick by mistake
- Loading branch information
Showing
17 changed files
with
578 additions
and
176 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -243,4 +243,8 @@ public void close() { | |
| } | ||
| channel.release(); | ||
| } | ||
|
|
||
| RemoteRetrier getRetrier() { | ||
| return this.retrier; | ||
| } | ||
| } | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
23 changes: 23 additions & 0 deletions
23
src/main/java/com/google/devtools/build/lib/remote/circuitbreaker/BUILD
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,23 @@ | ||
| load("@rules_java//java:defs.bzl", "java_library") | ||
|
|
||
| package( | ||
| default_applicable_licenses = ["//:license"], | ||
| default_visibility = ["//src:__subpackages__"], | ||
| ) | ||
|
|
||
| filegroup( | ||
| name = "srcs", | ||
| srcs = glob(["*"]), | ||
| visibility = ["//src:__subpackages__"], | ||
| ) | ||
|
|
||
| java_library( | ||
| name = "circuitbreaker", | ||
| srcs = glob(["*.java"]), | ||
| deps = [ | ||
| "//src/main/java/com/google/devtools/build/lib/remote:Retrier", | ||
| "//src/main/java/com/google/devtools/build/lib/remote/common:cache_not_found_exception", | ||
| "//src/main/java/com/google/devtools/build/lib/remote/options", | ||
| "//third_party:guava", | ||
| ], | ||
| ) |
45 changes: 45 additions & 0 deletions
45
src/main/java/com/google/devtools/build/lib/remote/circuitbreaker/CircuitBreakerFactory.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,45 @@ | ||
| // Copyright 2023 The Bazel Authors. All rights reserved. | ||
| // | ||
| // Licensed under the Apache License, Version 2.0 (the "License"); | ||
| // you may not use this file except in compliance with the License. | ||
| // You may obtain a copy of the License at | ||
| // | ||
| // http://www.apache.org/licenses/LICENSE-2.0 | ||
| // | ||
| // Unless required by applicable law or agreed to in writing, software | ||
| // distributed under the License is distributed on an "AS IS" BASIS, | ||
| // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| // See the License for the specific language governing permissions and | ||
| // limitations under the License. | ||
| package com.google.devtools.build.lib.remote.circuitbreaker; | ||
|
|
||
| import com.google.common.collect.ImmutableSet; | ||
| import com.google.devtools.build.lib.remote.Retrier; | ||
| import com.google.devtools.build.lib.remote.common.CacheNotFoundException; | ||
| import com.google.devtools.build.lib.remote.options.RemoteOptions; | ||
|
|
||
| /** Factory for {@link Retrier.CircuitBreaker} */ | ||
| public class CircuitBreakerFactory { | ||
|
|
||
| public static final ImmutableSet<Class<? extends Exception>> DEFAULT_IGNORED_ERRORS = | ||
| ImmutableSet.of(CacheNotFoundException.class); | ||
|
|
||
| private CircuitBreakerFactory() {} | ||
|
|
||
| /** | ||
| * Creates the instance of the {@link Retrier.CircuitBreaker} as per the strategy defined in | ||
| * {@link RemoteOptions}. In case of undefined strategy defaults to {@link | ||
| * Retrier.ALLOW_ALL_CALLS} implementation. | ||
| * | ||
| * @param remoteOptions The configuration for the CircuitBreaker implementation. | ||
| * @return an instance of CircuitBreaker. | ||
| */ | ||
| public static Retrier.CircuitBreaker createCircuitBreaker(final RemoteOptions remoteOptions) { | ||
| if (remoteOptions.circuitBreakerStrategy == RemoteOptions.CircuitBreakerStrategy.FAILURE) { | ||
| return new FailureCircuitBreaker( | ||
| remoteOptions.remoteFailureThreshold, | ||
| (int) remoteOptions.remoteFailureWindowInterval.toMillis()); | ||
| } | ||
| return Retrier.ALLOW_ALL_CALLS; | ||
| } | ||
| } |
83 changes: 83 additions & 0 deletions
83
src/main/java/com/google/devtools/build/lib/remote/circuitbreaker/FailureCircuitBreaker.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,83 @@ | ||
| // Copyright 2023 The Bazel Authors. All rights reserved. | ||
| // | ||
| // Licensed under the Apache License, Version 2.0 (the "License"); | ||
| // you may not use this file except in compliance with the License. | ||
| // You may obtain a copy of the License at | ||
| // | ||
| // http://www.apache.org/licenses/LICENSE-2.0 | ||
| // | ||
| // Unless required by applicable law or agreed to in writing, software | ||
| // distributed under the License is distributed on an "AS IS" BASIS, | ||
| // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| // See the License for the specific language governing permissions and | ||
| // limitations under the License. | ||
| package com.google.devtools.build.lib.remote.circuitbreaker; | ||
|
|
||
| import com.google.common.collect.ImmutableSet; | ||
| import com.google.devtools.build.lib.remote.Retrier; | ||
| import java.util.concurrent.Executors; | ||
| import java.util.concurrent.ScheduledExecutorService; | ||
| import java.util.concurrent.TimeUnit; | ||
| import java.util.concurrent.atomic.AtomicInteger; | ||
|
|
||
| /** | ||
| * The {@link FailureCircuitBreaker} implementation of the {@link Retrier.CircuitBreaker} prevents | ||
| * further calls to a remote cache once the number of failures within a given window exceeds a | ||
| * specified threshold for a build. In the context of Bazel, a new instance of {@link | ||
| * Retrier.CircuitBreaker} is created for each build. Therefore, if the circuit breaker trips during | ||
| * a build, the remote cache will be disabled for that build. However, it will be enabled again for | ||
| * the next build as a new instance of {@link Retrier.CircuitBreaker} will be created. | ||
| */ | ||
| public class FailureCircuitBreaker implements Retrier.CircuitBreaker { | ||
|
|
||
| private State state; | ||
| private final AtomicInteger failures; | ||
| private final int failureThreshold; | ||
| private final int slidingWindowSize; | ||
| private final ScheduledExecutorService scheduledExecutor; | ||
| private final ImmutableSet<Class<? extends Exception>> ignoredErrors; | ||
|
|
||
| /** | ||
| * Creates a {@link FailureCircuitBreaker}. | ||
| * | ||
| * @param failureThreshold is used to set the number of failures required to trip the circuit | ||
| * breaker in given time window. | ||
| * @param slidingWindowSize the size of the sliding window in milliseconds to calculate the number | ||
| * of failures. | ||
| */ | ||
| public FailureCircuitBreaker(int failureThreshold, int slidingWindowSize) { | ||
| this.failureThreshold = failureThreshold; | ||
| this.failures = new AtomicInteger(0); | ||
| this.slidingWindowSize = slidingWindowSize; | ||
| this.state = State.ACCEPT_CALLS; | ||
| this.scheduledExecutor = | ||
| slidingWindowSize > 0 ? Executors.newSingleThreadScheduledExecutor() : null; | ||
| this.ignoredErrors = CircuitBreakerFactory.DEFAULT_IGNORED_ERRORS; | ||
| } | ||
|
|
||
| @Override | ||
| public State state() { | ||
| return this.state; | ||
| } | ||
|
|
||
| @Override | ||
| public void recordFailure(Exception e) { | ||
| if (!ignoredErrors.contains(e.getClass())) { | ||
| int failureCount = failures.incrementAndGet(); | ||
| if (slidingWindowSize > 0) { | ||
| var unused = | ||
| scheduledExecutor.schedule( | ||
| failures::decrementAndGet, slidingWindowSize, TimeUnit.MILLISECONDS); | ||
| } | ||
| // Since the state can only be changed to the open state, synchronization is not required. | ||
| if (failureCount > this.failureThreshold) { | ||
| this.state = State.REJECT_CALLS; | ||
| } | ||
| } | ||
| } | ||
|
|
||
| @Override | ||
| public void recordSuccess() { | ||
| // do nothing, implement if we need to set threshold on failure rate instead of count. | ||
| } | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.