-
-
Notifications
You must be signed in to change notification settings - Fork 4.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(utils): Add stub for rate-limit-based CircuitBreaker
class
#74557
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
the
Scope: Backend
Automatically applied to PRs that change backend components
label
Jul 19, 2024
This was referenced Jul 19, 2024
vartec
approved these changes
Jul 19, 2024
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
some nitpicks, but LGTM!
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #74557 +/- ##
=======================================
Coverage 78.14% 78.14%
=======================================
Files 6730 6731 +1
Lines 300022 300065 +43
Branches 51618 51620 +2
=======================================
+ Hits 234455 234495 +40
- Misses 59242 59245 +3
Partials 6325 6325
|
lobsterkatie
force-pushed
the
kmclb-add-CircuitBreaker-class-stub
branch
from
July 23, 2024 00:47
f9e498f
to
2ba434a
Compare
lobsterkatie
force-pushed
the
kmclb-add-CircuitBreaker-class-stub
branch
from
July 23, 2024 06:16
2ba434a
to
7fc3a3d
Compare
lobsterkatie
added a commit
that referenced
this pull request
Jul 23, 2024
#74559) This is a follow-up to #74557, which added the beginnings of a rate-limit-based circuit breaker, in the form of a new `CircuitBreaker` class. In this PR, various helpers, for checking the state of the breaker and the underlying rate limiters and for communicating with redis, have been added to the class. It also adds a `MockCircuitBreaker` subclass for use in tests, which contains a number of methods for mocking both circuit breaker and rate limiter state. Note that though these helpers don't have accompanying tests, they are tested (indirectly) in the final PR in the series[1], as part of testing the methods which use them. [1] #74560
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This adds a new
CircuitBreaker
class, which will eventually contain the methods necessary for a rate-limit-based circuit breaker implementation. Our current circuit breaker implementation has four drawbacks that this is aiming to solve:circuit_breaker_activated
function checks how many errors have been tallied, but doesn't have an accompanying way to actually do the tallying, forcing a reimplementation of the error tracking piece in each scenario in which it's used.with_circuit_breaker
wrapper fixes that, but only works if the errors are allowed to bubble from the (potentially fairly low-level) spot where we're actually making the request all the way up to the (potentially much higher-level) spot where we're deciding whether or not to make the request in the first place. Handle the errors gracefully anywhere below thewith_circuit_breaker
call and the breaker has no idea they happened, negating the entire purpose of keeping track.try-except
,with_circuit_breaker
a) counts every kind of exception as an errored request, when that may or may not be accurate, and b) can only track instances of actualException
s, not any other problem (like a 500 response) which may occur.To solve these problems, the new implementation:
Exception
spots (solving both halves of problem 3).This PR adds the class (along with a
CircuitBreakerConfig
class), documentation of how to use it, a constructor which sets up the appropriate rate limits, and tests of said constructor. Future PRs will add other helper methods and add and test the core error-tracking and request-gating methods.The initial use of the circuit breaker will be for Seer-based grouping during ingest. Once we're happy with how it's working, we can go back and refactor the places which use the current circuit breaker to use this new one.