outlier ejection: implement consecutive 5xx ejection#261
Merged
mattklein123 merged 2 commits intomasterfrom Dec 1, 2016
Merged
outlier ejection: implement consecutive 5xx ejection#261mattklein123 merged 2 commits intomasterfrom
mattklein123 merged 2 commits intomasterfrom
Conversation
This commit sets up the ability to automatically eject hosts based on N consecutive 5xx responses. Hosts are ejected for an ejection interval multiplied by the number of times the host has been ejected (so if a host keeps getting ejected it will be ejected for longer and longer periods). The ejection interval, number of consecutive 5xxs required for ejection, and the maximum % of a cluster that can be auto ejected are all runtime configurable. This commit does not include documentation. The feature is going to evolve a bit and we will do documentation at a later time when the feature is more baked. The main thing missing from this is the ability to log ejections/additions so that external monitoring/alarming systems can visualize what is happening since it's possible for auto ejection to be quite confusing if there isn't enough information.
Member
Author
|
@lyft/network-team |
| void OutlierDetectorHostSinkImpl::putHttpResponseCode(uint64_t response_code) { | ||
| if (Http::CodeUtility::is5xx(response_code)) { | ||
| if (++consecutive_5xx_ == | ||
| detector_.runtime().snapshot().getInteger("outlier_detection.consecutive_5xx", 5)) { |
Member
There was a problem hiding this comment.
create jira on docs for outlier detection and runtime params?
Member
Author
There was a problem hiding this comment.
|
|
||
| private: | ||
| void addHostSink(HostPtr host); | ||
| void armIntervalTimer(); |
Member
There was a problem hiding this comment.
nit: call that in a way to easy get that this is associated with host unejection
Member
Author
There was a problem hiding this comment.
I named it like this because eventually it will be used for histogram computation, not just unejection.
| Runtime::Loader& runtime_; | ||
| SystemTimeSource& time_source_; | ||
| OutlierDetectionStats stats_; | ||
| Event::TimerPtr interval_timer_; |
Member
There was a problem hiding this comment.
nit: uneject_timer_ may be
Member
Author
There was a problem hiding this comment.
I named it like this because eventually it will be used for histogram computation, not just unejection.
RomanDzhabarov
approved these changes
Dec 1, 2016
rshriram
pushed a commit
to rshriram/envoy
that referenced
this pull request
Oct 30, 2018
lizan
pushed a commit
to lizan/envoy
that referenced
this pull request
Nov 25, 2019
Signed-off-by: gargnupur <gargnupur@google.com>
mathetake
added a commit
that referenced
this pull request
Mar 3, 2026
**Commit Message** Added glossary.md to the docs directory referencing [this draft](https://docs.google.com/document/d/1Mcsl-N2qwBOTThgxbCioUG-bOWiBM_5HpoxXFLE9OYw/edit?usp=sharing). This glossary makes the docs more accessible to beginners and users that may need refreshers on key terms. **Related Issues/PRs (if applicable)** Fixes #239 --------- Signed-off-by: Takeshi Yoneda <t.y.mathetake@gmail.com> Signed-off-by: Erica Hughberg <erica.sundberg.90@gmail.com> Signed-off-by: melsal13 <mmvsal13@gmail.com> Signed-off-by: ericmariasis <ericmariasis829@gmail.com> Signed-off-by: Sébastien Han <seb@redhat.com> Signed-off-by: soma00333 <soma03432303@gmail.com> Signed-off-by: Loong <long0dai@foxmail.com> Signed-off-by: Dan Sun <dsun20@bloomberg.net> Co-authored-by: Takeshi Yoneda <t.y.mathetake@gmail.com> Co-authored-by: Eric Mariasis <ericmariasis829@gmail.com> Co-authored-by: Sébastien Han <seb@redhat.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Soma Utsumi <53121322+soma00333@users.noreply.github.com> Co-authored-by: Loong Dai <long0dai@foxmail.com> Co-authored-by: Dan Sun <dsun20@bloomberg.net> Co-authored-by: Erica Hughberg <erica.sundberg.90@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This commit sets up the ability to automatically eject hosts based on
N consecutive 5xx responses. Hosts are ejected for an ejection interval
multiplied by the number of times the host has been ejected (so if a host
keeps getting ejected it will be ejected for longer and longer periods).
The ejection interval, number of consecutive 5xxs required for ejection,
and the maximum % of a cluster that can be auto ejected are all runtime
configurable.
This commit does not include documentation. The feature is going to evolve
a bit and we will do documentation at a later time when the feature is more
baked.
The main thing missing from this is the ability to log ejections/additions
so that external monitoring/alarming systems can visualize what is happening
since it's possible for auto ejection to be quite confusing if there isn't
enough information.