Cannon: Drain websockets in a controlled fashion on SIGTERM or SIGINT#2416
Merged
Cannon: Drain websockets in a controlled fashion on SIGTERM or SIGINT#2416
Conversation
akshaymankar
changed the title
Do not timeout the actual drain, a subsequent SIGKILL will do that anyway.
akshaymankar
changed the title
jschaul
approved these changes
May 20, 2022
jschaul
approved these changes
May 20, 2022
5 tasks
jschaul
added a commit
that referenced
this pull request
May 25, 2022
By default, incoming network traffic for websockets comes through these network hops: Internet -> LoadBalancer -> kube-proxy -> nginx-ingress-controller -> nginz -> cannon In order to have graceful draining of websockets when something gets restarted (as implemented in #2416 ), as it is not easily possible to implement the graceful draining on nginx-ingress-controller or nginz by itself, with this PR there is now a configuration option to get the following network hops: Internet -> separate LoadBalancer for cannon only -> kube-proxy -> [nginz->cannon (2 containers in the same pod)] More context: https://wearezeta.atlassian.net/wiki/spaces/PS/pages/585564424/How+to+gracefully+drain+cannon+but+not+so+slowly FUTUREWORK: this introduces some nginz config duplication; some way to refactor this (e.g. by moving charts/{cannon, nginz}/* to charts/wire-server/ in a backwards-compatible way) would allow to reduce this duplication. Co-authored-by: jschaul <jschaul@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The plan to implement this is described internally here: https://wearezeta.atlassian.net/wiki/spaces/PS/pages/585564424/How+to+gracefully+drain+cannon+but+not+so+slowly
Related PR: https://github.com/zinfra/cailleach/pull/1092
Checklist
changelog.d.