[v16] fix: prevent panic in httputils.Forwarder on client cancellation#55768
Merged
tigrato merged 1 commit intobranch/v16from Jun 16, 2025
Merged
[v16] fix: prevent panic in httputils.Forwarder on client cancellation#55768tigrato merged 1 commit intobranch/v16from
httputils.Forwarder on client cancellation#55768tigrato merged 1 commit intobranch/v16from
Conversation
tigrato
force-pushed
the
bot/backport-55764-branch/v16
branch
2 times, most recently
from
a832c40 to
9fb29d7
Compare
rosstimothy
approved these changes
Jun 16, 2025
espadolini
approved these changes
Jun 16, 2025
public-teleport-github-review-bot
Bot
removed the request for review
from mjsmithnh
…55764) * fix: prevent panic in `httputils.Forwarder` on client cancellation Kubernetes Watchers are usually a long-lived HTTP request where the server pushes new updates to clients. When clients are no longer interested in the stream because they already received the information they were looking for, the client cancels the request. The request cancellation is propagated via context cancelation. Go's `httputils.ReverseProxy` has a special handling for cases where the read or write body operations fail. This happens when the client initially started the request, receive the contents and then cancelled it. The reverse proxy sees the error as `context.Canceled` error - i.e. the client canceled it - but [`copyBuffer`](https://cs.opensource.google/go/go/+/refs/tags/go1.24.4:src/net/http/httputil/reverseproxy.go;l=664-669;drc=e64f7ef03fdfa1c0d847c21b16c9302cc824e79b) only has a special case for `io.EOF`. This means that `context.Canceled` error is propagated to [`ServeHTTP`](https://cs.opensource.google/go/go/+/refs/tags/go1.24.4:src/net/http/httputil/reverseproxy.go;l=520-530;drc=e64f7ef03fdfa1c0d847c21b16c9302cc824e79b) and hits the `shouldPanicOnCopyError` function. This function checks if the function should panic or not. Since we are running inside `http.Server`, the decision is always to panic which caused all watcher resources to never cleanup and accumulate. This PR changes the logic of our `httputils.Forwarder` to remove the `http.ServerContextKey` to avoid any panic. It also fixes a possible deadlock caused by incorrect condition. Changes: - Remove http.ServerContextKey from httputils.Forwarder to prevent panic - Fix potential deadlock caused by incorrect condition This ensures graceful handling of client cancellations and proper resource cleanup for Kubernetes Watchers. Signed-off-by: Tiago Silva <tiago.silva@goteleport.com> * Update lib/httplib/reverseproxy/reverse_proxy.go Co-authored-by: rosstimothy <39066650+rosstimothy@users.noreply.github.com> --------- Signed-off-by: Tiago Silva <tiago.silva@goteleport.com> Co-authored-by: rosstimothy <39066650+rosstimothy@users.noreply.github.com>
tigrato
force-pushed
the
bot/backport-55764-branch/v16
branch
from
9fb29d7 to
c007e11
Compare
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backport #55764 to branch/v16
changelog: Fixes a memory leak in Kubernetes Access caused by resources not being cleaned up when clients terminate watch streams.