net/http, x/net/http2: rapid stream resets can cause excessive work (CVE-2023-39325)

[neild]

A malicious HTTP/2 client which rapidly creates requests and
immediately resets them can cause excessive server resource consumption.
While the total number of requests is bounded to the
http2.Server.MaxConcurrentStreams setting, resetting an in-progress
request allows the attacker to create a new request while the existing
one is still executing.

HTTP/2 servers now bound the number of simultaneously executing
handler goroutines to the stream concurrency limit. New requests
arriving when at the limit (which can only happen after the client
has reset an existing, in-flight request) will be queued until a
handler exits. If the request queue grows too large, the server
will terminate the connection.

This issue is also fixed in golang.org/x/net/http2 v0.17.0,
for users manually configuring HTTP/2.

The default stream concurrency limit is 250 streams (requests)
per HTTP/2 connection. This value may be adjusted using the
golang.org/x/net/http2 package; see the Server.MaxConcurrentStreams
setting and the ConfigureServer function.

This is CVE-2023-39325 and Go issue https://go.dev/issue/63417.
This is also tracked by CVE-2023-44487.

---

This is a PRIVATE issue for CVE-2023-39325, tracked in http://b/303836512.

/cc @golang/security and @golang/release

[neild]

@gopherbot please open backport issues

[gopherbot]

Backport issue(s) opened: #63426 (for 1.20), #63427 (for 1.21).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://go.dev/wiki/MinorReleases.

[gopherbot]

Change https://go.dev/cl/534215 mentions this issue: http2: limit maximum handler goroutines to MaxConcurrentStreams

[gopherbot]

Change https://go.dev/cl/534235 mentions this issue: [release-branch.go1.21] net/http: regenerate h2_bundle.go

[gopherbot]

Change https://go.dev/cl/534255 mentions this issue: [release-branch.go1.20] net/http: regenerate h2_bundle.go

[dmitshur]

Reopening to track bundling into the main tree, for Go 1.22.

[gopherbot]

Change https://go.dev/cl/534295 mentions this issue: all: pull in x/net v0.17.0 and its dependencies

[gopherbot]

Change https://go.dev/cl/534218 mentions this issue: [internal-branch.go1.21-vendor] http2: limit maximum handler goroutines to MaxConcurrentStreams

[gopherbot]

Change https://go.dev/cl/534236 mentions this issue: [internal-branch.go1.20-vendor] http2: limit maximum handler goroutines to MaxConcurrentStreams

[gopherbot]

Change https://go.dev/cl/534297 mentions this issue: [release-branch.go1.20] all: tidy dependency versioning after release

[gopherbot]

Change https://go.dev/cl/534415 mentions this issue: [release-branch.go1.21] all: tidy dependency versioning after release

[ATP3530]

Does this CVE-2023-39325 impacts Go version 1.18? If yes is there a plan to backport required fix into version 1.18? Or we have to move to the latest version of go (1.20 or 1.21) to have this CVE fixed

[neild]

1.18 is affected. There are no plans to backport to 1.18, which is out of the support window; see: https://go.dev/doc/devel/release#policy

As an alternative to upgrading to a recent Go version, you can use a recent version of the golang.org/x/net/http2 package and http2.ConfigureServer to replace the standard library's bundled HTTP/2 implementation, although I'd really recommend upgrading to a Go version that's still getting security fixes.

[aojea]

FYI, this fix is not sufficient for Kube, and possibly other projects. We are working on this in kubernetes/kubernetes#121120. pprof svg from a single client with a single connection attempting to DOS a Kube API server that has this fix and max streams set to 100 (memory usage grew to 5 GB in a few mins before it stabilized - with multiple connections the API server would have easily OOM'd):

@enj from the pprof attached the bulk of the memory consumed comes from the 4kb buffer allocated to the responseWriter that is created after the Headers frame is received. The buffer is also obtained from the sync.Pool that will make the memory usage more efficient.

https://github.com/golang/net/blob/f12db26b1c9293fa3eb95c936e548d2c1fba4ba9/http2/server.go#L69-L75

In this case, since that the attack is based only on headers frames, postponing the allocation of the responseWriter will reduce a lot the impact on memory usage, however, I've been checking around the code and I don't know how feasible is to do that, based on the rfc7540

Sending or receiving a HEADERS frame causes the stream to become "open"

A stream in the "open" state may be used by both peers to send frames of any type. In this state, sending peers observe
advertised stream-level flow-control limits (Section 5.2)

[tuk2]

The issue with HTTP/2 rapid reset is still active. When an HTTP/2 server is attacked using rapid resets, it results in memory leaks (up to peak memory usage). This problem is especially noticeable when using net/http/httputil (reverse proxy). IMPORTANT: The memory leak occurs ONLY during rapid reset attacks.

[tuk2]

@bradfitz