net/http: race in http2Transport #31192
Comments
bradfitz
changed the title
A race is detected between a bytes.Buffer generated with cmd/rpc.Pool and http2 module. An issue is raised in golang (golang/go#31192). Meanwhile, this commit disables Pool in RPC code and it generates a new 1kb of bytes.Buffer for each RPC call.
A race is detected between a bytes.Buffer generated with cmd/rpc.Pool and http2 module. An issue is raised in golang (golang/go#31192). Meanwhile, this commit disables Pool in RPC code and it generates a new 1kb of bytes.Buffer for each RPC call.
andybons
added
the
NeedsInvestigation
bcmills
changed the title
|
@bradfitz This might be the same issue with http. There is no guarantee that the bodyWriter is complete when roundtrip exits. |
|
Here is a standalone adapted from @vadmeste's original program but here one doesn't have to manually create certificates but also it is reduced down package main
import (
"bytes"
"fmt"
"io"
"io/ioutil"
"log"
"net/http"
"net/http/httptest"
"golang.org/x/net/http2"
)
func main() {
cst := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, "Pong")
}))
if err := http2.ConfigureServer(cst.Config, new(http2.Server)); err != nil {
log.Fatalf("Failed to configure http2 server: %v", err)
}
cst.TLS = cst.Config.TLSConfig
tr := &http.Transport{TLSClientConfig: cst.Config.TLSConfig}
if err := http2.ConfigureTransport(tr); err != nil {
log.Fatalf("Failed to configure http2 transport: %v", err)
}
tr.TLSClientConfig.InsecureSkipVerify = true
cst.StartTLS()
defer cst.Close()
// Then finally create the HTTP client with the HTTP/2 transport.
client := &http.Client{Transport: tr}
for i := 0; i < 3; i++ {
// Fill buf with data
buf := bytes.NewBuffer(bytes.Repeat([]byte("A"), 128*1024))
// Make a POST http call and read and discard response
req, err := http.NewRequest("POST", cst.URL, buf)
resp, err := client.Do(req)
if err != nil {
log.Fatalln(err)
}
if _, err := io.Copy(ioutil.Discard, resp.Body); err != nil {
log.Fatalln(err)
}
resp.Body.Close()
// Accessing buf here causes a RACE WARNING
buf.Reset()
fmt.Printf(".")
}
} |
|
@fraenkel Modifying @odeke-em 's example to use http like so: package main
import (
"bytes"
"fmt"
"io"
"io/ioutil"
"log"
"net/http"
"net/http/httptest"
)
func main() {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, "Pong")
}))
defer srv.Close()
client := srv.Client()
for i := 0; i < 100; i++ {
// Fill buf with data
buf := bytes.NewBuffer(bytes.Repeat([]byte("A"), 128*1024))
// Make a POST http call, read and discard response
req, err := http.NewRequest("POST", srv.URL, buf)
resp, err := client.Do(req)
if err != nil {
log.Fatalln(err)
}
if _, err := io.Copy(ioutil.Discard, resp.Body); err != nil {
log.Fatalln(err)
}
resp.Body.Close()
buf.Reset()
fmt.Printf(".")
}
}I do not observe a data race. The original issue is still present in gotip from
|
|
@pmalek <https://github.com/pmalek> unfortunately your modification
doesn’t create an HTTP/2 server and transport. You either want to have the
long setup version I provided, or if you are using Go1.14, use and
Unstarted httptest.Server and then enable EnableHTTP2 = true and finally
srv.StartTLS() as we do here
https://golang.org/pkg/net/http/httptest/#example_Server_hTTP2
…On Sun, Jun 14, 2020 at 4:27 AM Patryk Małek ***@***.***> wrote:
@fraenkel <https://github.com/fraenkel> Modifying @odeke-em
<https://github.com/odeke-em> 's example to use http like so:
package main
import (
"bytes"
"fmt"
"io"
"io/ioutil"
"log"
"net/http"
"net/http/httptest"
)
func main() {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, "Pong")
}))
defer srv.Close()
client := srv.Client()
for i := 0; i < 100; i++ {
// Fill buf with data
buf := bytes.NewBuffer(bytes.Repeat([]byte("A"), 128*1024))
// Make a POST http call, read and discard response
req, err := http.NewRequest("POST", srv.URL, buf)
resp, err := client.Do(req)
if err != nil {
log.Fatalln(err)
}
if _, err := io.Copy(ioutil.Discard, resp.Body); err != nil {
log.Fatalln(err)
}
resp.Body.Close()
buf.Reset()
fmt.Printf(".")
}
}
I do not observe a data race.
The original issue is still present in gotip from go version devel
+bd486c3 Sat Jun 13 16:03:19 2020 +0000 linux/amd64 and in 1.14.2.
There is no guarantee that the bodyWriter is complete when roundtrip exits.
I'm wondering what might be the fix for this. Could we make it so that the
bodyWriter *is* complete when roundtrip exits?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#31192 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABFL3V6DCJY3VPZ6VVTLIRDRWSX3VANCNFSM4HC2FYMA>
.
|
|
@odeke-em My bad. But the TLS is also required to trigger this bug so both |
|
It took me a bit to reproduce the issue and then find the statement that covers this situation. https://golang.org/pkg/net/http/#RoundTripper The body isn't closed until net/http.(*http2clientStream).writeRequestBody() returns. |
That's interesting because as far as I can see there's no way of telling outside of the e.g. |
|
Looking at this further, the issue is when a GOAWAY is sent from the server all outstanding streams are sent a response errClientConnGotGoAway. However, cs.writeRequestBody() may still be executing. |
|
Good news, its easy to fix. Just have to create a test that always fails. |
|
Change https://golang.org/cl/253259 mentions this issue: |
|
Could you please add this fix to next milestone, as Kubernetes affected. |
|
|
@odeke-em are you available to review the above CL? |
|
We had a request to backport this, as it impacts Kubernetes, per @answer1991. |
|
@dmitshur what do you think about the requested backport from @answer1991 and @networkimprov per #31192 (comment)? |
|
Glad to see the fix merged. |
|
I don't have enough information to know the severity of impact caused by this problem and whether there's a reasonable workaround. We need that information to balance it against the risk of backporting a commit when making a decision. @answer1991 The process for requesting a backport is described at https://golang.org/wiki/MinorReleases. If after reading it you think this issue should be considered for backporting, please use gopherbot to make a request and provide a brief rationale for it. I know you've said this impacts Kubernetes, but please provide a little more detail about how serious it is. Thanks. CC @golang/release. |
|
Kubernetes team had already announced GOAWAY feature in Kubernetes v1.19 release notes. However, if Kubernetes enable GOAWAY feature, the client will receive unexpected 500 status code errors which I described at #41234, and there is no workaround. I think backport to golang 1.15 is necessary. @gopherbot please backport, as Kubernetes is affected and there is no workaround. |
|
Backport issue(s) opened: #42539 (for 1.15). Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://golang.org/wiki/MinorReleases. |
|
Yep, since the issue affects both Go 1.15 and 1.14, it needs to be considered for backport to both, or neither. (We typically double-check that as part reviewing backport requests, but it's helpful to also catch it earlier.) |
|
I don't know why the bot didn't create a cherry-pick to Go 1.14. Because I mentioned 1.15 in my comment? Re-try |
|
Kubernetes team had already announced GOAWAY feature in Kubernetes v1.19 release notes. However, if Kubernetes enable GOAWAY feature, the client will receive unexpected 500 status code errors which I described at #41234, and there is no workaround. I think backport is necessary. @gopherbot please backport, as Kubernetes is affected and there is no workaround. |
|
thanks! |
dmitshur
added
NeedsFix
|
Reopening as the This issue got closed because CL 253259 was submitted with the phrase "Fixes #31192". However that CL is for x/net. The bundled copy of Doing the bundle update was attempted, but it caused an existing |
|
#42498 has been resolved (I believe), and should no longer block doing the bundle update. |
|
Should we backport #42498 too? |
|
Change https://golang.org/cl/275446 mentions this issue: |
|
Change https://golang.org/cl/277012 mentions this issue: |
When the clientConn is done with a request, if there is a request body, we must wait until the body is written otherwise there can be a race when attempting to rewind the body. Fixes golang/go#31192 Fixes golang/go#41234 Change-Id: I77606cc19372eea8bbd8995102354f092b8042be Reviewed-on: https://go-review.googlesource.com/c/net/+/253259 Reviewed-by: Damien Neil <[email protected]> Trust: Emmanuel Odeke <[email protected]> Run-TryBot: Emmanuel Odeke <[email protected]> TryBot-Result: Go Bot <[email protected]>
What version of Go are you using (
go version)?Does this issue reproduce with the latest release?
Yes
What operating system and processor architecture are you using (
go env)?go envOutputWhat did you do?
What did you expect to see?
No race detector warning
What did you see instead?
The text was updated successfully, but these errors were encountered: