net/http: TestClientTimeout* failures with "missing timeout substring" on Windows #43120

bcmills · 2020-12-10T15:00:35Z

#!watchflakes
post <- goos == "windows" && pkg == "net/http" && test ~ `^TestClientTimeout.*` && `missing timeout substring`

2020-12-09T01:34:53-ae9b442/windows-arm-zx2c4
2020-12-07T17:19:41-7f9a2bc/windows-arm-zx2c4
2020-12-03T02:35:36-da54dfb/windows-arm-zx2c4
2020-12-02T20:17:57-2d0258d/windows-arm-zx2c4

--- FAIL: TestClientTimeout_h1 (0.20s)
    client_test.go:1277: error string = "context deadline exceeded"; missing timeout substring
--- FAIL: TestClientTimeout_h2 (0.20s)
    client_test.go:1277: error string = "context deadline exceeded"; missing timeout substring
FAIL
FAIL	net/http	43.328s

CC @zx2c4 @bradfitz @neild @empijei

The text was updated successfully, but these errors were encountered:

zx2c4 · 2020-12-10T15:16:49Z

We increased the timeout here: golang/build@b6220de

But I don't know if that change has been deployed yet. @dmitshur - do you know?

zx2c4 · 2020-12-10T15:18:46Z

Oh, the latest log shows "GO_TEST_TIMEOUT_SCALE=3", so looks like it has been deployed.

So I'm not quite sure...

dmitshur · 2020-12-10T16:17:12Z

Yes, it has. You can find out the deployed version by visiting https://farmer.golang.org, it tells the version at the top:

[...] Uptime 10h21m44s. Version 90d5e7c418c099f6d88cfc0e6269a94fa4da7f0e.

bcmills · 2021-08-31T16:37:26Z

Looks like it has started occurring on other builders too.

greplogs --dashboard -md -l -E 'missing timeout substring' --since=2020-12-10

CC @neild

bcmills · 2021-11-04T01:56:01Z

Curiously, this has still only ever occurred on windows-arm variants and openbsd-amd64-68. I don't know whether we can really discount the two OpenBSD failures, but the apparent specificity to windows-arm is puzzling. (Maybe this is a memory-model synchronization issue, like #48072?)

greplogs --dashboard -md -l -e '(?ms)FAIL: TestClientTimeout.*missing timeout substring' --since=2021-10-01

2021-11-03T18:37:22-cfd016d/windows-arm64-10
2021-10-22T21:26:30-5d414d1/windows-arm64-10
2021-10-14T18:43:59-765c911/windows-arm64-10
2021-10-14T04:18:44-9e4dc6f/windows-arm64-10
2021-10-05T15:47:57-017ffcd/windows-arm64-10

gopherbot · 2021-11-04T02:24:43Z

Change https://golang.org/cl/361275 mentions this issue: net/http: reduce TestClientTimeout_h{1,2} latency

The test had been setting an arbitrary 200ms timeout to allow the server's handler to set up before timing out. That is not only potentially flaky on slow machines, but also typically much longer than necessary. Replace the hard-coded timeout with a much shorter initial timeout, and use exponential backoff to lengthen it if needed. This allows the test to be run about 20x faster in the typical case, which may make it easier to reproduce rare failure modes by running with a higher -count flag. For #43120 Change-Id: I1e0d0ec99d5a107fff56e3bcc7174d686ec582d1 Reviewed-on: https://go-review.googlesource.com/c/go/+/361275 Trust: Bryan C. Mills <[email protected]> Run-TryBot: Bryan C. Mills <[email protected]> TryBot-Result: Go Bot <[email protected]> Reviewed-by: Damien Neil <[email protected]>

bcmills · 2022-01-05T19:14:47Z

greplogs --dashboard -md -l -e '(?ms)FAIL: TestClientTimeout.*missing timeout substring' --since=2021-11-05

2021-12-29T19:23:11-8a306e2/windows-arm64-10
2021-11-12T23:34:09-fdee1b2/windows-arm64-10

bcmills · 2022-01-05T19:19:07Z

windows/arm64 is not currently listed as a first class port, so — given the consistency of the failing platform — I think a skip may be in order to reduce windows/arm64 builder flakiness.

I honestly don't know what to make of the couple of openbsd-amd64-68 failures, though.

(CC @bufflig)

gopherbot · 2022-01-05T19:26:12Z

Change https://golang.org/cl/375635 mentions this issue: net/http: skip TestClientTimeout_h{1,2} on windows/arm and windows/arm64

bcmills · 2022-01-05T19:26:47Z

Marking as release-blocker until the skip is added, at which point this can be moved back to the backlog.

These tests are empirically flaky on the windows/arm and windows/arm64 builders, with a consistent (but rare) failure mode. This change skips the test if that particular failure mode is encountered on those platforms; the skip can be removed if and when someone has the time to pin down the root cause. For #43120 Change-Id: Ie3a9a06bf47e3a907c7b07441acc1494a4631135 Reviewed-on: https://go-review.googlesource.com/c/go/+/375635 Trust: Bryan Mills <[email protected]> Run-TryBot: Bryan Mills <[email protected]> Reviewed-by: Damien Neil <[email protected]> TryBot-Result: Gopher Robot <[email protected]>

bcmills · 2022-01-05T21:12:50Z

The skip is merged. Back to the Backlog. 😩

bcmills · 2022-01-18T17:10:32Z

Apparently TestClientTimeout_Headers_h[12] are also affected by the same underlying bug:

--- FAIL: TestClientTimeout_Headers_h2 (0.01s)
    client_test.go:1342: error string = "Get \"https://127.0.0.1:53626\": context deadline exceeded"; missing timeout substring
--- FAIL: TestClientTimeout_Headers_h1 (0.01s)
    client_test.go:1342: error string = "Get \"http://127.0.0.1:53627\": context deadline exceeded"; missing timeout substring
2022/01/17 01:29:00 http: TLS handshake error from 127.0.0.1:53639: read tcp 127.0.0.1:53638->127.0.0.1:53639: wsarecv: An existing connection was forcibly closed by the remote host.
FAIL
FAIL	net/http	6.571s

greplogs --dashboard -md -l -e '(?ms)FAIL: TestClientTimeout.*missing timeout substring' --since=2022-01-06

2022-01-17T09:23:25-897b3da/windows-arm64-10

gopherbot · 2022-01-18T17:25:27Z

Change https://golang.org/cl/379156 mentions this issue: net/http: skip TestClientTimeout_Headers_h{1,2} on windows/arm and windows/arm64

…ndows/arm64 This extends the skip added in CL 375635 to the "_Headers" variant of the test, since we have observed similar failures in that variant on the builders. For #43120 Change-Id: Ib1c97fbb776b576271629272f3194da77913a941 Reviewed-on: https://go-review.googlesource.com/c/go/+/379156 Trust: Bryan Mills <[email protected]> Run-TryBot: Bryan Mills <[email protected]> TryBot-Result: Gopher Robot <[email protected]> Reviewed-by: Damien Neil <[email protected]>

These tests are empirically flaky on the windows/arm and windows/arm64 builders, with a consistent (but rare) failure mode. This change skips the test if that particular failure mode is encountered on those platforms; the skip can be removed if and when someone has the time to pin down the root cause. For golang#43120 Change-Id: Ie3a9a06bf47e3a907c7b07441acc1494a4631135 Reviewed-on: https://go-review.googlesource.com/c/go/+/375635 Trust: Bryan Mills <[email protected]> Run-TryBot: Bryan Mills <[email protected]> Reviewed-by: Damien Neil <[email protected]> TryBot-Result: Gopher Robot <[email protected]>

…ndows/arm64 This extends the skip added in CL 375635 to the "_Headers" variant of the test, since we have observed similar failures in that variant on the builders. For golang#43120 Change-Id: Ib1c97fbb776b576271629272f3194da77913a941 Reviewed-on: https://go-review.googlesource.com/c/go/+/379156 Trust: Bryan Mills <[email protected]> Run-TryBot: Bryan Mills <[email protected]> TryBot-Result: Gopher Robot <[email protected]> Reviewed-by: Damien Neil <[email protected]>

bcmills · 2022-08-22T19:02:52Z

Looks like the skip isn't quite wide enough — this can also manifest during the initial Get call if the deadline happens to fire before the headers are processed.

2022-08-08T16:40:58-e730703/windows-arm64-10:

--- FAIL: TestClientTimeout_h2 (0.01s)
    client_test.go:1260: attempting test with timeout 10ms
    client_test.go:1275: Get "https://127.0.0.1:51129/?nonce=0": context deadline exceeded
FAIL
FAIL	net/http	6.996s

gopherbot · 2022-08-22T19:38:13Z

Change https://go.dev/cl/425096 mentions this issue: net/http: skip Get flakes in TestClientTimeout tests on windows/arm4

There is already a skip in case of a later failure in the same test on these platforms. Skip the failure if it occurs earlier too. For #43120. Change-Id: Id530370caa6a7df8cae593f6fdcb66871b86b125 Reviewed-on: https://go-review.googlesource.com/c/go/+/425096 Run-TryBot: Bryan Mills <[email protected]> TryBot-Result: Gopher Robot <[email protected]> Auto-Submit: Bryan Mills <[email protected]> Reviewed-by: Damien Neil <[email protected]>

bcmills · 2023-08-30T14:12:55Z

This test is still skipped at HEAD, but there have been a fair number of net/http fixes since it was added.

@neild, do you think it's worth unskipping at this point?

bcmills · 2023-10-13T17:03:58Z

Apparently not fixed: it reproduced on windows/amd64 in #63365 (comment).

(CC @golang/windows)

gopherbot · 2023-10-25T15:04:34Z

Found new dashboard test flakes for:

#!watchflakes
post <- goos == "windows" && pkg == "net/http" && test ~ `^TestClientTimeout.*` && `missing timeout substring`

2023-10-04 02:55 windows-amd64-2016 go@9018601e net/http.TestClientTimeout_Headers (log)

--- FAIL: TestClientTimeout_Headers (0.00s)
    --- FAIL: TestClientTimeout_Headers/h1 (0.01s)
        client_test.go:1313: error string = "Get \"http://127.0.0.1:51755\": context deadline exceeded"; missing timeout substring

— watchflakes

gopherbot · 2024-08-07T18:37:26Z

Found new dashboard test flakes for:

#!watchflakes
post <- goos == "windows" && pkg == "net/http" && test ~ `^TestClientTimeout.*` && `missing timeout substring`

2024-08-07 17:23 gotip-windows-amd64 go@b696250e net/http.TestClientTimeout/h2 (log)

=== RUN   TestClientTimeout/h2
=== PAUSE TestClientTimeout/h2
=== CONT  TestClientTimeout/h2
    client_test.go:1212: attempting test with timeout 10ms
    client_test.go:1259: error string = "context deadline exceeded"; missing timeout substring
--- FAIL: TestClientTimeout/h2 (0.01s)

— watchflakes

gopherbot · 2024-12-17T03:51:32Z

Found new dashboard test flakes for:

#!watchflakes
post <- goos == "windows" && pkg == "net/http" && test ~ `^TestClientTimeout.*` && `missing timeout substring`

2024-12-16 21:43 gotip-windows-amd64 go@d92c34a3 net/http.TestClientTimeout/h1 (log)

=== RUN   TestClientTimeout/h1
=== PAUSE TestClientTimeout/h1
=== CONT  TestClientTimeout/h1
    client_test.go:1212: attempting test with timeout 10ms
    client_test.go:1259: error string = "context deadline exceeded"; missing timeout substring
--- FAIL: TestClientTimeout/h1 (0.01s)

— watchflakes

bcmills added OS-Windows NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Dec 10, 2020

bcmills added this to the Backlog milestone Dec 10, 2020

bcmills mentioned this issue Feb 12, 2021

x/tools/gopls: multiple failures on windows-arm-zx2c4 #44232

Open

bcmills changed the title ~~net/http: TestClientTimeout* failures with "missing timeout substring" on windows-arm-zx2c4~~ net/http: TestClientTimeout* failures with "missing timeout substring" Aug 31, 2021

bcmills removed the OS-Windows label Aug 31, 2021

bcmills added the release-blocker label Jan 5, 2022

bcmills modified the milestones: Backlog, Go1.18 Jan 5, 2022

bcmills removed the release-blocker label Jan 5, 2022

bcmills modified the milestones: Go1.18, Backlog Jan 5, 2022

bcmills added the release-blocker label Jan 18, 2022

bcmills modified the milestones: Backlog, Go1.18 Jan 18, 2022

bcmills changed the title ~~net/http: TestClientTimeout* failures with "missing timeout substring"~~ net/http: TestClientTimeout* failures with "missing timeout substring" on windows/arm* Jan 18, 2022

bcmills added arch-arm Issues solely affecting the 32-bit arm architecture. arch-arm64 OS-Windows and removed release-blocker labels Jan 18, 2022

bcmills modified the milestones: Go1.18, Backlog Jan 18, 2022

bcmills mentioned this issue Aug 22, 2022

build dashboard triage log #52653

Closed

bcmills mentioned this issue Oct 13, 2023

net/http: TestClientTimeout_Headers failures #63365

Closed

bcmills changed the title net/http: TestClientTimeout* failures with "missing timeout substring" on windows/arm* net/http: TestClientTimeout* failures with "missing timeout substring" on Windows Oct 13, 2023

bcmills removed arch-arm64 arch-arm Issues solely affecting the 32-bit arm architecture. labels Oct 13, 2023

bcmills added this to Test Flakes Oct 13, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

net/http: TestClientTimeout* failures with "missing timeout substring" on Windows #43120

net/http: TestClientTimeout* failures with "missing timeout substring" on Windows #43120

bcmills commented Dec 10, 2020 •

edited

Loading

zx2c4 commented Dec 10, 2020

zx2c4 commented Dec 10, 2020

dmitshur commented Dec 10, 2020

bcmills commented Aug 31, 2021

bcmills commented Nov 4, 2021

gopherbot commented Nov 4, 2021

bcmills commented Jan 5, 2022

bcmills commented Jan 5, 2022

gopherbot commented Jan 5, 2022

bcmills commented Jan 5, 2022

bcmills commented Jan 5, 2022

bcmills commented Jan 18, 2022 •

edited

Loading

gopherbot commented Jan 18, 2022

bcmills commented Aug 22, 2022

gopherbot commented Aug 22, 2022

bcmills commented Aug 30, 2023

bcmills commented Oct 13, 2023 •

edited

Loading

gopherbot commented Oct 25, 2023

gopherbot commented Aug 7, 2024

gopherbot commented Dec 17, 2024

net/http: TestClientTimeout* failures with "missing timeout substring" on Windows #43120

net/http: TestClientTimeout* failures with "missing timeout substring" on Windows #43120

Comments

bcmills commented Dec 10, 2020 • edited Loading

zx2c4 commented Dec 10, 2020

zx2c4 commented Dec 10, 2020

dmitshur commented Dec 10, 2020

bcmills commented Aug 31, 2021

bcmills commented Nov 4, 2021

gopherbot commented Nov 4, 2021

bcmills commented Jan 5, 2022

bcmills commented Jan 5, 2022

gopherbot commented Jan 5, 2022

bcmills commented Jan 5, 2022

bcmills commented Jan 5, 2022

bcmills commented Jan 18, 2022 • edited Loading

gopherbot commented Jan 18, 2022

bcmills commented Aug 22, 2022

gopherbot commented Aug 22, 2022

bcmills commented Aug 30, 2023

bcmills commented Oct 13, 2023 • edited Loading

gopherbot commented Oct 25, 2023

gopherbot commented Aug 7, 2024

gopherbot commented Dec 17, 2024

bcmills commented Dec 10, 2020 •

edited

Loading

bcmills commented Jan 18, 2022 •

edited

Loading

bcmills commented Oct 13, 2023 •

edited

Loading