x/tools/gopls/internal/regtest/*: frequent failures on smaller builders #46183

bcmills · 2021-05-15T02:27:06Z

CC @stamblerre @findleyr

bcmills · 2021-05-15T02:28:35Z

Given that these failures are daily-or-better, and especially given that they affect the linux/arm builder (which is a first-class port), marking as release-blocker for Go 1.17 via #11811 (CC @golang/release).

bcmills · 2021-05-15T02:30:46Z

(This may also be a contributing factor for #45931. Perhaps we need to scale back the concurrency in some of the tests in x/tools in general?)

findleyr · 2021-05-16T23:01:48Z

I'm not sure if this would be a contributing factor in #45931, considering this is a separate module. My understanding was that tests for distinct modules run sequentially. If not, we should change that.

That's a long list of failures, though among first-class ports it appears to be almost entirely Darwin and Scaleway failures. The Darwin failures from Feb/March have been resolved by upgrading out MacStadium cluster, the Scaleway builders are grayed out on the build dashboard and slated for deprecation.

Not to say we shouldn't fix this: we absolutely should. I can start by reducing parallelism for tests in the gopls module, and skipping the regtests on severely broken builders. I'd prefer not to skip freebsd and openbsd, but if the flakes don't stop we can skip those as well.

gopherbot · 2021-05-24T23:54:08Z

Change https://golang.org/cl/322409 mentions this issue: internal/testenv: update linux-arm to linux-arm-scaleway

CL 303230 renamed the linux-arm builder to linux-arm-scaleway, but the x/tools ExitIfSmallMachine guard was not updated accordingly, resulting in consistent failures for some tests. For golang/go#45931 For golang/go#46183 Updates golang/go#32834 Change-Id: Ief5f17fc61cb38cf6b7cf63b6cd5e64f9d56261e Reviewed-on: https://go-review.googlesource.com/c/tools/+/322409 Trust: Robert Findley <[email protected]> Run-TryBot: Robert Findley <[email protected]> gopls-CI: kokoro <[email protected]> TryBot-Result: Go Bot <[email protected]> Reviewed-by: Rebecca Stambler <[email protected]>

findleyr · 2021-05-25T13:53:03Z

Interesting: TestQuickFixEmptyFiles is by far the most frequent test to fail, particularly on openbsd. Notably, the test does not exhibit any particular slowness, but just hangs computing diagnostics after one of the edits. For example https://build.golang.org/log/aec98163fc527daf71584d13feab2a00684f7644

There may be a real bug in gopls. This test makes several edits in quick succession; I wonder if we have a deadlock. While this failure sometimes coincides with an apparent gcimporter OOM, the test process is still able to clean up and write the logs, so it doesn't seem to be an OOM, and any OOM of the go command should be reported by gopls.

findleyr · 2021-05-25T14:00:46Z

@bcmills the linux-arm failures turned out to be due to linux-arm being renamed to linux-arm-scaleway. We already had categorized linux-arm as a 'Small Machine', which the regtests and gcimporter test skip.

Given that, and the fact that the darwin failures have been largely mitigated by the macstadium upgrade, are you OK removing the 'release-blocker' label from this issue?

bcmills · 2021-05-25T14:22:50Z

For the most recent couple of runs, I'm seeing several gcimporter failures (tracked separately as #38415), and two failures in golang.org/x/tools/gopls/internal/regtest/workspace on android-*-emu builders.

I think if we can add appropriate skips for the android builders too, that will resolve the release-blocker parts of this issue.

gopherbot · 2021-05-25T20:49:18Z

Change https://golang.org/cl/322649 mentions this issue: gopls/internal/regtest: skip TestDeleteModule_Interdependent

This test is flaky due to orphaned file reloading. Since it is for an experimental feature, skip it until we understand the problem. For golang/go#46375 Updates golang/go#46183 Change-Id: Id6b369f2b61730c8503e3532547708f7689343c3 Reviewed-on: https://go-review.googlesource.com/c/tools/+/322649 Trust: Robert Findley <[email protected]> Run-TryBot: Robert Findley <[email protected]> gopls-CI: kokoro <[email protected]> TryBot-Result: Go Bot <[email protected]> Reviewed-by: Rebecca Stambler <[email protected]>

findleyr · 2021-05-26T01:06:14Z

two failures in golang.org/x/tools/gopls/internal/regtest/workspace on android-*-emu builders.

Thanks for pointing that out. I looked in to the particular test that is flaking on Android, and it appears to just be a gopls bug. Filed as a separate issue and skipped the test for now.

Will wait for a bit more evidence tomorrow, then remove the release-blocker label.

findleyr · 2021-05-27T13:44:11Z

Ok, we've had no failures on Android in the 16 runs since the CL was submitted to skip the problematic test, so I'll remove the release-blocker label now.

bcmills · 2021-05-27T14:13:09Z

FWIW, even at the latest CL I see several failures on various smaller builders:

2021-05-25T21:54:09-a3eb095/dragonfly-amd64
2021-05-25T21:54:09-a3eb095/freebsd-arm64-dmgk
2021-05-25T21:54:09-a3eb095/netbsd-arm64-bsiegert
2021-05-25T21:54:09-a3eb095/plan9-386-0intro

Perhaps we need to expand the list of suspected-small builders? (But I thought plan9 was already in that list...)

CL 303230 renamed the linux-arm builder to linux-arm-scaleway, but the x/tools ExitIfSmallMachine guard was not updated accordingly, resulting in consistent failures for some tests. For golang/go#45931 For golang/go#46183 Updates golang/go#32834 Change-Id: Ief5f17fc61cb38cf6b7cf63b6cd5e64f9d56261e Reviewed-on: https://go-review.googlesource.com/c/tools/+/322409 Trust: Robert Findley <[email protected]> Run-TryBot: Robert Findley <[email protected]> gopls-CI: kokoro <[email protected]> TryBot-Result: Go Bot <[email protected]> Reviewed-by: Rebecca Stambler <[email protected]>

bcmills added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label May 15, 2021

gopherbot added Tools This label describes issues relating to any tools in the x/tools repository. gopls Issues related to the Go language server, gopls. labels May 15, 2021

gopherbot added this to the Unreleased milestone May 15, 2021

bcmills added the release-blocker label May 15, 2021

bcmills modified the milestones: Unreleased, Go1.17 May 15, 2021

bcmills added the okay-after-beta1 Used by release team to mark a release-blocker issue as okay to resolve either before or after beta1 label May 15, 2021

bcmills changed the title ~~x/tools/gopls/internal/regtest/workspace: frequent failures on smaller builders~~ x/tools/gopls/internal/regtest/*: frequent failures on smaller builders May 15, 2021

findleyr mentioned this issue May 25, 2021

x/tools/gopls/internal/regtest: TestDeleteModule_Interdependent flakes due to orphaned file reloading #46375

Closed

findleyr removed okay-after-beta1 Used by release team to mark a release-blocker issue as okay to resolve either before or after beta1 release-blocker labels May 27, 2021

findleyr modified the milestones: Go1.17, gopls/unplanned May 27, 2021

LeGEC mentioned this issue Jun 29, 2021

x/tools/gopls: TestResolveImportCycle is flaky #46773

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

x/tools/gopls/internal/regtest/*: frequent failures on smaller builders #46183

x/tools/gopls/internal/regtest/*: frequent failures on smaller builders #46183

bcmills commented May 15, 2021 •

edited

Loading

bcmills commented May 15, 2021

bcmills commented May 15, 2021

findleyr commented May 16, 2021

gopherbot commented May 24, 2021

findleyr commented May 25, 2021

findleyr commented May 25, 2021

bcmills commented May 25, 2021

gopherbot commented May 25, 2021

findleyr commented May 26, 2021

findleyr commented May 27, 2021

bcmills commented May 27, 2021

x/tools/gopls/internal/regtest/*: frequent failures on smaller builders #46183

x/tools/gopls/internal/regtest/*: frequent failures on smaller builders #46183

Comments

bcmills commented May 15, 2021 • edited Loading

bcmills commented May 15, 2021

bcmills commented May 15, 2021

findleyr commented May 16, 2021

gopherbot commented May 24, 2021

findleyr commented May 25, 2021

findleyr commented May 25, 2021

bcmills commented May 25, 2021

gopherbot commented May 25, 2021

findleyr commented May 26, 2021

findleyr commented May 27, 2021

bcmills commented May 27, 2021

bcmills commented May 15, 2021 •

edited

Loading