x/tools/gopls: frequent failures with "panic: operation on destroyed generation" #48774

bcmills · 2021-10-04T19:18:36Z

greplogs --dashboard -md -l -e 'operation on destroyed generation'

CC @stamblerre

The text was updated successfully, but these errors were encountered:

bcmills · 2021-10-04T19:18:49Z

At this failure rate, this looks like a release-blocker for Go 1.18 via #11811.

LeGEC · 2021-10-07T15:21:13Z

(note : I don't know if it's relevant to the current issue, I met this while trying to reproduce the failure in this ticket)

There seems to be an issue with how the test environment is setup/destroyed, which shows up on my machine when running tests with --count=2.

To target a single test, here is the output of what I get when targetting TestPackageCompletion/package_completion_at_valid_position within gopls/internal/regtest/completion :

# running with -count=1 always works :
$ go test -count=1 golang.org/x/tools/gopls/internal/regtest/completion -run TestPackageCompletion/package_completion_at_valid_position
ok  	golang.org/x/tools/gopls/internal/regtest/completion	0.096s

# running with -count=2 always fails :
$ go test -count=2 golang.org/x/tools/gopls/internal/regtest/completion -run TestPackageCompletion/package_completion_at_valid_position
--- FAIL: TestPackageCompletion (0.00s)
    --- FAIL: TestPackageCompletion/package_completion_at_valid_position (0.00s)
        --- FAIL: TestPackageCompletion/package_completion_at_valid_position/singleton (0.00s)
            runner.go:283: mkdir /tmp/gopls-regtest-3631473615/TestPackageCompletion/package_completion_at_valid_position/singleton/gopath: file exists
        --- FAIL: TestPackageCompletion/package_completion_at_valid_position/experimental (0.00s)
            runner.go:283: mkdir /tmp/gopls-regtest-3631473615/TestPackageCompletion/package_completion_at_valid_position/experimental/gopath: file exists
FAIL
closing test runner: errors closing the test runner:
	error(s) cleaning sandbox: cleaning modcache: go command failed (stdout: ) (stderr: ): chdir /tmp/gopls-regtest-3631473615/TestPackageCompletion/package_completion_at_valid_position/singleton/work: no such file or directory; removing files: <nil>
	error(s) cleaning sandbox: cleaning modcache: go command failed (stdout: ) (stderr: ): chdir /tmp/gopls-regtest-3631473615/TestPackageCompletion/package_completion_at_valid_position/experimental/work: no such file or directory; removing files: <nil>
FAIL	golang.org/x/tools/gopls/internal/regtest/completion	0.097s
FAIL

to be clear : I see this running any test in this package (TestSourceFixup, TestUnimportedCompletion ...) with --count=2. I haven't tried tests outside this specific package.

Happens at current master (commit d477ef3e), and at the few commits I tested among the list posted by @bcmills .

findleyr · 2021-11-03T19:18:27Z

I'll address this before beta1.

findleyr · 2021-11-03T19:27:53Z

s/before beta1/soon.

We know why this happens (there are many races during shutdown), but I think it can be avoided.

gopherbot · 2021-11-05T17:25:42Z

Change https://golang.org/cl/361655 mentions this issue: internal/memoize: don't pass a context to Acquire

findleyr · 2021-11-10T00:15:26Z

This is going to have to wait until next week. I went down a rabbit hole on gopls shutdown on Friday, and didn't dig my way out in time. I need to focus on the type checker this week.

toothrot · 2021-11-19T17:30:22Z

This issue is currently blocking the first beta release of Go 1.18. Are there any updates?

findleyr · 2021-11-19T17:35:03Z

I had planned to continue work on this today. But with @bcmills agreement I think this should be marked okay-after-beta1. It's not actually a Go issue, and based on @bcmills' thorough investigation I think we know that it's not masking failures right now (the problem being that it would mask failures in the future).

gopherbot · 2021-11-29T20:09:26Z

Change https://golang.org/cl/367675 mentions this issue: internal/memoize: do not allow (*Generation).Acquire to fail

gopherbot · 2021-11-29T20:09:26Z

Change https://golang.org/cl/367674 mentions this issue: internal/memoize: record the caller of Destroy

bcmills · 2021-11-29T20:10:34Z

I've spotted the Shutdown race and mailed a fix for review.

The hypothesis for golang/go#48774 is that the generation is being destroyed by a call to (*View).shutdown. This change adds a bit of logging to confirm that hypothesis. For golang/go#48774 Change-Id: I34be2e16a0dcab4cea7e9b704b56f4cf0abb0c71 Reviewed-on: https://go-review.googlesource.com/c/tools/+/367674 Trust: Bryan C. Mills <[email protected]> Run-TryBot: Bryan C. Mills <[email protected]> gopls-CI: kokoro <[email protected]> TryBot-Result: Go Bot <[email protected]> Reviewed-by: Robert Findley <[email protected]>

bcmills added NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. release-blocker labels Oct 4, 2021

bcmills added this to the Go1.18 milestone Oct 4, 2021

bcmills added gopls Issues related to the Go language server, gopls. Tools This label describes issues relating to any tools in the x/tools repository. labels Oct 4, 2021

findleyr self-assigned this Nov 3, 2021

bcmills added the okay-after-beta1 Used by release team to mark a release-blocker issue as okay to resolve either before or after beta1 label Nov 19, 2021

bcmills self-assigned this Nov 29, 2021

gopherbot closed this as completed in golang/tools@e212aff Dec 2, 2021

rsc unassigned bcmills and findleyr Jun 23, 2022

golang locked and limited conversation to collaborators Jun 23, 2023

gopherbot added the FrozenDueToAge label Jun 23, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

x/tools/gopls: frequent failures with "panic: operation on destroyed generation" #48774

x/tools/gopls: frequent failures with "panic: operation on destroyed generation" #48774

bcmills commented Oct 4, 2021

bcmills commented Oct 4, 2021

LeGEC commented Oct 7, 2021 •

edited

Loading

findleyr commented Nov 3, 2021

findleyr commented Nov 3, 2021

gopherbot commented Nov 5, 2021

findleyr commented Nov 10, 2021

toothrot commented Nov 19, 2021

findleyr commented Nov 19, 2021

gopherbot commented Nov 29, 2021

gopherbot commented Nov 29, 2021

bcmills commented Nov 29, 2021

x/tools/gopls: frequent failures with "panic: operation on destroyed generation" #48774

x/tools/gopls: frequent failures with "panic: operation on destroyed generation" #48774

Comments

bcmills commented Oct 4, 2021

bcmills commented Oct 4, 2021

LeGEC commented Oct 7, 2021 • edited Loading

findleyr commented Nov 3, 2021

findleyr commented Nov 3, 2021

gopherbot commented Nov 5, 2021

findleyr commented Nov 10, 2021

toothrot commented Nov 19, 2021

findleyr commented Nov 19, 2021

gopherbot commented Nov 29, 2021

gopherbot commented Nov 29, 2021

bcmills commented Nov 29, 2021

LeGEC commented Oct 7, 2021 •

edited

Loading