x/tools/go/ssa: CreateTestMainPackage broken by fuzzing changes

[bcmills]

~/x/tools$ gotip version
go version devel go1.18-051df0d72 Wed Sep 22 03:45:00 2021 +0000 linux/amd64

~/x/tools$ gotip test ./go/pointer
Entering directory `/usr/local/google/home/bcmills/x/tools/go/pointer'
Entering directory `/usr/local/google/home/bcmills/x/tools/go/pointer'
2021/09/22 11:06:56 internal error type-checking main$testmain: main$testmain.go:43:59: not enough arguments in call to testing.MainStart
FAIL    golang.org/x/tools/go/pointer   2.067s
FAIL

That failure comes from here:

		log.Fatalf("internal error type-checking %s: %v", path, err)

This function is a known technical debt:

// TODO(adonovan): throws this all away now that x/tools/go/packages
// provides access to the actual synthetic test main files.

CC @jayconrod @ianthehat

[gopherbot]

Change https://golang.org/cl/351509 mentions this issue: go/pointer: skip tests that depend on an unstable API changed in Go 1.18

[bcmills]

Note that CL 351509 doesn't really resolve the problem: we shouldn't leave the test permanently unskipped.

We can resolve this issue by one of:

• switching x/tools/go/ssa to use x/tools/go/packages instead of rolling its own test binary, or
• updating x/tools/go/ssa to use the new 1.18 signature when working with a 1.18 toolchain, or
• reverting the signature change (and perhaps finding some other way to pass the list of fuzz tests to the package).

(Or perhaps some other option I haven't thought of.)

[zpavlinovic]

Sorry for removing my original comment, I realized later that the fix just skips some tests.

Is CreateTestMainPackage deprecated? Is it possible to use a different API instead, would that be another possible solution?

[bcmills]

Is CreateTestMainPackage deprecated?

Yep! Per https://pkg.go.dev/golang.org/x/tools/go/ssa#Program.CreateTestMainPackage:
“Deprecated: Use golang.org/x/tools/go/packages to access synthetic testmain packages.”

Is it possible to use a different API instead, would that be another possible solution?

That is exactly the “switching x/tools/go/ssa to use x/tools/go/packages” option, I think. 🙂

[jayconrod]

switching x/tools/go/ssa to use x/tools/go/packages instead of rolling its own test binary, or

This is the best option I think. Depending on an unstable API makes it unnecessarily difficult to make changes like this.

[timothy-king]

+1 to migrating x/tools away from using this function.

If there are 0 other users in the wild, is there any objection to deleting CreateTestMainPackage or replacing its body with panic("deprecated")? This is clearly a semantic change for the module. Is there a good precedent for this in x/tools?

[bcmills]

I suspect that it would be fine to delete CreateTestMainPackage entirely if it no longer works. That is a breaking change, but OTOH we don't yet promise strict API stability in the x/ repos (that's part of why they exist!), and in this case the package doc already carries a prominent warning that the API “IS EXPERIMENTAL AND IS LIKELY TO CHANGE”.

[zpavlinovic]

More info. This API is also used in guru, godoc analysis, and another place in the pointer package.

[gopherbot]

Change https://golang.org/cl/357834 mentions this issue: godoc/analysis: replace use of deprecated ssa.CreateTestMainPackage

[gopherbot]

Change https://golang.org/cl/360837 mentions this issue: go/pointer: skip TestInput on 32-bit platforms

[bcmills]

@zpavlinovic, anything left to do for this issue, or are we calling it good enough?

[zpavlinovic]

I am still working on removing usage of this API from guru, which will hopefully allow us to delete this API altogether.

However, this particular issue might be interpreted as being concerned with ssa only. If that is the case, then we can close this issue and I can open another issue for tracking the progress of getting rid of this API.

[bcmills]

Either approach is ok by me.

[zpavlinovic]

Then I suggest keeping it simple and just use this issue to address the whole API. (I do not have an eta on when will I finish this, but hopefully soon.)

[gopherbot]

Change https://golang.org/cl/363655 mentions this issue: cmd/guru: remove use of deprecated ssa.CreateTestMainPackage

[gopherbot]

Change https://golang.org/cl/363657 mentions this issue: go/pointer: remove use of deprecated ssa.CreateTestMainPackage

[gopherbot]

Change https://golang.org/cl/363659 mentions this issue: go/ssa: remove deprecated FindTets and CreateTestMainPackage