proposal: x/tools/go/packages: add no-network package lookup

[james-lawrence]

I have a couple projects where I want to resolve the location of a package if it exists, but not retrieve/verify the package.

My initial attempts to use go/build.Context.Import trigger a retrieval of the package if its not found. which is undesirable due to both security and performance concerns.

I have no problem with that behavior, but go build and go install do not require network access and I'd prefer to not reinvent the wheel to locate packages that golang uses itself.

go/importer looked promising, but it suffers from the same issue as it uses go/build under the hood and it does not return the same information.

the underlying cause seems to be where Import shells out to go list to resolve the paths, though I stopped following the code once I hit the cmd execution in go/build.

Not opposed to implementing the changes myself, just looking for ideas/buyin/alternatives.

possible implementation ideas:

1. add a Locate method to build.Context, same signature as Import but doesn't generate any network calls to resolve the package.
2. add a NoNetwork to build.ImportMode flags. (obviously happy to bikeshed names)

What version of Go are you using (go version)?

go version go1.14.4 linux/amd64

Does this issue reproduce with the latest release?

yes

What did you do?

# clear out cache/gopath
mv ~/go{,.bak}

package main

import (
	"go/build"
	"log"
)

func main() {
	b := build.Default
	p, err := b.Import("github.com/lib/pq", "", build.FindOnly)
	if err != nil {
		log.Fatal(err)
	}

	log.Println("package found:", p.Name, p.Dir)
}

What did you expect to see?

an error about package not found.

What did you see instead?

the package was downloaded and found.

[rsc]

golang.org/x/tools/go/packages is the right one to be using, not go/build.
That said, I don't think it handles this either.
But it probably could if we could figure out the go command flag to mean "don't use the network".

/cc @bcmills @matloob @jayconrod @ianthehat

[jayconrod]

I think this should be a feature of go list, for example, something like go list -f '{{.Dir}}' example.com/pkg could work in more cases. Both go/build and golang.org/x/tools/go/packages use go list to load package metadata, so if go list works, they would work.

GOPROXY is already used to control where modules are fetched from. GOPROXY=off is pretty close already. The main limitation is that you can't find the "latest" version of a module (or match any version query) when using GOPROXY=off. #32337 suggests letting go list -m -versions list cached versions when offline. That would get us most of the way there.

#26717 is also a request for a general mechanism for exploring the module cache.

[james-lawrence]

@rsc thanks for pointing me in the right direction.

@jayconrod go list is likely the right spot to implement a change. let me know if you need anything for this, including implementation work.

[rsc]

Sounds like we need to decide how to tell the go command to do this, and then circle back to x/tools/go/packages.

[rsc]

I talked to @jayconrod, @matloob, and @bcmills. The plan we suggest is to make GOPROXY=modcache mean use the module cache as the full proxy content, including for list requests, and then users would set that environment variable in the go/packages config.

[rsc]

I talked to @jayconrod, @matloob, and @bcmills again, and I also started a discussion on golang-tools about the right defaults for the go command.

At this point, it seems like a likely outcome could be that the go command stops doing network lookups during go list by default, which would then make this proposal unnecessary - x/tools/go/packages would stop doing network lookups too.

So we should probably wait until that gets resolved. Putting this proposal on hold.

[james-lawrence]

as a related note: part of the problem here is the lack of a well known API for interacting with dependency source code. (downloading, package directory locations/files) I had thought go/build was that but @rsc corrected me here. something to chew on for thought.

[james-lawrence]

@rsc bumping this =). its been quiet for awhile. and its unclear if goteam is considering this resolved with the go get / go install separation. I havent reviewed if those changes resolve my issues yet.

[bcmills]

@james-lawrence, did the change to default to -mod=readonly in Go 1.16 (#40728) resolve this?

[james-lawrence]

believe it might have. lets consider this wrapped up then until I can prove otherwise =) my main concern was the network access and that change looks like it should have resolved it.