cmd/go: go list reports net as "stale" right after ./make.bash (but only inside GOROOT/src)

[dmitshur]

What version of Go are you using (go version)?

$ go version
go version go1.16 darwin/amd64

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/dmitshur/Library/Caches/go-build"
GOENV="/Users/dmitshur/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/dmitshur/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/dmitshur/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GOVCS=""
GOVERSION="go1.16"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/dev/null"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch x86_64 -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/zb/5p8cwfhj29gf_m8vdy8ylmlr00jwcj/T/go-build203663307=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

I tried building Go from source at Go 1.16 and 1.15.8, comparing whether the net package is reported as stale or not right after a ./make.bash.

What did you expect to see?

I expected that net is reported not stale after ./make.bash with both Go versions.

What did you see instead?

net is reported as not stale when using 1.15.8 both inside GOROOT/src and outside:

$ cd $(mktemp -d)  # Or equivalent that doesn't involve symlinks.
$ git clone https://go.googlesource.com/go && cd go
$ git checkout go1.15.8
$ cd src && ./make.bash && cd ..
$ export PATH="$(pwd)/bin:$PATH"
$ go list -f '{{.ImportPath}}: {{if .Stale}}stale; reason={{.StaleReason}}{{else}}not stale{{end}}' net
net: not stale
$ cd src
$ go list -f '{{.ImportPath}}: {{if .Stale}}stale; reason={{.StaleReason}}{{else}}not stale{{end}}' net
net: not stale

net is reported as not stale when using 1.16 outside GOROOT/src, but stale inside:

$ cd $(mktemp -d)  # Or equivalent that doesn't involve symlinks.
$ git clone https://go.googlesource.com/go && cd go
$ git checkout go1.16
$ cd src && ./make.bash && cd ..
$ export PATH="$(pwd)/bin:$PATH"
$ go list -f '{{.ImportPath}}: {{if .Stale}}stale; reason={{.StaleReason}}{{else}}not stale{{end}}' net
net: not stale
$ cd src
$ go list -f '{{.ImportPath}}: {{if .Stale}}stale; reason={{.StaleReason}}{{else}}not stale{{end}}' net
net: stale; reason=stale dependency: golang.org/x/net/dns/dnsmessage

I ran into this while investigating #44714 with @toothrot, and we both thought this was worth reporting so that it can be investigated as needed.

(While writing up this issue report, I realized it makes a difference whether the go list command is run in GOROOT/src or outside, which is not something I realized earlier.)

CC @jayconrod, @bcmills, @matloob.

[bcmills]

This is possibly a side-effect of the way the vendored std module dependencies work in module mode.

When you are inside GOROOT/src, the module dependencies of std have their actual, external-module identities. (That makes it possible to run commands like go mod tidy and go mod vendor.)

However, when you move outside of GOROOT/src, those vendored dependencies can no longer be ordinary module dependencies: their versions are fixed to what is shipped in std and the packages no longer retain their module identities. For example, the vendored copy of golang.org/x/net/dns/dnsmessage turns into the package vendor/golang.org/x/net/dns/dnsmessage.

When that happens, the package itself becomes stale, because its identity (and therefore its debug information, and perhaps its export metadata) has changed.

[bcmills]

This probably changed as of CL 251159, which removed special-case code for go get and go mod … within std.

That special-case forced the vendored std module dependencies to their vendor/… paths even when running the go command within GOROOT/src.

@jayconrod, @matloob: do you think we should restore that special-case (so that the dependencies of packages in std are identical whether inside or outside of the std module), or keep it as it is in 1.16 (so that the dependencies reported for those packages for general go commands match the paths used in go get and go mod tidy commands)?

[jayconrod]

I'd lean toward restoring the special case (I hate saying that). It's pretty confusing that these packages have multiple identities. It's taken me longer than I'd like to admit to re-read your comments and the CL to work out what's going on.

[bcmills]

Yeah, I agree. I have a CL in progress. 😁

[gopherbot]

Change https://golang.org/cl/297869 mentions this issue: cmd/go: resolve std-vendored dependencies as std packages except in 'go get' and 'go mod'