go test does not work without CGO_ENABLED=1

[virtuald]

What version of Go are you using (go version)?

$ go version
go version go1.12.5 linux/amd64

Dockerfile I'm running in:

FROM ubuntu:16.04

RUN apt-get update && \
    apt-get install -y git python curl make && \
    curl https://dl.google.com/go/go1.12.5.linux-amd64.tar.gz | tar -xzC /usr/local && \
    apt-get clean

ENV PATH="/usr/local/go/bin:${PATH}"

Does this issue reproduce with the latest release?

Yes (1.12.5 is latest published version).

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/root/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build279503093=/tmp/go-build -gno-record-gcc-switches"

What did you do?

$ cd ~/go/path/to/program
$ go test ./pkg/...
# runtime/cgo
exec: "gcc": executable file not found in $PATH
FAIL path/to/program/pkg/subpkg [build failed]

Workaround

• Running CGO_ENABLED=0 go test ./pkg/... works without issues

Notes

Running these same tests on go 1.10 works without issues (and I only found it when I upgraded to go 1.12). There seem to be several go 1.11 bugs filed about issues related to this.

I'm not explicitly using go modules at this point, but I do use dep + a vendor directory.

There must be something complex/subtle that is causing this error, as it's really difficult to create a minimal reproducible testcase for this. Unfortunately, the source I'm working with isn't public, and I'm having a really difficult time creating a reproducible testcase for this -- despite that it occurs without fail with my entire source tree. When I try removing large portions of the tree, it starts working, but it doesn't seem to be particularly deterministic.

Running with -x contains mostly 'mkdir $WORK/random', but one place has the following snippet which might explain why it's running into an error:

...
cd /usr/local/go/src/runtime/cgo
CGO_LDFLAGS='"-g" "-O2" "-lpthread"' /usr/local/go/pkg/tool/linux_amd64/cgo -objdir $WORK/b067/ -importpath runtime/cgo -import_runtime_cgo=false -import_syscall=false -- -I $WORK/b067/ -g -O2 -Wall -Werror ./cgo.go
...

But I'm having difficulty determining why it has decided to use CGO (and not use it when it's disabled).

My hope is that even though I can't reproduce this error minimally, that someone else will find this issue and (a) be able to reproduce or at least (b) see my workaround and find it works for them.

[bcmills]

Duplicate of #28065