Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

x/build/cmd/release: buildlets inexplicably got stuck #50890

Closed
mknyszek opened this issue Jan 28, 2022 · 5 comments
Closed

x/build/cmd/release: buildlets inexplicably got stuck #50890

mknyszek opened this issue Jan 28, 2022 · 5 comments
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone
Unreleased

Comments

@mknyszek
Copy link
Contributor

The below image basically summarizes it:

image

Running releasebot -mode=prepare goX.YbetaN resulted in some builders getting stuck like this. Nothing in the output appears to indicate a problem (except for a darwin/amd64 issue, filed separately):

~ $ releasebot -mode=prepare go1.18beta2
2022/01/28 12:29:24 Loading data from log *maintner.netMutSource ...
HTTP: 12:29:24.983 + https://maintner.golang.org/logs
HTTP: 12:29:25.322 - /logs 200 OK (0.339s)
2022/01/28 12:29:25 Downloading 2519 bytes of https://maintner.golang.org/logs/128 ...
HTTP: 12:29:25.349  + https://maintner.golang.org/logs/128
HTTP: 12:29:25.401  - /128 206 Partial Content (0.052s)
2022/01/28 12:29:25 wrote /Users/mknyszek/Library/Caches/golang-maintner/0128.growing.mutlog
2022/01/28 12:29:57 Reloaded data from log *maintner.netMutSource.
HTTP: 12:29:57.196   + https://oauth2.googleapis.com/token
HTTP: 12:29:57.352   - /token 200 OK (0.156s)
2022/01/28 12:29:57 starting
2022/01/28 12:29:57 working in /Users/mknyszek/go-releasebot-work/go1.18beta2
2022/01/28 12:29:57 $ git fetch origin master
2022/01/28 12:29:58 $ git fetch origin master
2022/01/28 12:30:09 $ git clone --reference /Users/mknyszek/go-releasebot-work/go1.18beta2/gitmirror -b master https://go.googlesource.com/go /Users/mknyszek/go-releasebot-work/go1.18beta2/gitwork
2022/01/28 12:30:13 $ git codereview change relwork
2022/01/28 12:30:14 $ git config gc.auto 0
2022/01/28 12:30:14 an associated security release branch "master-security" does not exist; assuming it has been merged and deleted, so proceeding as usual
2022/01/28 12:30:14 $ git rev-parse go1.18beta2
2022/01/28 12:30:14 Release status issue title: "all: Go 1.18beta2 release status"
HTTP: 12:30:14.925    + https://api.github.com/repos/golang/go/issues?per_page=100&state=all
HTTP: 12:30:16.042    - /issues 200 OK (1.117s)
HTTP: 12:30:16.154 + https://api.github.com/repos/golang/go/issues
HTTP: 12:30:16.571 - /issues 201 Created (0.418s)
2022/01/28 12:30:16 Release status issue: https://golang.org/issue/50886
2022/01/28 12:30:16 $ go clean -modcache
2022/01/28 12:30:16 $ go install golang.org/x/build/cmd/release@latest
2022/01/28 12:30:51 BUILDRELEASE go1.18beta2 linux-386
2022/01/28 12:30:51 BUILDRELEASE go1.18beta2 src
2022/01/28 12:30:51 $ /Users/mknyszek/go-releasebot-work/go1.18beta2/gopath/bin/release -target linux-386 -user mknyszek -version go1.18beta2 -staging_dir /Users/mknyszek/go-releasebot-work/go1.18beta2/release-staging/9ff00398489d9eb1822b3de028cd6ccf5674ebb3_1256542821 -rev 9ff00398489d9eb1822b3de028cd6ccf5674ebb3
2022/01/28 12:30:51 BUILDRELEASE go1.18beta2 linux-arm64
2022/01/28 12:30:51 BUILDRELEASE go1.18beta2 windows-amd64-longtest
2022/01/28 12:30:51 $ /Users/mknyszek/go-releasebot-work/go1.18beta2/gopath/bin/release -target windows-amd64-longtest -user mknyszek -version go1.18beta2 -staging_dir /Users/mknyszek/go-releasebot-work/go1.18beta2/release-staging/9ff00398489d9eb1822b3de028cd6ccf5674ebb3_1256542821 -rev 9ff00398489d9eb1822b3de028cd6ccf5674ebb3
2022/01/28 12:30:51 BUILDRELEASE go1.18beta2 linux-armv6l
2022/01/28 12:30:51 $ /Users/mknyszek/go-releasebot-work/go1.18beta2/gopath/bin/release -target linux-armv6l -user mknyszek -version go1.18beta2 -staging_dir /Users/mknyszek/go-releasebot-work/go1.18beta2/release-staging/9ff00398489d9eb1822b3de028cd6ccf5674ebb3_1256542821 -rev 9ff00398489d9eb1822b3de028cd6ccf5674ebb3
2022/01/28 12:30:51 $ /Users/mknyszek/go-releasebot-work/go1.18beta2/gopath/bin/release -target linux-arm64 -user mknyszek -version go1.18beta2 -staging_dir /Users/mknyszek/go-releasebot-work/go1.18beta2/release-staging/9ff00398489d9eb1822b3de028cd6ccf5674ebb3_1256542821 -rev 9ff00398489d9eb1822b3de028cd6ccf5674ebb3
2022/01/28 12:30:51 $ /Users/mknyszek/go-releasebot-work/go1.18beta2/gopath/bin/release -target src -user mknyszek -version go1.18beta2 -staging_dir /Users/mknyszek/go-releasebot-work/go1.18beta2/release-staging/9ff00398489d9eb1822b3de028cd6ccf5674ebb3_1256542821 -rev 9ff00398489d9eb1822b3de028cd6ccf5674ebb3
2022/01/28 12:30:51 BUILDRELEASE go1.18beta2 linux-amd64
2022/01/28 12:30:51 $ /Users/mknyszek/go-releasebot-work/go1.18beta2/gopath/bin/release -target linux-amd64 -user mknyszek -version go1.18beta2 -staging_dir /Users/mknyszek/go-releasebot-work/go1.18beta2/release-staging/9ff00398489d9eb1822b3de028cd6ccf5674ebb3_1256542821 -rev 9ff00398489d9eb1822b3de028cd6ccf5674ebb3
2022/01/28 12:30:51 BUILDRELEASE go1.18beta2 darwin-amd64
2022/01/28 12:30:51 BUILDRELEASE go1.18beta2 windows-386
2022/01/28 12:30:51 $ /Users/mknyszek/go-releasebot-work/go1.18beta2/gopath/bin/release -target darwin-amd64 -user mknyszek -version go1.18beta2 -staging_dir /Users/mknyszek/go-releasebot-work/go1.18beta2/release-staging/9ff00398489d9eb1822b3de028cd6ccf5674ebb3_1256542821 -rev 9ff00398489d9eb1822b3de028cd6ccf5674ebb3
2022/01/28 12:30:51 $ /Users/mknyszek/go-releasebot-work/go1.18beta2/gopath/bin/release -target windows-386 -user mknyszek -version go1.18beta2 -staging_dir /Users/mknyszek/go-releasebot-work/go1.18beta2/release-staging/9ff00398489d9eb1822b3de028cd6ccf5674ebb3_1256542821 -rev 9ff00398489d9eb1822b3de028cd6ccf5674ebb3
2022/01/28 12:30:51 BUILDRELEASE go1.18beta2 freebsd-386
2022/01/28 12:30:51 $ /Users/mknyszek/go-releasebot-work/go1.18beta2/gopath/bin/release -target freebsd-386 -user mknyszek -version go1.18beta2 -staging_dir /Users/mknyszek/go-releasebot-work/go1.18beta2/release-staging/9ff00398489d9eb1822b3de028cd6ccf5674ebb3_1256542821 -rev 9ff00398489d9eb1822b3de028cd6ccf5674ebb3
2022/01/28 12:30:51 BUILDRELEASE go1.18beta2 windows-amd64
2022/01/28 12:30:51 $ /Users/mknyszek/go-releasebot-work/go1.18beta2/gopath/bin/release -target windows-amd64 -user mknyszek -version go1.18beta2 -staging_dir /Users/mknyszek/go-releasebot-work/go1.18beta2/release-staging/9ff00398489d9eb1822b3de028cd6ccf5674ebb3_1256542821 -rev 9ff00398489d9eb1822b3de028cd6ccf5674ebb3
2022/01/28 12:30:51 BUILDRELEASE go1.18beta2 windows-arm64
2022/01/28 12:30:51 $ /Users/mknyszek/go-releasebot-work/go1.18beta2/gopath/bin/release -target windows-arm64 -user mknyszek -version go1.18beta2 -staging_dir /Users/mknyszek/go-releasebot-work/go1.18beta2/release-staging/9ff00398489d9eb1822b3de028cd6ccf5674ebb3_1256542821 -rev 9ff00398489d9eb1822b3de028cd6ccf5674ebb3
2022/01/28 12:30:51 BUILDRELEASE go1.18beta2 darwin-arm64
2022/01/28 12:30:51 $ /Users/mknyszek/go-releasebot-work/go1.18beta2/gopath/bin/release -target darwin-arm64 -user mknyszek -version go1.18beta2 -staging_dir /Users/mknyszek/go-releasebot-work/go1.18beta2/release-staging/9ff00398489d9eb1822b3de028cd6ccf5674ebb3_1256542821 -rev 9ff00398489d9eb1822b3de028cd6ccf5674ebb3
2022/01/28 12:30:51 BUILDRELEASE go1.18beta2 linux-s390x
2022/01/28 12:30:51 $ /Users/mknyszek/go-releasebot-work/go1.18beta2/gopath/bin/release -target linux-s390x -user mknyszek -version go1.18beta2 -staging_dir /Users/mknyszek/go-releasebot-work/go1.18beta2/release-staging/9ff00398489d9eb1822b3de028cd6ccf5674ebb3_1256542821 -rev 9ff00398489d9eb1822b3de028cd6ccf5674ebb3
2022/01/28 12:30:51 BUILDRELEASE go1.18beta2 linux-ppc64le
2022/01/28 12:30:51 $ /Users/mknyszek/go-releasebot-work/go1.18beta2/gopath/bin/release -target linux-ppc64le -user mknyszek -version go1.18beta2 -staging_dir /Users/mknyszek/go-releasebot-work/go1.18beta2/release-staging/9ff00398489d9eb1822b3de028cd6ccf5674ebb3_1256542821 -rev 9ff00398489d9eb1822b3de028cd6ccf5674ebb3
2022/01/28 12:30:51 BUILDRELEASE go1.18beta2 linux-amd64-longtest
2022/01/28 12:30:51 BUILDRELEASE go1.18beta2 freebsd-amd64
2022/01/28 12:30:51 $ /Users/mknyszek/go-releasebot-work/go1.18beta2/gopath/bin/release -target freebsd-amd64 -user mknyszek -version go1.18beta2 -staging_dir /Users/mknyszek/go-releasebot-work/go1.18beta2/release-staging/9ff00398489d9eb1822b3de028cd6ccf5674ebb3_1256542821 -rev 9ff00398489d9eb1822b3de028cd6ccf5674ebb3
2022/01/28 12:30:52 $ /Users/mknyszek/go-releasebot-work/go1.18beta2/gopath/bin/release -target linux-amd64-longtest -user mknyszek -version go1.18beta2 -staging_dir /Users/mknyszek/go-releasebot-work/go1.18beta2/release-staging/9ff00398489d9eb1822b3de028cd6ccf5674ebb3_1256542821 -rev 9ff00398489d9eb1822b3de028cd6ccf5674ebb3
2022/01/28 12:30:52 BUILDRELEASE go1.18beta2 linux-386-longtest
2022/01/28 12:30:52 $ /Users/mknyszek/go-releasebot-work/go1.18beta2/gopath/bin/release -target linux-386-longtest -user mknyszek -version go1.18beta2 -staging_dir /Users/mknyszek/go-releasebot-work/go1.18beta2/release-staging/9ff00398489d9eb1822b3de028cd6ccf5674ebb3_1256542821 -rev 9ff00398489d9eb1822b3de028cd6ccf5674ebb3
2022/01/28 12:31:48 release -target="src": build succeeded (after 0 retries)
2022/01/28 12:31:48 DONE BUILDRELEASE go1.18beta2 src
2022/01/28 12:35:34 release -target="linux-ppc64le": build succeeded (after 0 retries)
2022/01/28 12:35:34 DONE BUILDRELEASE go1.18beta2 linux-ppc64le
2022/01/28 12:36:17 release -target="linux-s390x": build succeeded (after 0 retries)
2022/01/28 12:36:17 DONE BUILDRELEASE go1.18beta2 linux-s390x
2022/01/28 12:40:25 release -target="darwin-amd64" did not produce expected output files [go1.18beta2.darwin-amd64.tar.gz]:
error from cmd/release binary = <nil>
output from cmd/release binary:
2022/01/28 12:30:53 darwin-amd64: Start.
2022/01/28 12:30:53 darwin-amd64: Creating buildlet.
2022/01/28 12:30:55 darwin-amd64: Pushing source to buildlet.
2022/01/28 12:31:03 darwin-amd64: Writing VERSION file.
2022/01/28 12:31:03 darwin-amd64: Cleaning goroot (pre-build).
2022/01/28 12:31:03 darwin-amd64: Building (make.bash only).
2022/01/28 12:33:02 darwin-amd64: Building race detector.
2022/01/28 12:33:21 darwin-amd64: Cleaning goroot (post-build).
2022/01/28 12:33:21 darwin-amd64: Pushing and running releaselet.
2022/01/28 12:33:22 darwin-amd64: Cleaning workdir.
2022/01/28 12:33:24 darwin-amd64: Downloading tarball.
2022/01/28 12:33:45 darwin-amd64: Wrote "/Users/mknyszek/go-releasebot-work/go1.18beta2/release-staging/9ff00398489d9eb1822b3de028cd6ccf5674ebb3_1256542821/go1.18beta2.darwin-amd64.tar.gz.untested".
2022/01/28 12:33:45 darwin-amd64: Building (all.bash to ensure tests pass).
2022/01/28 12:40:25 darwin-amd64: Error: Build failed: exit status 1
Output:
Building Go cmd/dist using /Users/gopher/goboot. (go1.17.3 darwin/amd64)
Building Go toolchain1 using /Users/gopher/goboot.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
Building Go toolchain2 using go_bootstrap and Go toolchain1.
Building Go toolchain3 using go_bootstrap and Go toolchain2.
Building packages and commands for darwin/amd64.

##### Testing packages.
ok  	archive/tar	0.300s
ok  	archive/zip	0.743s
ok  	bufio	0.303s
ok  	bytes	3.697s
ok  	compress/bzip2	0.335s
ok  	compress/flate	1.637s
ok  	compress/gzip	0.545s
ok  	compress/lzw	0.492s
ok  	compress/zlib	1.328s
ok  	constraints	0.298s
ok  	container/heap	0.173s
ok  	container/list	0.385s
ok  	container/ring	0.137s
ok  	context	0.184s
ok  	crypto	0.261s
ok  	crypto/aes	0.196s
ok  	crypto/cipher	0.340s
ok  	crypto/des	0.196s
ok  	crypto/dsa	0.482s
ok  	crypto/ecdsa	0.218s
ok  	crypto/ed25519	0.274s
ok  	crypto/ed25519/internal/edwards25519	8.250s
ok  	crypto/ed25519/internal/edwards25519/field	2.614s
ok  	crypto/elliptic	0.237s
ok  	crypto/elliptic/internal/fiat	0.181s [no tests to run]
ok  	crypto/elliptic/internal/nistec	1.356s
ok  	crypto/hmac	0.262s
ok  	crypto/internal/subtle	0.186s
ok  	crypto/md5	0.211s
ok  	crypto/rand	0.272s
ok  	crypto/rc4	0.241s
ok  	crypto/rsa	0.325s
ok  	crypto/sha1	0.246s
ok  	crypto/sha256	0.246s
ok  	crypto/sha512	0.185s
ok  	crypto/subtle	0.205s
ok  	crypto/tls	1.807s
ok  	crypto/x509	0.932s
ok  	database/sql	0.844s
ok  	database/sql/driver	0.195s
ok  	debug/buildinfo	0.160s
ok  	debug/dwarf	0.208s
ok  	debug/elf	0.192s
ok  	debug/gosym	0.194s
ok  	debug/macho	0.317s
ok  	debug/pe	0.383s
ok  	debug/plan9obj	0.568s
ok  	embed	0.291s [no tests to run]
ok  	embed/internal/embedtest	0.192s
ok  	encoding/ascii85	0.428s
ok  	encoding/asn1	0.250s
ok  	encoding/base32	0.367s
ok  	encoding/base64	0.394s
ok  	encoding/binary	0.170s
ok  	encoding/csv	0.179s
ok  	encoding/gob	0.272s
ok  	encoding/hex	0.235s
ok  	encoding/json	0.785s
ok  	encoding/pem	0.286s
ok  	encoding/xml	0.465s
ok  	errors	0.262s
ok  	expvar	0.304s
ok  	flag	0.266s
ok  	fmt	0.380s
ok  	go/ast	0.187s
ok  	go/build	3.269s
ok  	go/build/constraint	0.158s
ok  	go/constant	0.430s
ok  	go/doc	0.439s
ok  	go/format	0.348s
ok  	go/importer	0.350s
ok  	go/internal/gccgoimporter	0.185s
ok  	go/internal/gcimporter	11.260s
ok  	go/internal/srcimporter	15.459s
ok  	go/parser	0.290s
ok  	go/printer	0.408s
ok  	go/scanner	0.218s
ok  	go/token	0.182s
ok  	go/types	24.239s
ok  	hash	0.170s
ok  	hash/adler32	0.163s
ok  	hash/crc32	0.218s
ok  	hash/crc64	0.226s
ok  	hash/fnv	0.324s
ok  	hash/maphash	0.215s
ok  	html	0.208s
ok  	html/template	0.315s
ok  	image	0.224s
ok  	image/color	0.176s
ok  	image/draw	0.216s
ok  	image/gif	21.135s
ok  	image/jpeg	3.667s
ok  	image/png	0.336s
ok  	index/suffixarray	0.277s
ok  	internal/abi	0.233s
ok  	internal/buildcfg	0.148s
ok  	internal/cpu	0.219s
ok  	internal/execabs	0.148s
ok  	internal/fmtsort	0.191s
ok  	internal/fuzz	0.183s
ok  	internal/godebug	0.253s
ok  	internal/intern	0.672s
ok  	internal/itoa	0.207s
ok  	internal/poll	0.415s
ok  	internal/profile	0.144s
ok  	internal/reflectlite	0.412s
ok  	internal/singleflight	0.188s
ok  	internal/trace	0.216s
ok  	internal/unsafeheader	0.187s
ok  	internal/xcoff	0.270s
ok  	io	0.209s
ok  	io/fs	0.191s
ok  	io/ioutil	0.197s
ok  	log	0.199s
ok  	log/syslog	1.529s
ok  	math	0.182s
ok  	math/big	2.180s
ok  	math/bits	0.406s
ok  	math/cmplx	0.287s
ok  	math/rand	0.453s
ok  	mime	0.166s
ok  	mime/multipart	0.411s
ok  	mime/quotedprintable	0.148s
ok  	net	13.619s
ok  	net/http	9.803s
ok  	net/http/cgi	1.356s
ok  	net/http/cookiejar	0.229s
ok  	net/http/fcgi	0.495s
ok  	net/http/httptest	0.251s
ok  	net/http/httptrace	0.185s
ok  	net/http/httputil	0.828s
ok  	net/http/internal	0.158s
ok  	net/http/internal/ascii	0.156s
ok  	net/http/pprof	5.285s
ok  	net/internal/socktest	0.328s
ok  	net/mail	0.167s
ok  	net/netip	0.855s
ok  	net/rpc	0.220s
ok  	net/rpc/jsonrpc	0.171s
ok  	net/smtp	0.171s
ok  	net/textproto	0.160s
ok  	net/url	0.187s
ok  	os	2.097s
ok  	os/exec	0.959s
ok  	os/exec/internal/fdtest	0.218s
ok  	os/signal	2.536s
ok  	os/user	0.157s
ok  	path	0.184s
ok  	path/filepath	0.261s
ok  	plugin	0.341s
ok  	reflect	0.912s
ok  	regexp	0.323s
ok  	regexp/syntax	0.717s
ok  	runtime	47.848s
ok  	runtime/cgo	0.256s
ok  	runtime/debug	0.254s
ok  	runtime/internal/atomic	0.481s
ok  	runtime/internal/math	0.195s
ok  	runtime/internal/sys	0.507s
ok  	runtime/metrics	0.555s
ok  	runtime/pprof	11.277s
ok  	runtime/trace	1.926s
ok  	sort	0.260s
ok  	strconv	0.991s
ok  	strings	3.548s
ok  	sync	0.509s
ok  	sync/atomic	5.571s
ok  	syscall	1.570s
ok  	testing	1.638s
ok  	testing/fstest	0.154s
ok  	testing/iotest	0.174s
ok  	testing/quick	0.272s
ok  	text/scanner	0.160s
ok  	text/tabwriter	0.332s
ok  	text/template	0.406s
ok  	text/template/parse	0.171s
ok  	time	2.074s
ok  	unicode	0.197s
ok  	unicode/utf16	0.198s
ok  	unicode/utf8	0.186s
ok  	cmd/addr2line	2.633s
ok  	cmd/api	24.757s
ok  	cmd/asm/internal/asm	1.264s
ok  	cmd/asm/internal/lex	0.268s
ok  	cmd/compile/internal/amd64	11.869s
ok  	cmd/compile/internal/dwarfgen	0.838s
ok  	cmd/compile/internal/importer	6.028s
ok  	cmd/compile/internal/ir	0.195s
ok  	cmd/compile/internal/logopt	0.802s
ok  	cmd/compile/internal/noder	0.283s
ok  	cmd/compile/internal/ssa	47.879s
ok  	cmd/compile/internal/syntax	0.181s
ok  	cmd/compile/internal/test	42.507s
ok  	cmd/compile/internal/typecheck	1.149s
ok  	cmd/compile/internal/types	0.151s
ok  	cmd/compile/internal/types2	10.706s
ok  	cmd/cover	4.252s
ok  	cmd/dist	0.209s
ok  	cmd/doc	0.737s
ok  	cmd/fix	6.169s
go test proxy running at GOPROXY=http://127.0.0.1:51589/mod
--- FAIL: TestScript (0.03s)
    --- FAIL: TestScript/cgo_stale_precompiled (0.38s)
        script_test.go:257:
            # Regression test for https://go.dev/issue/47215 and https://go.dev/issue/50183:
            # A mismatched $GOROOT_FINAL or missing $CC caused the C dependencies of the net
            # package to appear stale, and it could not be rebuilt due to a missing $CC. (0.000s)
            # Control case: net must not already be stale. (0.374s)
            > ! stale net
            FAIL: testdata/script/cgo_stale_precompiled.txt:8: net is unexpectedly stale


FAIL
FAIL	cmd/go	124.028s
ok  	cmd/go/internal/auth	0.182s
ok  	cmd/go/internal/cache	0.849s
ok  	cmd/go/internal/fsys	0.321s
ok  	cmd/go/internal/generate	0.296s
ok  	cmd/go/internal/get	0.487s
ok  	cmd/go/internal/imports	0.466s
ok  	cmd/go/internal/load	0.224s
ok  	cmd/go/internal/lockedfile	0.257s
ok  	cmd/go/internal/lockedfile/internal/filelock	0.208s
ok  	cmd/go/internal/modconv	0.463s
ok  	cmd/go/internal/modfetch	0.477s
ok  	cmd/go/internal/modfetch/codehost	0.457s
ok  	cmd/go/internal/modfetch/zip_sum_test	0.741s
ok  	cmd/go/internal/modload	0.405s
ok  	cmd/go/internal/mvs	0.200s
ok  	cmd/go/internal/par	0.303s
ok  	cmd/go/internal/search	0.180s
ok  	cmd/go/internal/str	0.252s
ok  	cmd/go/internal/test	0.408s
ok  	cmd/go/internal/vcs	0.297s
ok  	cmd/go/internal/web	0.607s
ok  	cmd/go/internal/work	0.396s
ok  	cmd/gofmt	0.279s
ok  	cmd/internal/archive	2.720s
ok  	cmd/internal/buildid	0.828s
ok  	cmd/internal/dwarf	0.273s
ok  	cmd/internal/edit	0.193s
ok  	cmd/internal/goobj	0.209s
ok  	cmd/internal/moddeps	8.080s
ok  	cmd/internal/obj	0.207s
ok  	cmd/internal/obj/ppc64	0.476s
ok  	cmd/internal/obj/riscv	0.240s
ok  	cmd/internal/obj/s390x	0.292s
ok  	cmd/internal/obj/x86	14.339s
ok  	cmd/internal/objabi	0.182s
ok  	cmd/internal/pkgpath	0.318s
ok  	cmd/internal/quoted	0.461s
ok  	cmd/internal/src	0.446s
ok  	cmd/internal/sys	0.337s
ok  	cmd/internal/test2json	0.463s
ok  	cmd/link	60.705s
ok  	cmd/link/internal/benchmark	0.655s
ok  	cmd/link/internal/ld	34.839s
ok  	cmd/link/internal/loader	0.248s
ok  	cmd/nm	9.810s
ok  	cmd/objdump	12.877s
ok  	cmd/pack	9.122s
ok  	cmd/pprof	9.377s
ok  	cmd/trace	0.463s
ok  	cmd/vet	19.342s
FAIL
go tool dist: Failed: exit status 1
2022/01/28 12:40:25 release -target="darwin-amd64": waiting a bit and trying again
2022/01/28 12:41:15 release -target="darwin-arm64": build succeeded (after 0 retries)
2022/01/28 12:41:15 DONE BUILDRELEASE go1.18beta2 darwin-arm64
2022/01/28 12:41:25 $ /Users/mknyszek/go-releasebot-work/go1.18beta2/gopath/bin/release -target darwin-amd64 -user mknyszek -version go1.18beta2 -staging_dir /Users/mknyszek/go-releasebot-work/go1.18beta2/release-staging/9ff00398489d9eb1822b3de028cd6ccf5674ebb3_1256542821 -rev 9ff00398489d9eb1822b3de028cd6ccf5674ebb3
2022/01/28 12:43:00 release -target="linux-arm64": build succeeded (after 0 retries)
2022/01/28 12:43:00 DONE BUILDRELEASE go1.18beta2 linux-arm64
2022/01/28 12:45:12 release -target="linux-armv6l": build succeeded (after 0 retries)
2022/01/28 12:45:12 DONE BUILDRELEASE go1.18beta2 linux-armv6l
2022/01/28 12:46:20 release -target="freebsd-amd64": build succeeded (after 0 retries)
2022/01/28 12:46:20 DONE BUILDRELEASE go1.18beta2 freebsd-amd64
2022/01/28 12:49:11 release -target="windows-arm64": build succeeded (after 0 retries)
2022/01/28 12:49:11 DONE BUILDRELEASE go1.18beta2 windows-arm64
2022/01/28 12:50:40 release -target="darwin-amd64" did not produce expected output files [go1.18beta2.darwin-amd64.tar.gz]:
error from cmd/release binary = <nil>
output from cmd/release binary:
2022/01/28 12:41:25 darwin-amd64: Start.
2022/01/28 12:41:25 darwin-amd64: Creating buildlet.
2022/01/28 12:41:26 darwin-amd64: Pushing source to buildlet.
2022/01/28 12:41:32 darwin-amd64: Writing VERSION file.
2022/01/28 12:41:32 darwin-amd64: Cleaning goroot (pre-build).
2022/01/28 12:41:32 darwin-amd64: Building (make.bash only).
2022/01/28 12:43:28 darwin-amd64: Building race detector.
2022/01/28 12:43:47 darwin-amd64: Cleaning goroot (post-build).
2022/01/28 12:43:47 darwin-amd64: Pushing and running releaselet.
2022/01/28 12:43:48 darwin-amd64: Cleaning workdir.
2022/01/28 12:43:50 darwin-amd64: Downloading tarball.
2022/01/28 12:44:22 darwin-amd64: Wrote "/Users/mknyszek/go-releasebot-work/go1.18beta2/release-staging/9ff00398489d9eb1822b3de028cd6ccf5674ebb3_1256542821/go1.18beta2.darwin-amd64.tar.gz.untested".
2022/01/28 12:44:22 darwin-amd64: Building (all.bash to ensure tests pass).
2022/01/28 12:50:40 darwin-amd64: Error: Build failed: exit status 1
Output:
Building Go cmd/dist using /Users/gopher/goboot. (go1.17.3 darwin/amd64)
Building Go toolchain1 using /Users/gopher/goboot.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
Building Go toolchain2 using go_bootstrap and Go toolchain1.
Building Go toolchain3 using go_bootstrap and Go toolchain2.
Building packages and commands for darwin/amd64.

##### Testing packages.
ok  	archive/tar	0.439s
ok  	archive/zip	0.764s
ok  	bufio	0.233s
ok  	bytes	3.091s
ok  	compress/bzip2	0.340s
ok  	compress/flate	1.985s
ok  	compress/gzip	0.616s
ok  	compress/lzw	0.241s
ok  	compress/zlib	1.280s
ok  	constraints	0.561s
ok  	container/heap	0.180s
ok  	container/list	0.180s
ok  	container/ring	0.339s
ok  	context	0.180s
ok  	crypto	0.431s
ok  	crypto/aes	0.184s
ok  	crypto/cipher	0.474s
ok  	crypto/des	0.189s
ok  	crypto/dsa	0.210s
ok  	crypto/ecdsa	0.240s
ok  	crypto/ed25519	0.213s
ok  	crypto/ed25519/internal/edwards25519	8.097s
ok  	crypto/ed25519/internal/edwards25519/field	2.873s
ok  	crypto/elliptic	0.166s
ok  	crypto/elliptic/internal/fiat	0.213s [no tests to run]
ok  	crypto/elliptic/internal/nistec	1.343s
ok  	crypto/hmac	0.352s
ok  	crypto/internal/subtle	0.308s
ok  	crypto/md5	0.216s
ok  	crypto/rand	0.295s
ok  	crypto/rc4	0.341s
ok  	crypto/rsa	0.413s
ok  	crypto/sha1	0.243s
ok  	crypto/sha256	0.175s
ok  	crypto/sha512	0.338s
ok  	crypto/subtle	0.181s
ok  	crypto/tls	1.756s
ok  	crypto/x509	0.936s
ok  	database/sql	0.809s
ok  	database/sql/driver	0.178s
ok  	debug/buildinfo	0.197s
ok  	debug/dwarf	0.180s
ok  	debug/elf	0.172s
ok  	debug/gosym	0.321s
ok  	debug/macho	0.233s
ok  	debug/pe	0.335s
ok  	debug/plan9obj	0.448s
ok  	embed	0.423s [no tests to run]
ok  	embed/internal/embedtest	0.225s
ok  	encoding/ascii85	0.236s
ok  	encoding/asn1	0.267s
ok  	encoding/base32	0.490s
ok  	encoding/base64	0.447s
ok  	encoding/binary	0.167s
ok  	encoding/csv	0.160s
ok  	encoding/gob	0.245s
ok  	encoding/hex	0.303s
ok  	encoding/json	0.723s
ok  	encoding/pem	0.208s
ok  	encoding/xml	0.309s
ok  	errors	0.147s
ok  	expvar	0.289s
ok  	flag	0.263s
ok  	fmt	0.340s
ok  	go/ast	0.263s
ok  	go/build	3.170s
ok  	go/build/constraint	0.189s
ok  	go/constant	0.392s
ok  	go/doc	0.460s
ok  	go/format	0.388s
ok  	go/importer	0.477s
ok  	go/internal/gccgoimporter	0.201s
ok  	go/internal/gcimporter	10.991s
ok  	go/internal/srcimporter	14.778s
ok  	go/parser	0.258s
ok  	go/printer	0.359s
ok  	go/scanner	0.168s
ok  	go/token	0.237s
ok  	go/types	23.112s
ok  	hash	0.145s
ok  	hash/adler32	0.247s
ok  	hash/crc32	0.220s
ok  	hash/crc64	0.183s
ok  	hash/fnv	0.171s
ok  	hash/maphash	0.231s
ok  	html	0.185s
ok  	html/template	0.257s
ok  	image	0.244s
ok  	image/color	0.225s
ok  	image/draw	0.250s
ok  	image/gif	21.088s
ok  	image/jpeg	3.566s
ok  	image/png	0.322s
ok  	index/suffixarray	0.280s
ok  	internal/abi	0.204s
ok  	internal/buildcfg	0.150s
ok  	internal/cpu	0.155s
ok  	internal/execabs	0.159s
ok  	internal/fmtsort	0.139s
ok  	internal/fuzz	0.172s
ok  	internal/godebug	0.151s
ok  	internal/intern	0.978s
ok  	internal/itoa	0.210s
ok  	internal/poll	0.289s
ok  	internal/profile	0.168s
ok  	internal/reflectlite	0.435s
ok  	internal/singleflight	0.181s
ok  	internal/trace	0.281s
ok  	internal/unsafeheader	0.162s
ok  	internal/xcoff	0.199s
ok  	io	0.170s
ok  	io/fs	0.156s
ok  	io/ioutil	0.253s
ok  	log	0.176s
ok  	log/syslog	1.397s
ok  	math	0.165s
ok  	math/big	2.063s
ok  	math/bits	0.183s
ok  	math/cmplx	0.201s
ok  	math/rand	0.444s
ok  	mime	0.175s
ok  	mime/multipart	0.415s
ok  	mime/quotedprintable	0.308s
ok  	net	11.038s
ok  	net/http	8.477s
ok  	net/http/cgi	1.274s
ok  	net/http/cookiejar	0.236s
ok  	net/http/fcgi	0.279s
ok  	net/http/httptest	0.332s
ok  	net/http/httptrace	0.293s
ok  	net/http/httputil	0.843s
ok  	net/http/internal	0.153s
ok  	net/http/internal/ascii	0.169s
ok  	net/http/pprof	5.306s
ok  	net/internal/socktest	0.167s
ok  	net/mail	0.162s
ok  	net/netip	0.913s
ok  	net/rpc	0.243s
ok  	net/rpc/jsonrpc	0.193s
ok  	net/smtp	0.277s
ok  	net/textproto	0.188s
ok  	net/url	0.191s
ok  	os	1.618s
ok  	os/exec	1.055s
ok  	os/exec/internal/fdtest	0.205s
ok  	os/signal	2.316s
ok  	os/user	0.151s
ok  	path	0.175s
ok  	path/filepath	0.244s
ok  	plugin	0.193s
ok  	reflect	0.597s
ok  	regexp	0.221s
ok  	regexp/syntax	0.745s
ok  	runtime	40.162s
ok  	runtime/cgo	0.312s
ok  	runtime/debug	0.236s
ok  	runtime/internal/atomic	0.269s
ok  	runtime/internal/math	0.287s
ok  	runtime/internal/sys	0.302s
ok  	runtime/metrics	0.168s
ok  	runtime/pprof	8.854s
ok  	runtime/trace	0.999s
ok  	sort	0.188s
ok  	strconv	0.767s
ok  	strings	2.575s
ok  	sync	0.463s
ok  	sync/atomic	3.157s
ok  	syscall	0.822s
ok  	testing	0.924s
ok  	testing/fstest	0.154s
ok  	testing/iotest	0.152s
ok  	testing/quick	0.177s
ok  	text/scanner	0.240s
ok  	text/tabwriter	0.180s
ok  	text/template	0.258s
ok  	text/template/parse	0.152s
ok  	time	2.054s
ok  	unicode	0.227s
ok  	unicode/utf16	0.154s
ok  	unicode/utf8	0.165s
ok  	cmd/addr2line	1.738s
ok  	cmd/api	22.207s
ok  	cmd/asm/internal/asm	0.946s
ok  	cmd/asm/internal/lex	0.145s
ok  	cmd/compile/internal/amd64	11.589s
ok  	cmd/compile/internal/dwarfgen	0.737s
ok  	cmd/compile/internal/importer	5.659s
ok  	cmd/compile/internal/ir	0.207s
ok  	cmd/compile/internal/logopt	0.595s
ok  	cmd/compile/internal/noder	0.210s
ok  	cmd/compile/internal/ssa	41.322s
ok  	cmd/compile/internal/syntax	0.214s
ok  	cmd/compile/internal/test	36.106s
ok  	cmd/compile/internal/typecheck	0.845s
ok  	cmd/compile/internal/types	0.158s
ok  	cmd/compile/internal/types2	8.383s
ok  	cmd/cover	4.041s
ok  	cmd/dist	0.210s
ok  	cmd/doc	0.559s
ok  	cmd/fix	5.303s
go test proxy running at GOPROXY=http://127.0.0.1:51540/mod
--- FAIL: TestScript (0.02s)
    --- FAIL: TestScript/cgo_stale_precompiled (0.66s)
        script_test.go:257:
            # Regression test for https://go.dev/issue/47215 and https://go.dev/issue/50183:
            # A mismatched $GOROOT_FINAL or missing $CC caused the C dependencies of the net
            # package to appear stale, and it could not be rebuilt due to a missing $CC. (0.000s)
            # Control case: net must not already be stale. (0.654s)
            > ! stale net
            FAIL: testdata/script/cgo_stale_precompiled.txt:8: net is unexpectedly stale


FAIL
FAIL	cmd/go	117.938s
ok  	cmd/go/internal/auth	0.219s
ok  	cmd/go/internal/cache	0.785s
ok  	cmd/go/internal/fsys	0.396s
ok  	cmd/go/internal/generate	0.301s
ok  	cmd/go/internal/get	0.386s
ok  	cmd/go/internal/imports	0.191s
ok  	cmd/go/internal/load	0.440s
ok  	cmd/go/internal/lockedfile	0.264s
ok  	cmd/go/internal/lockedfile/internal/filelock	0.199s
ok  	cmd/go/internal/modconv	0.253s
ok  	cmd/go/internal/modfetch	0.358s
ok  	cmd/go/internal/modfetch/codehost	0.318s
ok  	cmd/go/internal/modfetch/zip_sum_test	0.602s
ok  	cmd/go/internal/modload	0.279s
ok  	cmd/go/internal/mvs	0.179s
ok  	cmd/go/internal/par	0.264s
ok  	cmd/go/internal/search	0.185s
ok  	cmd/go/internal/str	0.209s
ok  	cmd/go/internal/test	0.338s
ok  	cmd/go/internal/vcs	0.255s
ok  	cmd/go/internal/web	0.327s
ok  	cmd/go/internal/work	0.369s
ok  	cmd/gofmt	0.387s
ok  	cmd/internal/archive	2.357s
ok  	cmd/internal/buildid	0.714s
ok  	cmd/internal/dwarf	0.329s
ok  	cmd/internal/edit	0.171s
ok  	cmd/internal/goobj	0.157s
ok  	cmd/internal/moddeps	7.709s
ok  	cmd/internal/obj	0.175s
ok  	cmd/internal/obj/ppc64	0.412s
ok  	cmd/internal/obj/riscv	0.243s
ok  	cmd/internal/obj/s390x	0.377s
ok  	cmd/internal/obj/x86	13.578s
ok  	cmd/internal/objabi	0.185s
ok  	cmd/internal/pkgpath	0.270s
ok  	cmd/internal/quoted	0.378s
ok  	cmd/internal/src	0.230s
ok  	cmd/internal/sys	0.235s
ok  	cmd/internal/test2json	0.483s
ok  	cmd/link	52.501s
ok  	cmd/link/internal/benchmark	0.250s
ok  	cmd/link/internal/ld	32.176s
ok  	cmd/link/internal/loader	0.219s
ok  	cmd/nm	12.094s
ok  	cmd/objdump	10.416s
ok  	cmd/pack	6.588s
ok  	cmd/pprof	8.455s
ok  	cmd/trace	0.930s
ok  	cmd/vet	18.385s
FAIL
go tool dist: Failed: exit status 1
2022/01/28 12:50:40 release -target="darwin-amd64": waiting a bit and trying again
2022/01/28 12:51:10 release -target="linux-amd64": build succeeded (after 0 retries)
2022/01/28 12:51:10 DONE BUILDRELEASE go1.18beta2 linux-amd64
2022/01/28 12:51:13 release -target="freebsd-386": build succeeded (after 0 retries)
2022/01/28 12:51:13 DONE BUILDRELEASE go1.18beta2 freebsd-386
2022/01/28 12:51:40 $ /Users/mknyszek/go-releasebot-work/go1.18beta2/gopath/bin/release -target darwin-amd64 -user mknyszek -version go1.18beta2 -staging_dir /Users/mknyszek/go-releasebot-work/go1.18beta2/release-staging/9ff00398489d9eb1822b3de028cd6ccf5674ebb3_1256542821 -rev 9ff00398489d9eb1822b3de028cd6ccf5674ebb3
2022/01/28 12:52:22 release -target="linux-386": build succeeded (after 0 retries)
2022/01/28 12:52:22 DONE BUILDRELEASE go1.18beta2 linux-386
2022/01/28 12:57:21 release -target="linux-amd64-longtest": build succeeded (after 0 retries)
2022/01/28 12:57:21 DONE BUILDRELEASE go1.18beta2 linux-amd64-longtest
@mknyszek
Copy link
Contributor Author

CC @dmitshur @toothrot

@mknyszek mknyszek added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Jan 28, 2022
@mknyszek mknyszek added this to the Unreleased milestone Jan 28, 2022
@dmitshur dmitshur added the Builders x/build issues (builders, bots, dashboards) label Jan 28, 2022
@mknyszek mknyszek mentioned this issue Jan 28, 2022
Closed
@mknyszek
Copy link
Contributor Author

I think it's worth noting that the hangs are so long that this is unlikely to be a test deadlocking, because it should have timed out by now.

@cagedmantis
Copy link
Contributor

@golang/release

@toothrot
Copy link
Contributor

toothrot commented Feb 2, 2022

We may want to consider either adding an hour long timeout here, or waiting until these tasks are ported to relui.

@heschi
Copy link
Contributor

heschi commented Aug 8, 2022

Should be obsoleted by the watchdog checks in relui?

@heschi heschi moved this to Planned in Go Release Sep 27, 2022
@heschi heschi closed this as completed Mar 28, 2023
@github-project-automation github-project-automation bot moved this from Planned to Done in Go Release Mar 28, 2023
@golang golang locked and limited conversation to collaborators Mar 27, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
Go Release
Archived in project
Milestone
Unreleased
Development

No branches or pull requests
6 participants
@toothrot @cagedmantis @mknyszek @dmitshur @gopherbot @heschi