[7.13] packaging is failing

[v1v]

The packaging in 7.13 is broken, last successful build that happened was -> https://beats-ci.elastic.co/job/Beats/job/packaging/job/7.13/113/ , in case it has not been reported yet.

Failures happen in:

• The main pipeline
• the packaging pipeline -> https://beats-ci.elastic.co/blue/organizations/jenkins/Beats%2Fpackaging/detail/7.13/170/pipeline

[2021-06-29T03:10:34.754Z] + mage package
[2021-06-29T03:10:56.747Z] Generated fields.yml for auditbeat to /var/lib/jenkins/workspace/Beats_packaging_7.13/src/github.com/elastic/beats/auditbeat/fields.yml
[2021-06-29T03:10:56.747Z] Generated fields.yml for auditbeat to /var/lib/jenkins/workspace/Beats_packaging_7.13/src/github.com/elastic/beats/auditbeat/fields.yml
[2021-06-29T03:10:56.747Z] >> Building auditbeat.yml for linux/amd64
[2021-06-29T03:10:56.747Z] >> Building auditbeat.reference.yml for linux/amd64
[2021-06-29T03:10:56.747Z] >> Building auditbeat.docker.yml for linux/amd64
[2021-06-29T03:10:59.298Z] exec: go list -m
[2021-06-29T03:11:21.303Z] >> golangCrossBuild: Building for windows/amd64
[2021-06-29T03:11:21.303Z] >> golangCrossBuild: Building for linux/amd64
[2021-06-29T03:11:21.303Z] >> golangCrossBuild: Building for darwin/amd64
[2021-06-29T03:11:21.303Z] >> buildGoDaemon: Building for linux/amd64
[2021-06-29T03:11:21.303Z] >> buildGoDaemon: Building for linux/386
[2021-06-29T03:11:21.303Z] >> buildGoDaemon: Building for linux/arm64
[2021-06-29T03:11:21.303Z] >> golangCrossBuild: Building for linux/386
[2021-06-29T03:11:21.303Z] >> buildGoDaemon: Building for linux/armv7
[2021-06-29T03:11:23.218Z] >> Building using: cmd='build/mage-linux-amd64 buildGoDaemon', env=[CC=gcc, CXX=g++, GOARCH=386, GOARM=, GOOS=linux, PLATFORM_ID=linux-386]
[2021-06-29T03:11:23.480Z] >> Building using: cmd='build/mage-linux-amd64 golangCrossBuild', env=[CC=x86_64-w64-mingw32-gcc, CXX=x86_64-w64-mingw32-g++, GOARCH=amd64, GOARM=, GOOS=windows, PLATFORM_ID=windows-amd64]
[2021-06-29T03:11:23.480Z] >> Building using: cmd='build/mage-linux-amd64 buildGoDaemon', env=[CC=arm-linux-gnueabihf-gcc, CXX=arm-linux-gnueabihf-g++, GOARCH=arm, GOARM=7, GOOS=linux, PLATFORM_ID=linux-armv7]
[2021-06-29T03:11:23.480Z] >> Building using: cmd='build/mage-linux-amd64 golangCrossBuild', env=[CC=o64-clang, CXX=o64-clang++, GOARCH=amd64, GOARM=, GOOS=darwin, PLATFORM_ID=darwin-amd64]
[2021-06-29T03:11:23.745Z] >> Building using: cmd='build/mage-linux-amd64 golangCrossBuild', env=[CC=gcc, CXX=g++, GOARCH=386, GOARM=, GOOS=linux, PLATFORM_ID=linux-386]
[2021-06-29T03:11:24.010Z] >> Building using: cmd='build/mage-linux-amd64 buildGoDaemon', env=[CC=gcc, CXX=g++, GOARCH=amd64, GOARM=, GOOS=linux, PLATFORM_ID=linux-amd64]
[2021-06-29T03:11:24.010Z] >> Building using: cmd='build/mage-linux-amd64 buildGoDaemon', env=[CC=aarch64-linux-gnu-gcc, CXX=aarch64-linux-gnu-g++, GOARCH=arm64, GOARM=, GOOS=linux, PLATFORM_ID=linux-arm64]
[2021-06-29T03:11:24.010Z] >> Building using: cmd='build/mage-linux-amd64 golangCrossBuild', env=[CC=gcc, CXX=g++, GOARCH=amd64, GOARM=, GOOS=linux, PLATFORM_ID=linux-amd64]
[2021-06-29T03:11:24.010Z] Error: failed to run "arm-linux-gnueabihf-gcc /go/pkg/mod/github.com/tsg/go-daemon@v0.0.0-20200207173439-e704b93fd89b/src/god.c -o build/golang-crossbuild/god-linux-armv7 -lpthread -static: exec: "arm-linux-gnueabihf-gcc": executable file not found in $PATH"
[2021-06-29T03:11:24.010Z] Error: failed building for linux/armv7: exit status 1
[2021-06-29T03:11:24.010Z] failed building for linux/armv7: exit status 1
[2021-06-29T03:11:24.289Z] /tmp/cc5RYq04.o: In function `main':
[2021-06-29T03:11:24.289Z] god.c:(.text+0x2e0): warning: Using 'getgrnam' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
[2021-06-29T03:11:24.289Z] god.c:(.text+0x279): warning: Using 'getpwnam' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
[2021-06-29T03:11:24.289Z] >> golangCrossBuild: Building for linux/armv7
[2021-06-29T03:11:24.880Z] >> golangCrossBuild: Building for linux/arm64
[2021-06-29T03:11:24.880Z] /tmp/ccVk7CzU.o: In function `main':
[2021-06-29T03:11:24.880Z] god.c:(.text+0x2a6): warning: Using 'getgrnam' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
[2021-06-29T03:11:24.880Z] god.c:(.text+0x23e): warning: Using 'getpwnam' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
[2021-06-29T03:11:25.475Z] >> golangCrossBuild: Building for windows/386
[2021-06-29T03:11:25.476Z] /tmp/ccwd41ZR.o: In function `main':
[2021-06-29T03:11:25.476Z] god.c:(.text+0x2ac): warning: Using 'getgrnam' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
[2021-06-29T03:11:25.476Z] god.c:(.text+0x230): warning: Using 'getpwnam' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
[2021-06-29T03:11:25.476Z] >> Building using: cmd='build/mage-linux-amd64 golangCrossBuild', env=[CC=arm-linux-gnueabihf-gcc, CXX=arm-linux-gnueabihf-g++, GOARCH=arm, GOARM=7, GOOS=linux, PLATFORM_ID=linux-armv7]
[2021-06-29T03:11:25.753Z] >> Building using: cmd='build/mage-linux-amd64 golangCrossBuild', env=[CC=aarch64-linux-gnu-gcc, CXX=aarch64-linux-gnu-g++, GOARCH=arm64, GOARM=, GOOS=linux, PLATFORM_ID=linux-arm64]
[2021-06-29T03:11:26.346Z] >> Building using: cmd='build/mage-linux-amd64 golangCrossBuild', env=[CC=i686-w64-mingw32-gcc, CXX=i686-w64-mingw32-g++, GOARCH=386, GOARM=, GOOS=windows, PLATFORM_ID=windows-386]
[2021-06-29T03:11:29.724Z] # runtime/cgo
[2021-06-29T03:11:29.725Z] cgo: exec arm-linux-gnueabihf-gcc: exec: "arm-linux-gnueabihf-gcc": executable file not found in $PATH
[2021-06-29T03:12:51.242Z] # crypto/x509
[2021-06-29T03:12:51.242Z] ld: warning: object file (/tmp/go-build371873831/b100/_cgo_main.o) was built for newer OSX version (10.11) than being linked (10.10)
[2021-06-29T03:12:51.242Z] ld: warning: object file (/tmp/go-build371873831/b100/_x001.o) was built for newer OSX version (10.11) than being linked (10.10)
[2021-06-29T03:12:51.242Z] ld: warning: object file (/tmp/go-build371873831/b100/_x002.o) was built for newer OSX version (10.11) than being linked (10.10)
[2021-06-29T03:13:01.244Z] Error: running "go build -o build/golang-crossbuild/auditbeat-linux-armv7 -buildmode pie -ldflags -s -X github.com/elastic/beats/v7/libbeat/version.buildTime=2021-06-29T03:11:25Z -X github.com/elastic/beats/v7/libbeat/version.commit=7ec48d6f955d7960d4b60885ec565e1907907afb" failed with exit code 2
[2021-06-29T03:13:01.244Z] Error: failed building for linux/armv7: exit status 2
[2021-06-29T03:13:01.244Z] failed building for linux/armv7: exit status 2
[2021-06-29T03:15:07.818Z] # github.com/elastic/beats/v7/auditbeat
[2021-06-29T03:15:07.818Z] ld: warning: object file (/tmp/go-link-710788364/000029.o) was built for newer OSX version (10.11) than being linked (10.10)
[2021-06-29T03:15:07.818Z] ld: warning: object file (/tmp/go-link-710788364/000030.o) was built for newer OSX version (10.11) than being linked (10.10)
[2021-06-29T03:15:07.818Z] package ran for 4m8.541621481s
[2021-06-29T03:15:07.819Z] Error: running "docker run --env EXEC_UID=1168 --env EXEC_GID=1169 -v /var/lib/jenkins/workspace/Beats_packaging_7.13/pkg/mod:/go/pkg/mod:ro --rm --env GOFLAGS=-mod=readonly --env MAGEFILE_VERBOSE= --env MAGEFILE_TIMEOUT= --env SNAPSHOT=true --env DEV=true -v /var/lib/jenkins/workspace/Beats_packaging_7.13/src/github.com/elastic/beats:/go/src/github.com/elastic/beats -w /go/src/github.com/elastic/beats/auditbeat docker.elastic.co/beats-dev/golang-crossbuild:1.15.13-arm --build-cmd build/mage-linux-amd64 buildGoDaemon -p linux/armv7" failed with exit code 1
[2021-06-29T03:15:07.819Z] running "docker run --env EXEC_UID=1168 --env EXEC_GID=1169 -v /var/lib/jenkins/workspace/Beats_packaging_7.13/pkg/mod:/go/pkg/mod:ro --rm --env GOFLAGS=-mod=readonly --env MAGEFILE_VERBOSE= --env MAGEFILE_TIMEOUT= --env SNAPSHOT=true --env DEV=true -v /var/lib/jenkins/workspace/Beats_packaging_7.13/src/github.com/elastic/beats:/go/src/github.com/elastic/beats -w /go/src/github.com/elastic/beats/auditbeat docker.elastic.co/beats-dev/golang-crossbuild:1.15.13-arm --build-cmd build/mage-linux-amd64 golangCrossBuild -p linux/armv7" failed with exit code 1
script returned exit code 1

[botelastic]

This issue doesn't have a Team:<team> label.

[v1v]

My guess:

• https://beats-ci.elastic.co/blue/organizations/jenkins/Beats%2Fbeats/detail/7.13/144/pipeline worked find without the armv7 platform.
• the packaging pipeline had armv7 platform for a year or so, see https://github.com/elastic/beats/blame/7.13/.ci/packaging.groovy#L141

Recent changes in 7.13 have affected armv7 for some reason

[faec]

armv7 packaging was disabled May 28 in #25977, I think because of compatibility issues with recent versions (@urso, any insight?) It was reenabled yesterday in #26575, which seems to be what is causing the breakage... @v1v, can you revert that change until / unless the packaging for armv7 is fixed?