[packaging] Bump golang-crossbuild version debian8 by v1v · Pull Request #23872 · elastic/beats

v1v · 2021-02-05T10:46:29Z

What does this PR do?

Bump version:

from debian7 -> debian8. <-- Need to clarify this

Why is it important?

As discussed offline to use debian 8 images

Failures

x-pack/auditbeat packaging failed

Command = PACKAGES='+all linux/amd64 linux/386 linux/arm64 linux/armv7 windows/amd64 windows/386' mage package

Error

[2021-02-05T14:02:50.638Z] >> Building using: cmd='build/mage-linux-amd64 golangCrossBuild', env=[CC=gcc, CXX=g++, GOARCH=386, GOARM=, GOOS=linux, PLATFORM_ID=linux-386]
[2021-02-05T14:03:02.877Z] debconf: delaying package configuration, since apt-utils is not installed
[2021-02-05T14:03:17.797Z] W: GPG error: http://archive.debian.org jessie Release: The following signatures were invalid: KEYEXPIRED 1587841717
[2021-02-05T14:03:20.346Z] W: GPG error: http://archive.debian.org jessie Release: The following signatures were invalid: KEYEXPIRED 1587841717
[2021-02-05T14:03:30.356Z] E: There are problems and -y was used without --force-yes
[2021-02-05T14:03:30.356Z] Error: running "apt-get install -y --no-install-recommends librpm-dev" failed with exit code 100
[2021-02-05T14:03:30.356Z] Error: failed building for linux/amd64: exit status 100
[2021-02-05T14:03:30.356Z] failed building for linux/amd64: exit status 100
[2021-02-05T14:03:35.634Z] E: There are problems and -y was used without --force-yes
[2021-02-05T14:03:35.634Z] Error: running "apt-get install -y --no-install-recommends librpm-dev:i386" failed with exit code 100
[2021-02-05T14:03:35.634Z] Error: failed building for linux/386: exit status 100
[2021-02-05T14:03:35.634Z] failed building for linux/386: exit status 100
[2021-02-05T14:04:02.201Z] # crypto/x509
[2021-02-05T14:04:02.201Z] ld: warning: object file (/tmp/go-build127571197/b115/_cgo_main.o) was built for newer OSX version (10.11) than being linked (10.10)
[2021-02-05T14:04:02.201Z] ld: warning: object file (/tmp/go-build127571197/b115/_x001.o) was built for newer OSX version (10.11) than being linked (10.10)
[2021-02-05T14:04:02.201Z] ld: warning: object file (/tmp/go-build127571197/b115/_x002.o) was built for newer OSX version (10.11) than being linked (10.10)
[2021-02-05T14:07:23.765Z] # github.com/elastic/beats/v7/x-pack/auditbeat
[2021-02-05T14:07:23.765Z] ld: warning: object file (/tmp/go-link-221104519/000029.o) was built for newer OSX version (10.11) than being linked (10.10)
[2021-02-05T14:07:23.765Z] ld: warning: object file (/tmp/go-link-221104519/000030.o) was built for newer OSX version (10.11) than being linked (10.10)
[2021-02-05T14:07:23.765Z] package ran for 5m12.397231978s
[2021-02-05T14:07:23.765Z] Error: running "docker run --env EXEC_UID=1160 --env EXEC_GID=1161 -v /var/lib/jenkins/workspace/Beats_packaging_PR-23872/pkg/mod:/go/pkg/mod:ro --rm --env GOFLAGS=-mod=readonly --env MAGEFILE_VERBOSE= --env MAGEFILE_TIMEOUT= --env SNAPSHOT=true --env DEV=true -v /var/lib/jenkins/workspace/Beats_packaging_PR-23872/src/github.com/elastic/beats:/go/src/github.com/elastic/beats -w /go/src/github.com/elastic/beats/x-pack/auditbeat docker.elastic.co/beats-dev/golang-crossbuild:1.15.7-main-debian8 --build-cmd build/mage-linux-amd64 golangCrossBuild -p linux/amd64" failed with exit code 1
[2021-02-05T14:07:23.765Z] running "docker run --env EXEC_UID=1160 --env EXEC_GID=1161 -v /var/lib/jenkins/workspace/Beats_packaging_PR-23872/pkg/mod:/go/pkg/mod:ro --rm --env GOFLAGS=-mod=readonly --env MAGEFILE_VERBOSE= --env MAGEFILE_TIMEOUT= --env SNAPSHOT=true --env DEV=true -v /var/lib/jenkins/workspace/Beats_packaging_PR-23872/src/github.com/elastic/beats:/go/src/github.com/elastic/beats -w /go/src/github.com/elastic/beats/x-pack/auditbeat docker.elastic.co/beats-dev/golang-crossbuild:1.15.7-main-debian8 --build-cmd build/mage-linux-amd64 golangCrossBuild -p linux/386" failed with exit code 1
script returned exit code 1

…-> debian9)

elasticmachine · 2021-02-05T11:36:27Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2021-11-30T10:53:46.890+0000
Duration: 116 min 59 sec
Commit: 07cd4ce

Test stats 🧪

Test	Results
Failed	0
Passed	48630
Skipped	4286
Total	52916

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

v1v · 2021-02-05T11:42:41Z

/packaging

* upstream/master: (308 commits) [winlogbeat] Add support for sysmon v13 events 24 and 25 (elastic#24945) mergify: add backport label (elastic#25050) Add pod.ip in k8s metadata (elastic#25037) [elastic-agent] Use fleet.url for container cmd (elastic#25026) disable TestXPackEnabled flaky test in logstash metricbeat module (elastic#25034) Leverege leader election in agent k8s manifests (elastic#25016) libbeat/publisher/pipeline: expand monitoring (elastic#24700) libbeat: fix decode_json_fields config validation (elastic#24862) Remove make docs-preview instructions (elastic#25001) [Filebeat] Fix IPtables pipeline (elastic#24928) [DOCS] cd into correct directory before invoking mage. (elastic#17679) Add -buildmode=pie for supported platform (elastic#24964) Add agent's direcotry in k8s manifest generator (elastic#24987) [mergify] assign the original author (elastic#25007) Fix AWS module flaky tests (elastic#24852) [filebeat] Use fail_on_template_error on google_workspace and okta pagination (elastic#24967) Updated config to match defaults (elastic#25004) [Filebeat] Fix hardcoded amazonaws.com endpoint (elastic#24861) Add cloud.service.name to add_cloud_metadata (elastic#24993) [Ingest Manager] Expose processes and their metrics (elastic#24788) ...

v1v · 2021-04-14T12:11:27Z

@jsoriano , I might need some help with the existing failures when packaging the x-pack/auditbeat on Linux. I saw your changes in #21239 a few months ago, any hints what's happening?

Logs -> https://beats-ci.elastic.co//blue/rest/organizations/jenkins/pipelines/Beats/pipelines/beats/pipelines/PR-23872/runs/4/steps/12600/log/?start=0

jsoriano · 2021-04-15T10:58:43Z

@jsoriano , I might need some help with the existing failures when packaging the x-pack/auditbeat on Linux. I saw your changes in #21239 a few months ago, any hints what's happening?

Logs -> https://beats-ci.elastic.co//blue/rest/organizations/jenkins/pipelines/Beats/pipelines/beats/pipelines/PR-23872/runs/4/steps/12600/log/?start=0

@v1v it seems that the base image has an incorrect sources.list, or GPG keys for the repos listed there:

[2021-04-14T10:28:46.514Z] W: GPG error: http://archive.debian.org jessie Release: The following signatures were invalid: KEYEXPIRED 1587841717

It seems that jessie is still available under http://http.debian.net/debian, so maybe we should use this repository instead of http://archive.debian.org/debian.

Official debian:8 image uses http://deb.debian.org/debian.

mergify · 2021-04-15T15:45:06Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b feature/use-debian-9 upstream/feature/use-debian-9
git merge upstream/master
git push upstream feature/use-debian-9

* upstream/master: packer cache support for the 7.x and 7.latestMinor branches (elastic#25091) Remove EventFetcher and EventsFetcher interface (elastic#25093) Update go-structform to 0.0.8 (elastic#25051) Update copy_fields.asciidoc (elastic#25053) [elastic-agent] ensure container is backwards compatible (elastic#25092) Add --fleet-server-service-token. Rename --fleet-server to --fleet-server-es. (elastic#25083) Add cgroup.cpuacct percentages (elastic#25057) Add tests for truncated and symlinked files in filestream input (elastic#24425) Fix panic when Hearbeat monitor initialization fails twice (elastic#25073) [Filebeat][httpjson] Change append transform to initiate new fields as a slice (elastic#25074) Osquerybeat: Result values type translation (elastic#25012) Update Osquerybeat spec to get it downloading from the correct artifactory path (elastic#25076) Fix changelog (elastic#25079) Strip Azure EventHub connection string in debug logs (elastic#25066) Change googlecloud to gcp in field names (elastic#25038) Bump stack version to 7.12.0 for testing (elastic#24957) packer-cache: cache the existing docker images on ARM and some more (elastic#25068) Disable logstash TestFetch flaky test (elastic#25044)

v1v · 2021-06-09T20:51:58Z

/test

v1v · 2021-06-29T14:47:28Z

/test

v1v · 2021-07-05T15:46:57Z

@jsoriano , sorry that I missed this PR for a while.

It seems that jessie is still available under http://http.debian.net/debian, so maybe we should use this repository instead of http://archive.debian.org/debian.

IIUC, Jessie Long Term Support (LTS) ended in June 2020. elastic/golang-crossbuild#72 was the one that moved to debian-9 and changed to use archive for debian-8 and debian-7

I don't know the policy regarding what sources should be used, but I guess the changes to be done should be in the golang-crossbuild, right?

v1v · 2021-07-05T15:49:17Z

/test

(cherry picked from commit cb8f4d3)

(cherry picked from commit cb8f4d3) Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>

andrewkroh · 2022-02-03T14:40:30Z

Yes, ... provided that we add the TestLinuxForCentosGLIBC static test to check the binaries coming out of the build.

This PR proceeded without adding the tests and unfortunately Metricbeat, Packetbeat, and APM Server no longer run on CentOS 6 for 7.17 and 8.0. If you build the Dockerfile below you see the problem that occurs at runtime.

usr/share/metricbeat/bin/metricbeat: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /usr/share/metricbeat/bin/metricbeat)

/usr/share/packetbeat/bin/packetbeat: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /usr/share/packetbeat/bin/packetbeat)

/usr/share/apm-server/bin/apm-server: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /usr/share/apm-server/bin/apm-server)

# Dockerfile
FROM centos:6

ENV SNAPSHOT https://snapshots.elastic.co/7.17.0-fd761817/downloads

RUN curl -O $SNAPSHOT/beats/metricbeat/metricbeat-7.17.0-SNAPSHOT-x86_64.rpm \
      && rpm -ivh metricbeat-7.17.0-SNAPSHOT-x86_64.rpm

RUN curl -O $SNAPSHOT/beats/packetbeat/packetbeat-7.17.0-SNAPSHOT-x86_64.rpm \
      && rpm -ivh packetbeat-7.17.0-SNAPSHOT-x86_64.rpm

RUN curl -O $SNAPSHOT/apm-server/apm-server-7.17.0-SNAPSHOT-x86_64.rpm \
      && rpm -ivh apm-server-7.17.0-SNAPSHOT-x86_64.rpm

RUN /usr/share/metricbeat/bin/metricbeat version \
    || /usr/share/packetbeat/bin/packetbeat version \
    || /usr/share/apm-server/bin/apm-server version

v1v · 2022-02-03T14:55:32Z

:/

I actually misunderstood that we were safe to move, what do we do here?

…29687)" This reverts commit b4f83b3.

…29686)" This reverts commit 16b0da7.

…3872)" This reverts commit cb8f4d3.

andrewkroh · 2022-02-03T15:19:24Z

@v1v is reverting the changes so we can get those Beats back to a working state for CentOS 6. Currently the 7.17.0 release is affected and the 8.0.0 RCs.

Once we get back to a working state we can discuss how to safely move forward with this change to transition off of debian7 for building.

andrewkroh · 2022-02-03T16:32:54Z

I can't find a decision regarding 8.0 support for CentOS 6 in Beats/Agent. But if for 8.0 we are dropping support for Beats and Agent on CentOS/RHEL 6 then the changes might be OK for 8.0 and ~~master~~ main.

We should document the minimum supported GLIBC version for Beats and add the tests I mentioned for it. That will prevent us from accidentally moving to a new builder image in the future that goes beyond our minimum glibc version.

That minimum supported GLIBC would basically be the oldest glibc version found in all the supported OSes.

…29686)" (#30184) This reverts commit 16b0da7.

nimarezainia · 2022-02-04T17:43:02Z

@andrewkroh CentOS EOL November 2020. with all honesty it should have been removed from our support matrix 6-12 months after that. No hard and fast rule about this but I'm trying to clean up our support matrix and set up a policy as such.

I deprecated this support from the matrix in 8.0. the 7.17 matrix was just published. I can try and see if that can be modified.

jlind23 · 2022-02-08T07:27:31Z

@nimarezainia does it mean that we are not supporting RHEL6 from 7.17 on?

jlind23 · 2022-02-08T17:40:02Z

@v1v was it merged back to 7.17 and will it be part of a 7.17.1 release?

v1v · 2022-02-08T17:43:45Z

@v1v was it merged back to 7.17 and will it be part of a 7.17.1 release?

IIIUC, Andrew mentioned that it should not be since it broke compatibility with centos6, see #23872 (comment)
and #23872 (comment)

#30184 is the revert

8.0 and master are actually using this version, so a revert was proposed in :

Revert "[packaging] Bump golang-crossbuild version debian8 (#23872)" #30186
Revert "[8.0](backport #23872) [packaging] Bump golang-crossbuild version debian8" #30185

But they are stalled waiting for the final call whether they should remain or merged

nimarezainia · 2022-02-08T19:03:06Z

@nimarezainia does it mean that we are not supporting RHEL6 from 7.17 on?

yes correct. 7.17 is the last release.

jlind23 · 2022-02-08T19:11:40Z

@v1v sorry I was talking about the revert.
Was it revert on 7.17 branch? If yes, then I'm good, but thanks I have looked at the Pr and it is.
@nimarezainia the next 7.17 will be safe.

nimarezainia · 2022-02-08T19:27:05Z

thank you all. So we will let users know that 7.17.1 is fixed.

…3872) (elastic#29687)" This reverts commit d5e9cf5.

Bump golang-crossbuild version (from debian7 -> debian8, and debian8 …

3a2a016

…-> debian9)

botelastic Bot added the needs_team Indicates that the issue/PR needs a Team:* label label Feb 5, 2021

v1v self-assigned this Feb 5, 2021

v1v added the Packaging label Feb 5, 2021

v1v added 2 commits February 5, 2021 12:20

use main-debian8 docker image

05b8756

use main-debian9 docker image

9f34ddb

v1v requested review from a team, andrewkroh, jsoriano, kvch and urso February 5, 2021 14:54

mdelapenya approved these changes Feb 5, 2021

View reviewed changes

kuisathaverat reviewed Feb 5, 2021

View reviewed changes

Comment thread dev-tools/mage/crossbuild.go

andresrc added the Team:Automation Label for the Observability productivity team label Feb 5, 2021

botelastic Bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Feb 5, 2021

cachedout approved these changes Feb 6, 2021

View reviewed changes

v1v added 2 commits April 14, 2021 10:55

packer-cache update

f71dddc

v1v commented Apr 14, 2021

View reviewed changes

Comment thread .ci/packer_cache.sh Outdated

urso approved these changes Apr 14, 2021

View reviewed changes

v1v added the backport-v8.0.0 Automated backport with mergify label Jan 4, 2022

v1v merged commit cb8f4d3 into elastic:master Jan 4, 2022

v1v deleted the feature/use-debian-9 branch January 4, 2022 17:46

mergify Bot pushed a commit that referenced this pull request Jan 4, 2022

[packaging] Bump golang-crossbuild version debian8 (#23872)

cc07cb6

(cherry picked from commit cb8f4d3)

mergify Bot mentioned this pull request Jan 4, 2022

[7.17](backport #23872) [packaging] Bump golang-crossbuild version debian8 #29686

Merged

mergify Bot pushed a commit that referenced this pull request Jan 4, 2022

[packaging] Bump golang-crossbuild version debian8 (#23872)

c585a57

(cherry picked from commit cb8f4d3)

mergify Bot mentioned this pull request Jan 4, 2022

[8.0](backport #23872) [packaging] Bump golang-crossbuild version debian8 #29687

Merged

v1v added a commit that referenced this pull request Jan 5, 2022

[packaging] Bump golang-crossbuild version debian8 (#23872) (#29687)

b4f83b3

(cherry picked from commit cb8f4d3) Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>

v1v pushed a commit that referenced this pull request Jan 11, 2022

[packaging] Bump golang-crossbuild version debian8 (#23872) (#29686)

16b0da7

v1v restored the feature/use-debian-9 branch February 3, 2022 15:01

v1v added a commit that referenced this pull request Feb 3, 2022

Revert "[packaging] Bump golang-crossbuild version debian8 (#23872) (#…

0c0e1f9

…29687)" This reverts commit b4f83b3.

v1v added a commit that referenced this pull request Feb 3, 2022

Revert "[packaging] Bump golang-crossbuild version debian8 (#23872) (#…

c7c26b5

…29686)" This reverts commit 16b0da7.

v1v added a commit to v1v/beats that referenced this pull request Feb 3, 2022

Revert "[packaging] Bump golang-crossbuild version debian8 (elastic#2…

060e9ab

…3872)" This reverts commit cb8f4d3.

v1v mentioned this pull request Feb 3, 2022

Revert "[packaging] Bump golang-crossbuild version debian8 (#23872)" #30186

Closed

v1v added a commit that referenced this pull request Feb 3, 2022

Revert "[packaging] Bump golang-crossbuild version debian8 (#23872) (#…

13c9079

…29686)" (#30184) This reverts commit 16b0da7.

This was referenced Dec 1, 2022

Add workaround for expried GPG keys for auditbeat packaging #33915

Merged

Enforce "yes" when install debian packages in packaging #33895

Merged

v1v mentioned this pull request Apr 18, 2023

Stop crossbuilding using Debian 8 #34921

Merged

leweafan pushed a commit to leweafan/beats that referenced this pull request Apr 28, 2023

Revert "[packaging] Bump golang-crossbuild version debian8 (elastic#2…

231f253

…3872) (elastic#29687)" This reverts commit d5e9cf5.

Conversation

v1v commented Feb 5, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Why is it important?

Failures

Uh oh!

elasticmachine commented Feb 5, 2021 • edited by jenkins-beats-ci Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

Test stats 🧪

💚 Flaky test report

🤖 GitHub comments

Uh oh!

v1v commented Feb 5, 2021

Uh oh!

Uh oh!

Uh oh!

v1v commented Apr 14, 2021

Uh oh!

jsoriano commented Apr 15, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mergify Bot commented Apr 15, 2021

Uh oh!

v1v commented Jun 9, 2021

Uh oh!

v1v commented Jun 29, 2021

Uh oh!

v1v commented Jul 5, 2021

Uh oh!

v1v commented Jul 5, 2021

Uh oh!

andrewkroh commented Feb 3, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

v1v commented Feb 3, 2022

Uh oh!

andrewkroh commented Feb 3, 2022

Uh oh!

andrewkroh commented Feb 3, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nimarezainia commented Feb 4, 2022

Uh oh!

jlind23 commented Feb 8, 2022

Uh oh!

jlind23 commented Feb 8, 2022

Uh oh!

v1v commented Feb 8, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nimarezainia commented Feb 8, 2022

Uh oh!

jlind23 commented Feb 8, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nimarezainia commented Feb 8, 2022 • edited by andrewkroh Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

13 participants

v1v commented Feb 5, 2021 •

edited

Loading

elasticmachine commented Feb 5, 2021 •

edited by jenkins-beats-ci Bot

Loading

jsoriano commented Apr 15, 2021 •

edited

Loading

andrewkroh commented Feb 3, 2022 •

edited

Loading

andrewkroh commented Feb 3, 2022 •

edited

Loading

v1v commented Feb 8, 2022 •

edited

Loading

jlind23 commented Feb 8, 2022 •

edited

Loading

nimarezainia commented Feb 8, 2022 •

edited by andrewkroh

Loading