Fix/go14.4.6 by kuisathaverat · Pull Request #62 · elastic/golang-crossbuild

kuisathaverat · 2020-07-20T16:07:27Z

bump go version to 1.14.6
split ARM arch
split x86 arch
split MIPS arch
split ppc arch

see https://www.debian.org/releases/stretch/mips/ch02s01.html.en

related to elastic/beats#19677 elastic/beats#11750

andrewkroh · 2020-07-21T11:15:07Z

+  armv5:
+    CC:  arm-linux-gnueabi-gcc
+    CXX: arm-linux-gnueabi-g++
+  arm64:


I think arm64 and armv7 can be removed since these compilers are part of different images.

andrewkroh · 2020-07-21T11:16:37Z

+  armv7:
+    CC:  arm-linux-gnueabihf-gcc
+    CXX: arm-linux-gnueabihf-g++
+  armv6:


Only the armv7 compiler should be in this file.

andrewkroh · 2020-07-21T11:18:53Z

@@ -1,4 +1,4 @@
-IMAGES := base main darwin arm mips ppc s390x
+IMAGES := base main amd64 darwin arm armel armhf mips ppc s390x


Perhaps arm should be renamed to arm64.

And main has effectively become the i386 image.

We'll have to update anything that uses the images to pull the right arch flavor so I guess if we're changing one (main -> amd64) then it won't be too much to rename arm and main...?

yes, we have to change the arch we pass when we launch the images, I start to find where this is in beats but I did not found it yet, Do you known where I have to touch to change it.

It's at https://github.com/elastic/beats/blob/ded2e9912bb669e896b9c8d013dfabdb759023bd/dev-tools/mage/crossbuild.go#L180-L206. Journalbeat's magefile has a custom ImageSelectorFunc in order to use a different image due to debian7 + systemd issues. It will need updated too.

We should probably make the image location configurable in Beats to allow for easier testing of the staged images (e.g. export BEATS_GOLANG_CROSSBUILD_IMAGE=docker.elastic.co/observability-ci/golang-crossbuild)

BeatsCrossBuildImage = EnvOr("BEATS_GOLANG_CROSSBUILD_IMAGE", "docker.elastic.co/beats-dev/golang-crossbuild")

https://github.com/elastic/beats/blob/7580c72445ecc6e97cab6fb7bcffc8063abf105b/dev-tools/mage/settings.go#L46

kuisathaverat · 2020-07-22T13:36:12Z

I have checked supported architectures on CentOS, RedHat, Ubuntu, and Debian, the mips64 and ppc64 architectures are not supported by any of those distributions. On Debian, we have compilers for mips64 and ppc64 but do not have binary packages and it is not easy to prepare a bootstrap filesystem to compile all the packages we need. I think we can rid of mips64 and ppc64, these will be the supported architectures are:

Architecture	Debian Designation	Subarchitecture	Flavor
Intel x86-based	i386	default x86 machines	default
		Xen PV domains only	xen
AMD64 & Intel 64	amd64
ARM	armel	Marvell Kirkwood and Orion	marvell
ARM with hardware FPU	armhf	multiplatform	armmp
64bit ARM	arm64
32bit MIPS (big-endian)	mips	MIPS Malta	4kc-malta
		Cavium Octeon	octeon
64bit MIPS (little-endian)	mips64el	MIPS Malta	5kc-malta
		Cavium Octeon	octeon
		Loongson 3	loongson-3
32bit MIPS (little-endian)	mipsel	MIPS Malta	4kc-malta
		Cavium Octeon	octeon
		Loongson 3	loongson-3
Power Systems	ppc64el	IBM POWER8 or newer machines
64bit IBM S/390	s390x	IPL from VM-reader and DASD	generic

@andrewkroh @urso @elastic/observablt-robots

kuisathaverat · 2020-07-22T19:14:50Z

I have made a local test on how hard is to move down to Debian 8 (libc6-2.18), for amd64 and i386 is easy for the rest of the architectures we need to compile packages, I am not sure it worth it. Debian 7(libc6-2.13) need to compile a bunch of packages on any architecture. In my opinion, we can keep amd64 and i386 on Debian 8, and the rest should use Debian 9.

urso · 2020-07-23T13:17:04Z

I have made a test on how hard is to move down to Debian 8 (libc6-2.18), for amd64 and i386 is easy for the rest of the architectures we need to compile packages, I am not sure it worth it. Debian 7(libc6-2.13) need to compile a bunch of packages on any architecture. In my opinion, we can keep amd64 and i386 on Debian 8, and the rest should use Debian 9.

Can we reduce the amount of changes we introduce here? For the 7.9 release we only need go to be updated to go1.14.6

kuisathaverat · 2020-07-23T13:29:22Z

@urso In that case, I will open another PR with only the change on the go version nothin else.

the changes on this PR are the ones commented on the first comment nothing else, the images are generated with Debian 9, as they are now.

kuisathaverat · 2020-11-10T09:08:29Z

@andresrc @urso @andrewkroh we have to make a decision here, Do we want to split the Docker images in single architectures or not?

urso · 2020-11-10T09:54:23Z

we have to make a decision here, Do we want to split the Docker images in single architectures or not?

In general I'm +1 on splitting. What would be the disadvantages of doing this?

kuisathaverat · 2020-11-10T14:31:46Z

we have to make a decision here, Do we want to split the Docker images in single architectures or not?

In general I'm +1 on splitting. What would be the disadvantages of doing this?

I do not see any, the images are easy to maintain, are smaller, and you use only the one you need.

andrewkroh · 2020-12-04T16:12:45Z

+1 on splitting the images. Hopefully some of the layers can be shared between all of the images to minimize the amount of downloads/storage (but that's just an optimization and certainly not a requirement).

fearful-symmetry · 2020-12-16T17:08:58Z

What's the status on this? It's blocking #71, which we'd like to resolve.

kuisathaverat · 2020-12-17T16:02:50Z

What's the status on this? It's blocking #71, which we'd like to resolve.

This was open in Jul, we have to re-work it over the new Go version, I will finish with #72 and the nest thing to do is the split, give me a few days

urso · 2020-12-17T17:16:14Z

we have to re-work it over the new Go version

Beats will not update to go 1.15 for the 7.11 release btw.

kuisathaverat · 2020-12-17T19:20:01Z

@urso @fearful-symmetry Do we need it for 7.11? if so I will update this PR.

urso · 2020-12-17T19:21:14Z

AFAICT there is no immedate need for this.

fearful-symmetry · 2020-12-17T21:02:06Z

Yah, I think we're looking at 7.12 for the additional OS support.

kuisathaverat · 2021-05-17T18:56:36Z

superseded by #89

kuisathaverat added 3 commits July 20, 2020 18:04

feat: bump go version

3eb5680

feat:splt ARM arch

ca9dc1a

eat: split x86 arch

5a9d00b

kuisathaverat added enhancement New feature or request automation Team:Automation Label for the Observability productivity team labels Jul 20, 2020

kuisathaverat requested review from a team and urso July 20, 2020 16:07

urso approved these changes Jul 20, 2020

View reviewed changes

mdelapenya approved these changes Jul 20, 2020

View reviewed changes

cachedout approved these changes Jul 21, 2020

View reviewed changes

andrewkroh reviewed Jul 21, 2020

View reviewed changes

kuisathaverat added 7 commits July 21, 2020 19:23

feat: split ARM arch

e26d226

feat: split x86 arch

9493209

feta: split MIPS arch

de03bf1

feta: test for darwin

6eb80c7

feat: split ppc arch

fc70b6d

feat: test for s390x

29b4633

feat: build all archs

9732b82

kuisathaverat self-assigned this Jul 27, 2020

zube Bot added the [zube]: In Review label Jul 27, 2020

chore: refactor

ab00bb5

kuisathaverat mentioned this pull request Dec 1, 2020

Reenable MIPS, PPC, s309x builder images from go1.13, go1.14 #71

Closed

kuisathaverat mentioned this pull request Dec 1, 2020

Add support for Golang 1.15.4 #68

Closed

kuisathaverat mentioned this pull request Jan 7, 2021

Package conflicts in ARM images #67

Closed

kuisathaverat closed this May 17, 2021

zube Bot added [zube]: Done and removed [zube]: In Review labels May 17, 2021

zube Bot removed the [zube]: Done label Aug 16, 2021

		@@ -1,4 +1,4 @@
		IMAGES := base main darwin arm mips ppc s390x
		IMAGES := base main amd64 darwin arm armel armhf mips ppc s390x

Conversation

kuisathaverat commented Jul 20, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

kuisathaverat commented Jul 22, 2020

Uh oh!

kuisathaverat commented Jul 22, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

urso commented Jul 23, 2020

Uh oh!

kuisathaverat commented Jul 23, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kuisathaverat commented Nov 10, 2020

Uh oh!

urso commented Nov 10, 2020

Uh oh!

kuisathaverat commented Nov 10, 2020

Uh oh!

andrewkroh commented Dec 4, 2020

Uh oh!

fearful-symmetry commented Dec 16, 2020

Uh oh!

kuisathaverat commented Dec 17, 2020

Uh oh!

urso commented Dec 17, 2020

Uh oh!

kuisathaverat commented Dec 17, 2020

Uh oh!

urso commented Dec 17, 2020

Uh oh!

fearful-symmetry commented Dec 17, 2020

Uh oh!

kuisathaverat commented May 17, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

kuisathaverat commented Jul 20, 2020 •

edited

Loading

kuisathaverat commented Jul 22, 2020 •

edited

Loading

kuisathaverat commented Jul 23, 2020 •

edited

Loading