Improve the multiarch situation; armel => armhf; reenable pcc64le; remove the patched golang #38926

luxas · 2016-12-17T18:03:04Z

What this PR does / why we need it:

Improves the multiarch situation as described in Multi-architecture plan for Kubernetes #38067
Tries to bump to go1.8 for arm (and later enable ppc64le)
GOARM 6 => GOARM 7
Remove the golang 1.7 patch
armel => armhf
Bump QEMU version to v2.7.0

Release note:

Improve the ARM builds and make hyperkube on ARM working again by upgrading the Go version for ARM to go1.8beta2

@kubernetes/sig-testing-misc @jessfraz @ixdy @jbeda @david-mcmahon @pwittrock

k8s-reviewable · 2016-12-17T18:03:10Z

This change is

luxas · 2016-12-17T18:07:04Z

The problem is here that we can't upgrade to go1.8 yet due to vendoring issues. I caught this at least:

panic: standard_init_linux.go:175: exec user process caused "no such file or directory" [recovered]
	panic: standard_init_linux.go:175: exec user process caused "no such file or directory"

goroutine 1 [running, locked to thread]:
panic(0x3360b0, 0x56e56a80)
	/usr/local/go/src/runtime/panic.go:481 +0x330
github.com/urfave/cli.HandleAction.func1(0x56e03960)
	/go/src/github.com/opencontainers/runc/Godeps/_workspace/src/github.com/urfave/cli/app.go:478 +0x328
panic(0x3360b0, 0x56e56a80)
	/usr/local/go/src/runtime/panic.go:443 +0x448
github.com/opencontainers/runc/libcontainer.(*LinuxFactory).StartInitialization.func1(0x56e035e8, 0x56dda890, 0x56e03670)
	/go/src/github.com/opencontainers/runc/Godeps/_workspace/src/github.com/opencontainers/runc/libcontainer/factory_linux.go:259 +0x13c
github.com/opencontainers/runc/libcontainer.(*LinuxFactory).StartInitialization(0x56e08e70, 0x46433cb0, 0x56e56a80)
	/go/src/github.com/opencontainers/runc/Godeps/_workspace/src/github.com/opencontainers/runc/libcontainer/factory_linux.go:277 +0x498
main.glob.func8(0x56e42320, 0x0, 0x0)
	/go/src/github.com/opencontainers/runc/main_unix.go:26 +0x58
reflect.Value.call(0x2eb600, 0x3f2948, 0x13, 0x3648b8, 0x4, 0x56e03920, 0x1, 0x1, 0x0, 0x0, ...)
	/usr/local/go/src/reflect/value.go:435 +0xeb4
reflect.Value.Call(0x2eb600, 0x3f2948, 0x13, 0x56e03920, 0x1, 0x1, 0x0, 0x0, 0x0)
	/usr/local/go/src/reflect/value.go:303 +0x84
github.com/urfave/cli.HandleAction(0x2eb600, 0x3f2948, 0x56e42320, 0x0, 0x0)
	/go/src/github.com/opencontainers/runc/Godeps/_workspace/src/github.com/urfave/cli/app.go:487 +0x230
github.com/urfave/cli.Command.Run(0x366cd8, 0x4, 0x0, 0x0, 0x0, 0x0, 0x0, 0x3d0d50, 0x51, 0x0, ...)
	/go/src/github.com/opencontainers/runc/Godeps/_workspace/src/github.com/urfave/cli/command.go:191 +0xcd0
github.com/urfave/cli.(*App).Run(0x56e3a240, 0x56dca070, 0x2, 0x2, 0x0, 0x0)
	/go/src/github.com/opencontainers/runc/Godeps/_workspace/src/github.com/urfave/cli/app.go:240 +0x84c
main.main()
	/go/src/github.com/opencontainers/runc/main.go:137 +0xbb4

Is there anyone from Google that is responsible for upgrading to go1.8 in time for v1.6?
@jessfraz or @dims or someone else?

dims · 2016-12-17T18:15:57Z

@luxas here's the latest status from @jessfraz #38228 (comment)

ixdy

would it be worth fixing some of these things (e.g. moving everything to go1.7) while we're working through the go1.8 issues?

ixdy · 2016-12-22T17:35:53Z

cluster/images/etcd/Makefile

@@ -31,14 +31,14 @@ REGISTRY_TAG?=3.0.14
 ARCH?=amd64
 REGISTRY?=gcr.io/google_containers
 GOLANG_VERSION?=1.6.3


should we bump this one too?

yes I'll do it

ixdy · 2016-12-22T17:38:02Z

build/cni/Makefile

@@ -22,13 +22,14 @@ CNI_RELEASE?=07a8a28637e97b22eb8dfe710eeae1344f69d16e
 CNI_TARBALL=cni-$(ARCH)-$(CNI_RELEASE).tar.gz
 CUR_DIR=$(shell pwd)
 OUTPUT_DIR=$(CUR_DIR)/output
+GOLANG_VERSION=1.7


might want to pin this to an explicit version? otherwise docker run may not fetch a newer patch release.

is there any way to make docker always pull on docker run?
I'm sure there is, but seems like I've forgotten it for the moment

docker build has --pull, but for run you have to docker pull then docker run - see moby/moby#13331.

I think we might still want to fix this. can we make this golang:1.7.4 explicitly?

ixdy · 2016-12-22T17:39:57Z

build/kube-dns/Makefile

@@ -25,15 +25,15 @@ ARCH?=amd64
 TAG?=1.9
 REGISTRY?=gcr.io/google_containers

-GOLANG_VERSION=1.6
+GOLANG_VERSION=1.7


I don't think this is even used here?

nope it isn't. this was just an automatic replace. will fix

ixdy · 2016-12-22T17:41:26Z

test/images/serve_hostname/Makefile

@@ -29,7 +29,7 @@ ARCH ?= amd64

 ALL_ARCH = amd64 arm arm64 ppc64le s390x

-GOARM=6
+GOARM=7
 TEMP_DIR := $(shell mktemp -d)
 GOLANG_VERSION = 1.6.3


maybe update this one?

ixdy · 2016-12-22T17:42:05Z

build/build-image/cross/Dockerfile

@@ -79,15 +79,12 @@ RUN export ETCD_VERSION=v3.0.14; \
  && ln -s ../src/etcd/etcd-${ETCD_VERSION}-linux-amd64/etcd /usr/local/bin/

 # TODO: Remove the patched GOROOT when we have an official golang that has a working arm and ppc64le linker
-ENV K8S_PATCHED_GOLANG_VERSION=1.7.4 \
+ENV K8S_PATCHED_GOLANG_VERSION=go1.8beta2 \


I'd kinda like to see us get away from having two separate go versions. Wonder if go1.8 will land in time for k8s 1.6?

Yes, I really hope it will. That's my intention and goal.
We have to validate it and do the hard work with the betas though

The problem, and the reason this is wip is that k8s is not working with go1.8 yet :(

luxas · 2017-01-06T10:27:15Z

I think I found the root cause, we might just have to bump runc: opencontainers/runc#1116

luxas · 2017-01-08T20:16:59Z

@runcom Even after upgrading runc I get the following when running on arm for all kube binaries on startup:

panic: standard_init_linux.go:175: exec user process caused "no such file or directory" [recovered]
	panic: standard_init_linux.go:175: exec user process caused "no such file or directory"

goroutine 1 [running, locked to thread]:
panic(0x3339c8, 0x10747480)
	/usr/local/go/src/runtime/panic.go:481 +0x330
github.com/urfave/cli.HandleAction.func1(0x1073b960)
	/go/src/github.com/opencontainers/runc/Godeps/_workspace/src/github.com/urfave/cli/app.go:478 +0x328
panic(0x3339c8, 0x10747480)
	/usr/local/go/src/runtime/panic.go:443 +0x448
github.com/opencontainers/runc/libcontainer.(*LinuxFactory).StartInitialization.func1(0x1073b5e8, 0x1071a240, 0x1073b670)
	/go/src/github.com/opencontainers/runc/Godeps/_workspace/src/github.com/opencontainers/runc/libcontainer/factory_linux.go:259 +0x13c
github.com/opencontainers/runc/libcontainer.(*LinuxFactory).StartInitialization(0x10766f30, 0xb65e5448, 0x10747480)
	/go/src/github.com/opencontainers/runc/Godeps/_workspace/src/github.com/opencontainers/runc/libcontainer/factory_linux.go:277 +0x498
main.glob.func8(0x10790320, 0x0, 0x0)
	/go/src/github.com/opencontainers/runc/main_unix.go:26 +0x58
reflect.Value.call(0x2e90f8, 0x3efd4c, 0x13, 0x362250, 0x4, 0x1073b920, 0x1, 0x1, 0x0, 0x0, ...)
	/usr/local/go/src/reflect/value.go:435 +0xeb4
reflect.Value.Call(0x2e90f8, 0x3efd4c, 0x13, 0x1073b920, 0x1, 0x1, 0x0, 0x0, 0x0)
	/usr/local/go/src/reflect/value.go:303 +0x84
github.com/urfave/cli.HandleAction(0x2e90f8, 0x3efd4c, 0x10790320, 0x0, 0x0)
	/go/src/github.com/opencontainers/runc/Godeps/_workspace/src/github.com/urfave/cli/app.go:487 +0x230
github.com/urfave/cli.Command.Run(0x364678, 0x4, 0x0, 0x0, 0x0, 0x0, 0x0, 0x3ce288, 0x51, 0x0, ...)
	/go/src/github.com/opencontainers/runc/Godeps/_workspace/src/github.com/urfave/cli/command.go:191 +0xcd0
github.com/urfave/cli.(*App).Run(0x10782240, 0x10762050, 0x2, 0x2, 0x0, 0x0)
	/go/src/github.com/opencontainers/runc/Godeps/_workspace/src/github.com/urfave/cli/app.go:240 +0x84c
main.main()
	/go/src/github.com/opencontainers/runc/main.go:137 +0xbb4

using go1.8, latest runc and running on ARM. Do you know what the cause could be?

luxas · 2017-01-08T20:38:43Z

cc @vmarmol @crosbymichael @hqhq and @cyphar on the above exec user process caused "no such file or directory" runc panic on all k8s binaries when running on ARM and using hyperkube (at least w/ go1.8beta2 right now)

Do you know what the cause could be? Why is this code running on init?

cyphar · 2017-01-09T03:37:29Z

@luxas How is runc being executed, do you know? That error is usually when you try to runc start a container where the process.args process doesn't exist in the rootfs.

luxas · 2017-01-09T06:21:43Z

@cyphar I believe the problem is that LinuxFactory.StartInitialization() is run somehow during the init() in the k8s Godeps. It should absolutely not run when starting the scheduler with hyperkube for example.

The problem is that I can't find how it's run at startup and why...

luxas · 2017-01-09T06:27:35Z

What's even worse is that we don't even vendor /go/src/github.com/opencontainers/runc/main.go:137 or /go/src/github.com/opencontainers/runc/Godeps/_workspace/src/github.com/urfave/cli/command.go:191 into k8s, so I don't really know what's going on here, seems like we have compiled k8s with more code than we need or something

cyphar · 2017-01-09T07:08:31Z

@luxas Does the first argument equal "init"? Inside runC we have to use init in order to handle some of the setup inside Go (namely the StartInitialization code). Also, if you're vendoring github.com/opencontainers/runc/libcontainer/nsenter you're not going to have a fun time.

Pensu · 2017-01-11T10:10:25Z

build/build-image/cross/Dockerfile

  && GOROOT_FINAL=${K8S_PATCHED_GOROOT} GOROOT_BOOTSTRAP=/usr/local/go ./make.bash \
-  && for platform in linux/arm; do GOOS=${platform%/*} GOARCH=${platform##*/} GOROOT=${K8S_PATCHED_GOROOT} go install std; done
+  && for platform in linux/arm linux/ppc64le; do GOOS=${platform%/*} GOARCH=${platform##*/} GOROOT=${K8S_PATCHED_GOROOT} go install std; done


I think patching is not required for golang for ppc64le with go1.8. All the changes are already upstreamed in go1.8.

This PR makes k8s not use a patched golang, but the name will still be K8S_PATCHED_GOROOT until k8s uses go1.8 offically

ixdy · 2017-01-23T22:18:47Z

yeah, easy repro outside of our containerized build is to use go 1.7.4 with GOROOT set to 1.8rc2 (or rc1) and then go test -c any test package.

…1.8rc1 for arm and ppc64le instead. Reenable the ppc64le builds

…and goarm=6 => goarm=7 and use go 1.7.4

ixdy

looks like you already fixed the one main issue. minor stylistic comment, otherwise I think this basically LG. I'll do one more full pass in a little bit.

ixdy · 2017-01-27T18:26:13Z

hack/lib/golang.sh

-  V=2 kube::log::info "Env for ${platform}: GOOS=${GOOS-} GOARCH=${GOARCH-} GOROOT=${GOROOT-} CGO_ENABLED=${CGO_ENABLED-} CC=${CC-}"
+  # Temporary workaround while we have two GOROOT's (which we'll get rid of as soon as we upgrade to go1.8 for amd64 as well)
+  local GO=go
+  if [[ ${GOROOT} == ${K8S_EDGE_GOROOT:-} ]]; then


nit: per style, we normally put vars inside double quotes. so e.g. here

if [[ "${GOROOT}" == "${K8S_EDGE_GOROOT:-}" ]]; then

ixdy · 2017-01-27T18:26:32Z

hack/lib/golang.sh

+  # Temporary workaround while we have two GOROOT's (which we'll get rid of as soon as we upgrade to go1.8 for amd64 as well)
+  local GO=go
+  if [[ ${GOROOT} == ${K8S_EDGE_GOROOT:-} ]]; then
+    GO=${K8S_EDGE_GOROOT}/bin/go


"${K8s_EDGE_GOROOT}/bin/go"

ixdy · 2017-01-27T18:26:45Z

hack/lib/golang.sh

@@ -526,7 +532,7 @@ kube::golang::build_binaries_for_platform() {
    kube::log::progress "    "
    for binary in "${statics[@]:+${statics[@]}}"; do
      local outfile=$(kube::golang::output_filename_for_binary "${binary}" "${platform}")
-      CGO_ENABLED=0 go build -o "${outfile}" \
+      CGO_ENABLED=0 ${GO} build -o "${outfile}" \


"${GO}"

etc

…go executable to EDGE_GOROOT/bin/go when using the edge version

luxas · 2017-01-27T18:32:01Z

@ixdy Fixed the nits.

luxas · 2017-01-27T21:07:49Z

@ixdy Please lgtm now so we can get it in today.
Also, please shepherd it in if we are flaky at the moment.
(It's late in the evening here already, so can't retrigger tests during the night 😄)

jbeda · 2017-01-27T22:01:38Z

/approve

k8s-github-robot · 2017-01-27T22:02:54Z

[APPROVALNOTIFIER] This PR is APPROVED

The following people have approved this PR: jbeda

Needs approval from an approver in each of these OWNERS Files:

~~build/OWNERS~~ [jbeda]
~~cluster/OWNERS~~ [jbeda]
~~hack/OWNERS~~ [jbeda]
~~test/OWNERS~~ [jbeda]
~~test/e2e_node/OWNERS~~ [jbeda]

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

ixdy · 2017-01-27T22:39:02Z

/lgtm

MrHohn · 2017-01-27T23:22:00Z

@ixdy I accidently pushed addon-manager v6.3 to gcr.io two days ago and might overrided your push. Please re-push it before this merge. Sorry for the mistake :(

ixdy · 2017-01-27T23:23:45Z

I can repush after this merges. Nothing is actually using v6.3 yet, I think.

MrHohn · 2017-01-27T23:25:27Z

Oh I just notice you are not updating the salt file. Yeah sounds good, thanks!

luxas · 2017-01-28T09:04:41Z

@k8s-bot unit test this

luxas · 2017-01-28T10:28:59Z

@k8s-bot unit test this

k8s-github-robot · 2017-01-28T11:05:33Z

@k8s-bot test this [submit-queue is verifying that this PR is safe to merge]

k8s-github-robot · 2017-01-28T11:51:19Z

Automatic merge from submit-queue

luxas assigned jessfraz and ixdy Dec 17, 2016

luxas added release-note-none Denotes a PR that doesn't merit a release note. release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed release-note-none Denotes a PR that doesn't merit a release note. labels Dec 17, 2016

luxas added this to the v1.6 milestone Dec 17, 2016

luxas mentioned this pull request Dec 17, 2016

Continuously test Kubernetes against Go tip kubernetes/test-infra#1399

Closed

k8s-github-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Dec 17, 2016

luxas mentioned this pull request Dec 17, 2016

Update to Go 1.8; eval 1.8beta1 #38228

Closed

ixdy reviewed Dec 22, 2016

View reviewed changes

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Jan 6, 2017

luxas mentioned this pull request Jan 6, 2017

kube::release::package_hyperkube() fails to complete #39501

Closed

k8s-github-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jan 7, 2017

luxas mentioned this pull request Jan 8, 2017

WIP: Bump opencontainers/runc #39579

Closed

Pensu reviewed Jan 11, 2017

View reviewed changes

luxas force-pushed the update_cross_go18 branch from b4e8a23 to 758b455 Compare January 11, 2017 21:21

k8s-github-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jan 11, 2017

luxas force-pushed the update_cross_go18 branch from 758b455 to aa77111 Compare January 11, 2017 21:47

luxas mentioned this pull request Jan 11, 2017

[Part 2] Adding s390x cross-compilation support for gcr.io images in this repo #36050

Merged

luxas force-pushed the update_cross_go18 branch from aa77111 to 73d2053 Compare January 13, 2017 08:48

k8s-github-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jan 25, 2017

mkumatag mentioned this pull request Jan 27, 2017

Add linux/ppc64le to build-cross target helm/helm#1876

Merged

luxas added 2 commits January 27, 2017 20:03

Remove the patched golang from the kube-cross image and upgrade to go…

04bc44a

…1.8rc1 for arm and ppc64le instead. Reenable the ppc64le builds

Upgrade go version in Makefiles to 1.7, use qemu 2.7, armel => armhf …

8400660

…and goarm=6 => goarm=7 and use go 1.7.4

luxas force-pushed the update_cross_go18 branch 2 times, most recently from db48b8b to 3229c5e Compare January 27, 2017 18:20

k8s-github-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jan 27, 2017

ixdy reviewed Jan 27, 2017

View reviewed changes

Workaround that we have two GOROOTs; we have to set the path for the …

6789d4e

…go executable to EDGE_GOROOT/bin/go when using the edge version

luxas force-pushed the update_cross_go18 branch from 3229c5e to 6789d4e Compare January 27, 2017 18:31

Update go version to go1.8rc2 and fix the addon-manager changelog

914824c

k8s-github-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jan 27, 2017

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jan 27, 2017

k8s-github-robot merged commit 3897fc0 into kubernetes:master Jan 28, 2017

euank mentioned this pull request Feb 2, 2017

PPC builds are disabled due to a linking error #25886

Closed

spxtr mentioned this pull request Feb 17, 2017

Bump the PR cross build timeout. kubernetes/test-infra#1922

Merged

ixdy mentioned this pull request Feb 23, 2017

Automated cherry pick of #41771 #42005

Merged

Improve the multiarch situation; armel => armhf; reenable pcc64le; remove the patched golang #38926

Improve the multiarch situation; armel => armhf; reenable pcc64le; remove the patched golang #38926

Conversation

luxas commented Dec 17, 2016

k8s-reviewable commented Dec 17, 2016

luxas commented Dec 17, 2016

dims commented Dec 17, 2016

ixdy left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

luxas commented Jan 6, 2017 • edited Loading

luxas commented Jan 8, 2017 • edited Loading

luxas commented Jan 8, 2017

cyphar commented Jan 9, 2017

luxas commented Jan 9, 2017

luxas commented Jan 9, 2017

cyphar commented Jan 9, 2017

Pensu Jan 11, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ixdy commented Jan 23, 2017 • edited Loading

ixdy left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

luxas commented Jan 27, 2017

luxas commented Jan 27, 2017

jbeda commented Jan 27, 2017

k8s-github-robot commented Jan 27, 2017

ixdy commented Jan 27, 2017

MrHohn commented Jan 27, 2017

ixdy commented Jan 27, 2017

MrHohn commented Jan 27, 2017

luxas commented Jan 28, 2017

luxas commented Jan 28, 2017

k8s-github-robot commented Jan 28, 2017

k8s-github-robot commented Jan 28, 2017

luxas commented Jan 6, 2017 •

edited

Loading

luxas commented Jan 8, 2017 •

edited

Loading

Pensu Jan 11, 2017 •

edited

Loading

ixdy commented Jan 23, 2017 •

edited

Loading