Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve the multiarch situation; armel => armhf; reenable pcc64le; remove the patched golang #38926

Merged
merged 4 commits into from
Jan 28, 2017

Conversation

luxas
Copy link
Member

@luxas luxas commented Dec 17, 2016

What this PR does / why we need it:

  • Improves the multiarch situation as described in Multi-architecture plan for Kubernetes #38067
  • Tries to bump to go1.8 for arm (and later enable ppc64le)
  • GOARM 6 => GOARM 7
  • Remove the golang 1.7 patch
  • armel => armhf
  • Bump QEMU version to v2.7.0

Release note:

Improve the ARM builds and make hyperkube on ARM working again by upgrading the Go version for ARM to go1.8beta2

@kubernetes/sig-testing-misc @jessfraz @ixdy @jbeda @david-mcmahon @pwittrock

@k8s-reviewable
Copy link

This change is Reviewable

@luxas luxas added release-note-none Denotes a PR that doesn't merit a release note. release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed release-note-none Denotes a PR that doesn't merit a release note. labels Dec 17, 2016
@luxas luxas added this to the v1.6 milestone Dec 17, 2016
@luxas
Copy link
Member Author

luxas commented Dec 17, 2016

The problem is here that we can't upgrade to go1.8 yet due to vendoring issues. I caught this at least:

panic: standard_init_linux.go:175: exec user process caused "no such file or directory" [recovered]
	panic: standard_init_linux.go:175: exec user process caused "no such file or directory"

goroutine 1 [running, locked to thread]:
panic(0x3360b0, 0x56e56a80)
	/usr/local/go/src/runtime/panic.go:481 +0x330
github.com/urfave/cli.HandleAction.func1(0x56e03960)
	/go/src/github.com/opencontainers/runc/Godeps/_workspace/src/github.com/urfave/cli/app.go:478 +0x328
panic(0x3360b0, 0x56e56a80)
	/usr/local/go/src/runtime/panic.go:443 +0x448
github.com/opencontainers/runc/libcontainer.(*LinuxFactory).StartInitialization.func1(0x56e035e8, 0x56dda890, 0x56e03670)
	/go/src/github.com/opencontainers/runc/Godeps/_workspace/src/github.com/opencontainers/runc/libcontainer/factory_linux.go:259 +0x13c
github.com/opencontainers/runc/libcontainer.(*LinuxFactory).StartInitialization(0x56e08e70, 0x46433cb0, 0x56e56a80)
	/go/src/github.com/opencontainers/runc/Godeps/_workspace/src/github.com/opencontainers/runc/libcontainer/factory_linux.go:277 +0x498
main.glob.func8(0x56e42320, 0x0, 0x0)
	/go/src/github.com/opencontainers/runc/main_unix.go:26 +0x58
reflect.Value.call(0x2eb600, 0x3f2948, 0x13, 0x3648b8, 0x4, 0x56e03920, 0x1, 0x1, 0x0, 0x0, ...)
	/usr/local/go/src/reflect/value.go:435 +0xeb4
reflect.Value.Call(0x2eb600, 0x3f2948, 0x13, 0x56e03920, 0x1, 0x1, 0x0, 0x0, 0x0)
	/usr/local/go/src/reflect/value.go:303 +0x84
github.com/urfave/cli.HandleAction(0x2eb600, 0x3f2948, 0x56e42320, 0x0, 0x0)
	/go/src/github.com/opencontainers/runc/Godeps/_workspace/src/github.com/urfave/cli/app.go:487 +0x230
github.com/urfave/cli.Command.Run(0x366cd8, 0x4, 0x0, 0x0, 0x0, 0x0, 0x0, 0x3d0d50, 0x51, 0x0, ...)
	/go/src/github.com/opencontainers/runc/Godeps/_workspace/src/github.com/urfave/cli/command.go:191 +0xcd0
github.com/urfave/cli.(*App).Run(0x56e3a240, 0x56dca070, 0x2, 0x2, 0x0, 0x0)
	/go/src/github.com/opencontainers/runc/Godeps/_workspace/src/github.com/urfave/cli/app.go:240 +0x84c
main.main()
	/go/src/github.com/opencontainers/runc/main.go:137 +0xbb4

Is there anyone from Google that is responsible for upgrading to go1.8 in time for v1.6?
@jessfraz or @dims or someone else?

@dims
Copy link
Member

dims commented Dec 17, 2016

@luxas here's the latest status from @jessfraz #38228 (comment)

@k8s-github-robot k8s-github-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Dec 17, 2016
Copy link
Member

@ixdy ixdy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

would it be worth fixing some of these things (e.g. moving everything to go1.7) while we're working through the go1.8 issues?

@@ -31,14 +31,14 @@ REGISTRY_TAG?=3.0.14
ARCH?=amd64
REGISTRY?=gcr.io/google_containers
GOLANG_VERSION?=1.6.3
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we bump this one too?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes I'll do it

@@ -22,13 +22,14 @@ CNI_RELEASE?=07a8a28637e97b22eb8dfe710eeae1344f69d16e
CNI_TARBALL=cni-$(ARCH)-$(CNI_RELEASE).tar.gz
CUR_DIR=$(shell pwd)
OUTPUT_DIR=$(CUR_DIR)/output
GOLANG_VERSION=1.7
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

might want to pin this to an explicit version? otherwise docker run may not fetch a newer patch release.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is there any way to make docker always pull on docker run?
I'm sure there is, but seems like I've forgotten it for the moment

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

docker build has --pull, but for run you have to docker pull then docker run - see moby/moby#13331.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we might still want to fix this. can we make this golang:1.7.4 explicitly?

@@ -25,15 +25,15 @@ ARCH?=amd64
TAG?=1.9
REGISTRY?=gcr.io/google_containers

GOLANG_VERSION=1.6
GOLANG_VERSION=1.7
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this is even used here?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nope it isn't. this was just an automatic replace. will fix

@@ -29,7 +29,7 @@ ARCH ?= amd64

ALL_ARCH = amd64 arm arm64 ppc64le s390x

GOARM=6
GOARM=7
TEMP_DIR := $(shell mktemp -d)
GOLANG_VERSION = 1.6.3
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe update this one?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes gonna

@@ -79,15 +79,12 @@ RUN export ETCD_VERSION=v3.0.14; \
&& ln -s ../src/etcd/etcd-${ETCD_VERSION}-linux-amd64/etcd /usr/local/bin/

# TODO: Remove the patched GOROOT when we have an official golang that has a working arm and ppc64le linker
ENV K8S_PATCHED_GOLANG_VERSION=1.7.4 \
ENV K8S_PATCHED_GOLANG_VERSION=go1.8beta2 \
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd kinda like to see us get away from having two separate go versions. Wonder if go1.8 will land in time for k8s 1.6?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I really hope it will. That's my intention and goal.
We have to validate it and do the hard work with the betas though

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The problem, and the reason this is wip is that k8s is not working with go1.8 yet :(

@luxas
Copy link
Member Author

luxas commented Jan 6, 2017

I think I found the root cause, we might just have to bump runc: opencontainers/runc#1116

@k8s-ci-robot k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Jan 6, 2017
@k8s-github-robot k8s-github-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jan 7, 2017
@luxas
Copy link
Member Author

luxas commented Jan 8, 2017

@runcom Even after upgrading runc I get the following when running on arm for all kube binaries on startup:

panic: standard_init_linux.go:175: exec user process caused "no such file or directory" [recovered]
	panic: standard_init_linux.go:175: exec user process caused "no such file or directory"

goroutine 1 [running, locked to thread]:
panic(0x3339c8, 0x10747480)
	/usr/local/go/src/runtime/panic.go:481 +0x330
github.com/urfave/cli.HandleAction.func1(0x1073b960)
	/go/src/github.com/opencontainers/runc/Godeps/_workspace/src/github.com/urfave/cli/app.go:478 +0x328
panic(0x3339c8, 0x10747480)
	/usr/local/go/src/runtime/panic.go:443 +0x448
github.com/opencontainers/runc/libcontainer.(*LinuxFactory).StartInitialization.func1(0x1073b5e8, 0x1071a240, 0x1073b670)
	/go/src/github.com/opencontainers/runc/Godeps/_workspace/src/github.com/opencontainers/runc/libcontainer/factory_linux.go:259 +0x13c
github.com/opencontainers/runc/libcontainer.(*LinuxFactory).StartInitialization(0x10766f30, 0xb65e5448, 0x10747480)
	/go/src/github.com/opencontainers/runc/Godeps/_workspace/src/github.com/opencontainers/runc/libcontainer/factory_linux.go:277 +0x498
main.glob.func8(0x10790320, 0x0, 0x0)
	/go/src/github.com/opencontainers/runc/main_unix.go:26 +0x58
reflect.Value.call(0x2e90f8, 0x3efd4c, 0x13, 0x362250, 0x4, 0x1073b920, 0x1, 0x1, 0x0, 0x0, ...)
	/usr/local/go/src/reflect/value.go:435 +0xeb4
reflect.Value.Call(0x2e90f8, 0x3efd4c, 0x13, 0x1073b920, 0x1, 0x1, 0x0, 0x0, 0x0)
	/usr/local/go/src/reflect/value.go:303 +0x84
github.com/urfave/cli.HandleAction(0x2e90f8, 0x3efd4c, 0x10790320, 0x0, 0x0)
	/go/src/github.com/opencontainers/runc/Godeps/_workspace/src/github.com/urfave/cli/app.go:487 +0x230
github.com/urfave/cli.Command.Run(0x364678, 0x4, 0x0, 0x0, 0x0, 0x0, 0x0, 0x3ce288, 0x51, 0x0, ...)
	/go/src/github.com/opencontainers/runc/Godeps/_workspace/src/github.com/urfave/cli/command.go:191 +0xcd0
github.com/urfave/cli.(*App).Run(0x10782240, 0x10762050, 0x2, 0x2, 0x0, 0x0)
	/go/src/github.com/opencontainers/runc/Godeps/_workspace/src/github.com/urfave/cli/app.go:240 +0x84c
main.main()
	/go/src/github.com/opencontainers/runc/main.go:137 +0xbb4

using go1.8, latest runc and running on ARM. Do you know what the cause could be?

@luxas
Copy link
Member Author

luxas commented Jan 8, 2017

cc @vmarmol @crosbymichael @hqhq and @cyphar on the above exec user process caused "no such file or directory" runc panic on all k8s binaries when running on ARM and using hyperkube (at least w/ go1.8beta2 right now)

Do you know what the cause could be? Why is this code running on init?

@cyphar
Copy link

cyphar commented Jan 9, 2017

@luxas How is runc being executed, do you know? That error is usually when you try to runc start a container where the process.args process doesn't exist in the rootfs.

@luxas
Copy link
Member Author

luxas commented Jan 9, 2017

@cyphar I believe the problem is that LinuxFactory.StartInitialization() is run somehow during the init() in the k8s Godeps. It should absolutely not run when starting the scheduler with hyperkube for example.

The problem is that I can't find how it's run at startup and why...

@luxas
Copy link
Member Author

luxas commented Jan 9, 2017

What's even worse is that we don't even vendor /go/src/github.com/opencontainers/runc/main.go:137 or /go/src/github.com/opencontainers/runc/Godeps/_workspace/src/github.com/urfave/cli/command.go:191 into k8s, so I don't really know what's going on here, seems like we have compiled k8s with more code than we need or something

@cyphar
Copy link

cyphar commented Jan 9, 2017

@luxas Does the first argument equal "init"? Inside runC we have to use init in order to handle some of the setup inside Go (namely the StartInitialization code). Also, if you're vendoring github.com/opencontainers/runc/libcontainer/nsenter you're not going to have a fun time.

&& GOROOT_FINAL=${K8S_PATCHED_GOROOT} GOROOT_BOOTSTRAP=/usr/local/go ./make.bash \
&& for platform in linux/arm; do GOOS=${platform%/*} GOARCH=${platform##*/} GOROOT=${K8S_PATCHED_GOROOT} go install std; done
&& for platform in linux/arm linux/ppc64le; do GOOS=${platform%/*} GOARCH=${platform##*/} GOROOT=${K8S_PATCHED_GOROOT} go install std; done
Copy link
Contributor

@Pensu Pensu Jan 11, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think patching is not required for golang for ppc64le with go1.8. All the changes are already upstreamed in go1.8.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR makes k8s not use a patched golang, but the name will still be K8S_PATCHED_GOROOT until k8s uses go1.8 offically

@ixdy
Copy link
Member

ixdy commented Jan 23, 2017

yeah, easy repro outside of our containerized build is to use go 1.7.4 with GOROOT set to 1.8rc2 (or rc1) and then go test -c any test package.

@k8s-github-robot k8s-github-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jan 25, 2017
…1.8rc1 for arm and ppc64le instead. Reenable the ppc64le builds
@luxas luxas force-pushed the update_cross_go18 branch 2 times, most recently from db48b8b to 3229c5e Compare January 27, 2017 18:20
@k8s-github-robot k8s-github-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jan 27, 2017
Copy link
Member

@ixdy ixdy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks like you already fixed the one main issue. minor stylistic comment, otherwise I think this basically LG. I'll do one more full pass in a little bit.

V=2 kube::log::info "Env for ${platform}: GOOS=${GOOS-} GOARCH=${GOARCH-} GOROOT=${GOROOT-} CGO_ENABLED=${CGO_ENABLED-} CC=${CC-}"
# Temporary workaround while we have two GOROOT's (which we'll get rid of as soon as we upgrade to go1.8 for amd64 as well)
local GO=go
if [[ ${GOROOT} == ${K8S_EDGE_GOROOT:-} ]]; then
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: per style, we normally put vars inside double quotes. so e.g. here

if [[ "${GOROOT}" == "${K8S_EDGE_GOROOT:-}" ]]; then

# Temporary workaround while we have two GOROOT's (which we'll get rid of as soon as we upgrade to go1.8 for amd64 as well)
local GO=go
if [[ ${GOROOT} == ${K8S_EDGE_GOROOT:-} ]]; then
GO=${K8S_EDGE_GOROOT}/bin/go
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"${K8s_EDGE_GOROOT}/bin/go"

@@ -526,7 +532,7 @@ kube::golang::build_binaries_for_platform() {
kube::log::progress " "
for binary in "${statics[@]:+${statics[@]}}"; do
local outfile=$(kube::golang::output_filename_for_binary "${binary}" "${platform}")
CGO_ENABLED=0 go build -o "${outfile}" \
CGO_ENABLED=0 ${GO} build -o "${outfile}" \
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"${GO}"

etc

…go executable to EDGE_GOROOT/bin/go when using the edge version
@luxas
Copy link
Member Author

luxas commented Jan 27, 2017

@ixdy Fixed the nits.

@luxas
Copy link
Member Author

luxas commented Jan 27, 2017

@ixdy Please lgtm now so we can get it in today.
Also, please shepherd it in if we are flaky at the moment.
(It's late in the evening here already, so can't retrigger tests during the night 😄)

@jbeda
Copy link
Contributor

jbeda commented Jan 27, 2017

/approve

@k8s-github-robot
Copy link

[APPROVALNOTIFIER] This PR is APPROVED

The following people have approved this PR: jbeda

Needs approval from an approver in each of these OWNERS Files:

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

@k8s-github-robot k8s-github-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jan 27, 2017
@ixdy
Copy link
Member

ixdy commented Jan 27, 2017

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jan 27, 2017
@MrHohn
Copy link
Member

MrHohn commented Jan 27, 2017

@ixdy I accidently pushed addon-manager v6.3 to gcr.io two days ago and might overrided your push. Please re-push it before this merge. Sorry for the mistake :(

@ixdy
Copy link
Member

ixdy commented Jan 27, 2017

I can repush after this merges. Nothing is actually using v6.3 yet, I think.

@MrHohn
Copy link
Member

MrHohn commented Jan 27, 2017

Oh I just notice you are not updating the salt file. Yeah sounds good, thanks!

@luxas
Copy link
Member Author

luxas commented Jan 28, 2017

@k8s-bot unit test this

1 similar comment
@luxas
Copy link
Member Author

luxas commented Jan 28, 2017

@k8s-bot unit test this

@k8s-github-robot
Copy link

@k8s-bot test this [submit-queue is verifying that this PR is safe to merge]

@k8s-github-robot
Copy link

Automatic merge from submit-queue

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. lgtm "Looks good to me", indicates that a PR is ready to be merged. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.