kic entrypoint: retryable update-alternatives, move earlier into execution #8826

tstromberg · 2020-07-24T15:59:21Z

Makes #8814 easier to debug - and may actually just resolve it.

Let me know if you want a separate PR to add a pristine copy of the entrypoint.

k8s-ci-robot · 2020-07-24T15:59:36Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: tstromberg

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [tstromberg]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

hack/images/kicbase.Dockerfile

TravisBuddy · 2020-07-24T20:47:48Z

Travis tests have failed

Hey @tstromberg,
Please read the following log in order to understand the failure reason.
It'll be awesome if you fix what's wrong and commit the changes.

1st Build

View build log

make test

which go-bindata || GO111MODULE=off GOBIN="/home/travis/gopath/bin" go get github.com/jteeuwen/go-bindata/...
PATH="/home/travis/gopath/bin:/home/travis/.gimme/versions/go1.14.6.linux.amd64/bin:/home/travis/bin:/home/travis/bin:/home/travis/.local/bin:/usr/local/lib/jvm/openjdk11/bin:/opt/pyenv/shims:/home/travis/.phpenv/shims:/home/travis/perl5/perlbrew/bin:/home/travis/.nvm/versions/node/v8.12.0/bin:/home/travis/.rvm/gems/ruby-2.5.3/bin:/home/travis/.rvm/gems/ruby-2.5.3@global/bin:/home/travis/.rvm/rubies/ruby-2.5.3/bin:/home/travis/gopath/bin:/home/travis/.gimme/versions/go1.11.1.linux.amd64/bin:/usr/local/maven-3.6.3/bin:/usr/local/cmake-3.12.4/bin:/usr/local/clang-7.0.0/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin:/home/travis/.rvm/bin:/home/travis/.phpenv/bin:/opt/pyenv/bin:/home/travis/.yarn/bin:/home/travis/gopath/bin" go-bindata -nomemcopy -o pkg/minikube/assets/assets.go -pkg assets deploy/addons/...
gofmt -s -w pkg/minikube/assets/assets.go
which go-bindata || GO111MODULE=off GOBIN="/home/travis/gopath/bin" go get github.com/jteeuwen/go-bindata/...
/home/travis/gopath/bin/go-bindata
PATH="/home/travis/gopath/bin:/home/travis/.gimme/versions/go1.14.6.linux.amd64/bin:/home/travis/bin:/home/travis/bin:/home/travis/.local/bin:/usr/local/lib/jvm/openjdk11/bin:/opt/pyenv/shims:/home/travis/.phpenv/shims:/home/travis/perl5/perlbrew/bin:/home/travis/.nvm/versions/node/v8.12.0/bin:/home/travis/.rvm/gems/ruby-2.5.3/bin:/home/travis/.rvm/gems/ruby-2.5.3@global/bin:/home/travis/.rvm/rubies/ruby-2.5.3/bin:/home/travis/gopath/bin:/home/travis/.gimme/versions/go1.11.1.linux.amd64/bin:/usr/local/maven-3.6.3/bin:/usr/local/cmake-3.12.4/bin:/usr/local/clang-7.0.0/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin:/home/travis/.rvm/bin:/home/travis/.phpenv/bin:/opt/pyenv/bin:/home/travis/.yarn/bin:/home/travis/gopath/bin" go-bindata -nomemcopy -o pkg/minikube/translate/translations.go -pkg translate translations/...
gofmt -s -w pkg/minikube/translate/translations.go
MINIKUBE_LDFLAGS="-X k8s.io/minikube/pkg/version.version=v1.12.1 -X k8s.io/minikube/pkg/version.isoVersion=v1.12.0 -X k8s.io/minikube/pkg/version.isoPath=minikube/iso -X k8s.io/minikube/pkg/version.gitCommitID="c3c26bc27041a8861084d477c39451af42980ac0"" ./test.sh
= make lint =============================================================
golangci/golangci-lint info checking GitHub for tag 'v1.26.0'
golangci/golangci-lint info found version: 1.26.0 for v1.26.0/linux/amd64
golangci/golangci-lint info installed out/linters/golangci-lint
ok
= go mod ================================================================
ok
= boilerplate ===========================================================
boilerplate missing: /home/travis/gopath/src/github.com/kubernetes/minikube/test/stress/stress.sh
consider running: /home/travis/gopath/src/github.com/kubernetes/minikube/hack/boilerplate/fix.sh
Makefile:274: recipe for target 'test' failed
make: *** [test] Error 8

TravisBuddy Request Identifier: ed037a00-cdee-11ea-98fb-f5dcae6af814

TravisBuddy · 2020-07-24T21:01:48Z

Travis tests have failed

Hey @tstromberg,
Please read the following log in order to understand the failure reason.
It'll be awesome if you fix what's wrong and commit the changes.

1st Build

View build log

make test

which go-bindata || GO111MODULE=off GOBIN="/home/travis/gopath/bin" go get github.com/jteeuwen/go-bindata/...
PATH="/home/travis/gopath/bin:/home/travis/.gimme/versions/go1.14.6.linux.amd64/bin:/home/travis/bin:/home/travis/bin:/home/travis/.local/bin:/usr/local/lib/jvm/openjdk11/bin:/opt/pyenv/shims:/home/travis/.phpenv/shims:/home/travis/perl5/perlbrew/bin:/home/travis/.nvm/versions/node/v8.12.0/bin:/home/travis/.rvm/gems/ruby-2.5.3/bin:/home/travis/.rvm/gems/ruby-2.5.3@global/bin:/home/travis/.rvm/rubies/ruby-2.5.3/bin:/home/travis/gopath/bin:/home/travis/.gimme/versions/go1.11.1.linux.amd64/bin:/usr/local/maven-3.6.3/bin:/usr/local/cmake-3.12.4/bin:/usr/local/clang-7.0.0/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin:/home/travis/.rvm/bin:/home/travis/.phpenv/bin:/opt/pyenv/bin:/home/travis/.yarn/bin:/home/travis/gopath/bin" go-bindata -nomemcopy -o pkg/minikube/assets/assets.go -pkg assets deploy/addons/...
gofmt -s -w pkg/minikube/assets/assets.go
which go-bindata || GO111MODULE=off GOBIN="/home/travis/gopath/bin" go get github.com/jteeuwen/go-bindata/...
/home/travis/gopath/bin/go-bindata
PATH="/home/travis/gopath/bin:/home/travis/.gimme/versions/go1.14.6.linux.amd64/bin:/home/travis/bin:/home/travis/bin:/home/travis/.local/bin:/usr/local/lib/jvm/openjdk11/bin:/opt/pyenv/shims:/home/travis/.phpenv/shims:/home/travis/perl5/perlbrew/bin:/home/travis/.nvm/versions/node/v8.12.0/bin:/home/travis/.rvm/gems/ruby-2.5.3/bin:/home/travis/.rvm/gems/ruby-2.5.3@global/bin:/home/travis/.rvm/rubies/ruby-2.5.3/bin:/home/travis/gopath/bin:/home/travis/.gimme/versions/go1.11.1.linux.amd64/bin:/usr/local/maven-3.6.3/bin:/usr/local/cmake-3.12.4/bin:/usr/local/clang-7.0.0/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin:/home/travis/.rvm/bin:/home/travis/.phpenv/bin:/opt/pyenv/bin:/home/travis/.yarn/bin:/home/travis/gopath/bin" go-bindata -nomemcopy -o pkg/minikube/translate/translations.go -pkg translate translations/...
gofmt -s -w pkg/minikube/translate/translations.go
MINIKUBE_LDFLAGS="-X k8s.io/minikube/pkg/version.version=v1.12.1 -X k8s.io/minikube/pkg/version.isoVersion=v1.12.0 -X k8s.io/minikube/pkg/version.isoPath=minikube/iso -X k8s.io/minikube/pkg/version.gitCommitID="c3c26bc27041a8861084d477c39451af42980ac0"" ./test.sh
= make lint =============================================================
golangci/golangci-lint info checking GitHub for tag 'v1.26.0'
golangci/golangci-lint info found version: 1.26.0 for v1.26.0/linux/amd64
golangci/golangci-lint info installed out/linters/golangci-lint
ok
= go mod ================================================================
ok
= boilerplate ===========================================================
boilerplate missing: /home/travis/gopath/src/github.com/kubernetes/minikube/test/stress/stress.sh
consider running: /home/travis/gopath/src/github.com/kubernetes/minikube/hack/boilerplate/fix.sh
Makefile:274: recipe for target 'test' failed
make: *** [test] Error 8

TravisBuddy Request Identifier: e1f004b0-cdf0-11ea-98fb-f5dcae6af814

medyagh · 2020-07-25T01:14:09Z

@tstromberg 28 failures with this new image
full log can be seen in the artifact that u can download from https://github.com/kubernetes/minikube/pull/8826/checks?check_run_id=908356949

attached is the text fail log
testout.txt

---------------- 28 Failures :( ----------------------------
[
  "TestOffline/group/docker",
  "TestOffline/group/crio",
  "TestOffline/group/containerd",
  "TestErrorSpam",
  "TestJSONOutput/serial/DistinctCurrentSteps",
  "TestJSONOutput/serial/IncreasingCurrentSteps",
  "TestMissingContainerUpgrade",
  "TestNetworkPlugins/group/calico/Start",
  "TestNetworkPlugins/group/custom-weave/Start",
  "TestNetworkPlugins/group/kindnet/Start",
  "TestNetworkPlugins/group/auto/DNS",
  "TestNetworkPlugins/group/enable-default-cni/DNS",
  "TestNetworkPlugins/group/kubenet/DNS",
  "TestNetworkPlugins/group/bridge/DNS",
  "TestStartStop/group/crio/serial/FirstStart",
  "TestStartStop/group/containerd/serial/FirstStart",
  "TestStartStop/group/crio/serial/DeployApp",
  "TestStartStop/group/containerd/serial/DeployApp",
  "TestStartStop/group/crio/serial/SecondStart",
  "TestStartStop/group/containerd/serial/SecondStart",
  "TestStartStop/group/crio/serial/UserAppExistsAfterStop",
  "TestStartStop/group/containerd/serial/UserAppExistsAfterStop",
  "TestStartStop/group/crio/serial/AddonExistsAfterStop",
  "TestStartStop/group/containerd/serial/AddonExistsAfterStop",
  "TestStartStop/group/crio/serial/VerifyKubernetesImages",
  "TestStartStop/group/crio/serial/Pause",
  "TestStartStop/group/containerd/serial/VerifyKubernetesImages",
  "TestStartStop/group/containerd/serial/Pause"
]
-------------------------------------------------------
*** 28 Failed ***

I also noticed the following tests take a lot of time:

medyagh · 2020-07-25T01:23:09Z

for cio first start:

failed to set bridge addr: could not add IP address to "cni0": permission denied

for containerd I see this error


I0724 22:19:37.408877  335425 exit.go:58] WithError(failed to start node)=startup failed: wait 6m0s for node: waiting for apps_running: expected k8s-apps: missing components: kube-dns called from:
	goroutine 1 [running]:
	runtime/debug.Stack(0x0, 0x0, 0x100000000000000)
		/home/jenkins/actions-runner/_work/_tool/go/1.14.6/x64/src/runtime/debug/stack.go:24 +0x9d
	k8s.io/minikube/pkg/minikube/exit.WithError(0x1ba7c56, 0x14, 0x1ebf200, 0xc00090f1e0)
		/home/jenkins/actions-runner/_work/minikube/minikube/pkg/minikube/exit/exit.go:58 +0x34
	k8s.io/minikube/cmd/minikube/cmd.runStart(0x2cd0820, 0xc0003f6620, 0x2, 0xe)
		/home/jenkins/actions-runner/_work/minikube/minikube/cmd/minikube/cmd/start.go:206 +0x505
	github.com/spf13/cobra.(*Command).execute(0x2cd0820, 0xc0003f6540, 0xe, 0xe, 0x2cd0820, 0xc0003f6540)
		/home/jenkins/go/pkg/mod/github.com/spf13/[email protected]/command.go:846 +0x29d
	github.com/spf13/cobra.(*Command).ExecuteC(0x2ccf860, 0x0, 0x1, 0xc000a1a5f0)
		/home/jenkins/go/pkg/mod/github.com/spf13/[email protected]/command.go:950 +0x349
	github.com/spf13/cobra.(*Command).Execute(...)
		/home/jenkins/go/pkg/mod/github.com/spf13/[email protected]/command.go:887
	k8s.io/minikube/cmd/minikube/cmd.Execute()
		/home/jenkins/actions-runner/_work/minikube/minikube/cmd/minikube/cmd/root.go:106 +0x72c
	main.main()
		/home/jenkins/actions-runner/_work/minikube/minikube/cmd/minikube/main.go:71 +0x11f
	W0724 22:19:37.409094  335425 out.go:249] failed to start node: startup failed: wait 6m0s for node: waiting for apps_running: expected k8s-apps: missing components: kube-dns
	* 
	X failed to start node: startup failed: wait 6m0s for node: waiting for apps_running: expected k8s-apps: missing components: kube-dns
	* 
	* minikube is exiting due to an error. If the above message is not useful, open an issue:
	  - https://github.com/kubernetes/minikube/issues/new/choose
** /stderr **

medyagh · 2020-07-25T01:26:04Z

for containerd first start I see

I0724 22:19:37.408877  335425 exit.go:58] WithError(failed to start node)=startup failed: wait 6m0s for node: waiting for apps_running: expected k8s-apps: missing components: kube-dns called from:
	goroutine 1 [running]:
	runtime/debug.Stack(0x0, 0x0, 0x100000000000000)
		/home/jenkins/actions-runner/_work/_tool/go/1.14.6/x64/src/runtime/debug/stack.go:24 +0x9d
	k8s.io/minikube/pkg/minikube/exit.WithError(0x1ba7c56, 0x14, 0x1ebf200, 0xc00090f1e0)
		/home/jenkins/actions-runner/_work/minikube/minikube/pkg/minikube/exit/exit.go:58 +0x34
	k8s.io/minikube/cmd/minikube/cmd.runStart(0x2cd0820, 0xc0003f6620, 0x2, 0xe)
		/home/jenkins/actions-runner/_work/minikube/minikube/cmd/minikube/cmd/start.go:206 +0x505
	github.com/spf13/cobra.(*Command).execute(0x2cd0820, 0xc0003f6540, 0xe, 0xe, 0x2cd0820, 0xc0003f6540)
		/home/jenkins/go/pkg/mod/github.com/spf13/[email protected]/command.go:846 +0x29d
	github.com/spf13/cobra.(*Command).ExecuteC(0x2ccf860, 0x0, 0x1, 0xc000a1a5f0)
		/home/jenkins/go/pkg/mod/github.com/spf13/[email protected]/command.go:950 +0x349
	github.com/spf13/cobra.(*Command).Execute(...)
		/home/jenkins/go/pkg/mod/github.com/spf13/[email protected]/command.go:887
	k8s.io/minikube/cmd/minikube/cmd.Execute()
		/home/jenkins/actions-runner/_work/minikube/minikube/cmd/minikube/cmd/root.go:106 +0x72c
	main.main()
		/home/jenkins/actions-runner/_work/minikube/minikube/cmd/minikube/main.go:71 +0x11f
	W0724 22:19:37.409094  335425 out.go:249] failed to start node: startup failed: wait 6m0s for node: waiting for apps_running: expected k8s-apps: missing components: kube-dns
	* 
	X failed to start node: startup failed: wait 6m0s for node: waiting for apps_running: expected k8s-apps: missing components: kube-dns
	* 
	* minikube is exiting due to an error. If the above message is not useful, open an issue:
	  - https://github.com/kubernetes/minikube/issues/new/choose
** /stderr **

entrypoint: retryable update-alternatives, move earlier into execution

4124e50

tstromberg requested a review from medyagh July 24, 2020 15:59

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jul 24, 2020

k8s-ci-robot requested a review from blueelvis July 24, 2020 15:59

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 24, 2020

tstromberg changed the title ~~entrypoint: retryable update-alternatives, move earlier into execution~~ kic entrypoint: retryable update-alternatives, move earlier into execution Jul 24, 2020

Merge branch 'master' into longer-container-retry

b273111

medyagh reviewed Jul 24, 2020

View reviewed changes

hack/images/kicbase.Dockerfile Outdated Show resolved Hide resolved

Fix final RUN command continuation

0e50aa4

tstromberg mentioned this pull request Jul 27, 2020

Add entrypoint to repo, move docker image from hacks/to deploy/ #8849

Closed

medyagh mentioned this pull request Jul 28, 2020

kic entrypoint: retryable update-alternatives, move earlier into execution #8864

Merged

medyagh closed this Jul 28, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

kic entrypoint: retryable update-alternatives, move earlier into execution #8826

kic entrypoint: retryable update-alternatives, move earlier into execution #8826

tstromberg commented Jul 24, 2020 •

edited

Loading

k8s-ci-robot commented Jul 24, 2020

TravisBuddy commented Jul 24, 2020

TravisBuddy commented Jul 24, 2020

medyagh commented Jul 25, 2020 •

edited

Loading

medyagh commented Jul 25, 2020 •

edited

Loading

medyagh commented Jul 25, 2020

kic entrypoint: retryable update-alternatives, move earlier into execution #8826

kic entrypoint: retryable update-alternatives, move earlier into execution #8826

Conversation

tstromberg commented Jul 24, 2020 • edited Loading

k8s-ci-robot commented Jul 24, 2020

TravisBuddy commented Jul 24, 2020

Travis tests have failed

1st Build

TravisBuddy Request Identifier: ed037a00-cdee-11ea-98fb-f5dcae6af814

TravisBuddy commented Jul 24, 2020

Travis tests have failed

1st Build

TravisBuddy Request Identifier: e1f004b0-cdf0-11ea-98fb-f5dcae6af814

medyagh commented Jul 25, 2020 • edited Loading

medyagh commented Jul 25, 2020 • edited Loading

medyagh commented Jul 25, 2020

tstromberg commented Jul 24, 2020 •

edited

Loading

medyagh commented Jul 25, 2020 •

edited

Loading

medyagh commented Jul 25, 2020 •

edited

Loading