minikube-iso: add cri-o runtime #1998

vbatts · 2017-09-22T15:42:25Z

https://github.com/kubernetes-incubator/cri-o

Updated the runc version to its latest master commit.
Got crio into the automounter to get off the tmpfs

This feature bubbles up to the minikube command by reusing the
--container-runtime= flag, by enabling the value of "crio" (minikube start --container-runtime=crio), while
the flags/config passed to localkube are more like k8s
(--container-runtime=remote --remote-runtime-endpoint=/var/run/crio.sock)

This is mostly ready for review. It is still lacking having
--insecure-registry plumbed through, but for now the policy.json is
open.

Will rebase on crio master once cri-o/cri-o#940 is merged.

Signed-off-by: Vincent Batts [email protected]

minikube-bot · 2017-09-22T15:43:01Z

Can one of the admins verify this patch?

codecov-io · 2017-09-22T16:14:53Z

Codecov Report

Merging #1998 into master will decrease coverage by 0.03%.
The diff coverage is 0%.

@@            Coverage Diff             @@
##           master    #1998      +/-   ##
==========================================
- Coverage   29.85%   29.81%   -0.04%     
==========================================
  Files          77       77              
  Lines        4763     4769       +6     
==========================================
  Hits         1422     1422              
- Misses       3161     3167       +6     
  Partials      180      180

Impacted Files	Coverage Δ
pkg/localkube/localkube.go	`51.92% <ø> (ø)`	⬆️
pkg/localkube/kubelet.go	`0% <0%> (ø)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d7bb7c3...2788714. Read the comment docs.

r2d4 · 2017-09-22T16:49:36Z

@minikube-bot ok to test

vbatts · 2017-09-22T17:43:07Z

sorry. I had to update the checksum. Please retest.

r2d4 · 2017-09-22T18:20:04Z

@minikube-bot retest this please

r2d4 · 2017-09-23T19:30:03Z

@minikube-bot retest this please

vbatts · 2017-09-25T12:42:26Z

interesting that the cross build fails on a mkdir, i'll look into why that fails

vbatts · 2017-09-25T14:20:45Z

updated and rebased

vbatts · 2017-09-25T17:08:19Z

@r2d4 once I get the dust settled for this addition, a next step will be adding logic to the provisioner about not starting services that are unneeded for particular --container-runtime=. Opinions about that?

r2d4 · 2017-09-25T17:30:36Z

@vbatts I'm guessing you mean the docker service primarily? I think that should be fine and I'm open to whatever changes we need to make, although you might encounter some difficultly with the leaky abstractions in that a few things might wait on the docker endpoint to be available.

We're also planning to hopefully move away from localkube in the future and use kubeadm, for which we already have the --bootstrapper kubeadm flag for. I plan on sending a PR soon to make sure that all the configuration we set for localkube gets passed through kubeadm similarly. I can make sure this works for that bootstrapper, but just something to keep in mind.

vbatts · 2017-09-25T17:35:09Z

@r2d4 i expect leaky abstractions to be going away. But while testing this, it is both docker and rkt services are started.

vbatts · 2017-09-25T18:51:38Z

ohman go1.7. Let me work on getting go1.8 into the build 🙏

r2d4 · 2017-09-25T18:58:00Z

Ah, actually we have a dockerized build, if you add this

https://github.com/kubernetes/minikube/blob/master/hack/jenkins/minikube_cross_build_and_upload.sh#L38
This line should be changed to:
IN_DOCKER=1 make out/minikube.iso

the docker image will build with go 1.8

vbatts · 2017-09-25T19:00:49Z

How does that work for the jenkins job?

r2d4 · 2017-09-25T19:01:45Z

Jenkins just calls that script for the cross build job

vbatts · 2017-09-25T19:02:50Z

ah. lemme give it a try

vbatts · 2017-09-25T19:04:44Z

while that will show that it builds in such an environment, that line change would affect the ISO release process, no?

r2d4 · 2017-09-25T19:17:30Z

Ah, you're right. I thought our release process built the ISO in docker also, but it doesn't. The makefile rules around the ISO could use some refactoring.

diff --git a/Makefile b/Makefile
index a3f7e48b4..3abad1a3e 100755
--- a/Makefile
+++ b/Makefile
@@ -288,7 +288,7 @@ $(ISO_BUILD_IMAGE): deploy/iso/minikube-iso/Dockerfile
        @echo "$(@) successfully built"
 
 .PHONY: release-iso
-release-iso: minikube_iso checksum
+release-iso: out/minikube.iso checksum
        gsutil cp out/minikube.iso gs://$(ISO_BUCKET)/minikube-$(ISO_VERSION).iso
        gsutil cp out/minikube.iso.sha256 gs://$(ISO_BUCKET)/minikube-$(ISO_VERSION).iso.sha256
 
diff --git a/hack/jenkins/build_iso.sh b/hack/jenkins/build_iso.sh
index 32cc67131..6d76be548 100644
--- a/hack/jenkins/build_iso.sh
+++ b/hack/jenkins/build_iso.sh
@@ -20,4 +20,4 @@
 #      ISO_VERSION = the suffix for the iso (i.e. minikube-$(ISO_VERSION).iso)
 
 set -e
-${ARGS} make release-iso
+${ARGS} IN_DOCKER=1 make release-iso

That patch should fix the release job also

vbatts · 2017-09-26T17:02:23Z

hmm. Still appears to be failing due to go1.7 compiler

r2d4 · 2017-09-26T17:29:53Z

Ok, suspiciously there isn't even a go tool chain in the docker image we use to build the iso. I'm not even sure why we use a different image to build the iso, and I don't think that image is being build everytime.

I'll just go ahead and update the build slave to go 1.8, since that seems like the path of least resistance here. I'll update this issue when its upgraded.

r2d4 · 2017-09-26T17:36:27Z

upgraded

@minikube-bot retest this please

vbatts · 2017-09-26T18:17:59Z

I'll remove that docker variable when I get to my desk

https://github.com/kubernetes-incubator/cri-o Updated the runc version to its latest master commit. Got crio into the automounter to get off the tmpfs This feature bubbles up to the minikube command by reusing the `--container-runtime=` flag, by enabling the value of "`crio`" (`minikube start --container-runtime=crio`), while the flags/config passed to localkube are more like k8s (`--container-runtime=remote --remote-runtime-endpoint=/var/run/crio.sock`) This is mostly ready for review. It is still lacking having --insecure-registry plumbed through, but for now the policy.json is open. Signed-off-by: Vincent Batts <[email protected]>

vbatts · 2017-09-27T13:14:08Z

i've removed that IN_DOCKER=1 make out/minikube.iso commit.

Also, with cri-o/cri-o#940 merged, I've switched back to cri-o master commit (not pointing to my branch).

Not sure why the windows environment failed, but we'll see if it works this next time.

r2d4 · 2017-09-27T16:37:22Z

the windows machine is in a bad state and we've had some problems with it in the past, since the tests passed on most of the other platforms, I think it should be fine.

r2d4 · 2017-09-27T16:47:54Z

After this is merged I'll create an integration test using the --container-runtime=crio flag, since theres a little bit of setup on our side for adding a new integration test.

Pulled it down locally and ran it, seems to be working fine.

Is this expected on kubectl describe node:

Container Runtime Version: runc://Unknown?

vbatts · 2017-09-27T17:03:51Z

cool. Like I said, the --insecure-registry still needs to be plumbed through. And wouldn't mind getting the provisioner logic to result in fewer services. Those can be done in future PRs.

…

On Wed, Sep 27, 2017 at 12:51 PM Gerard Braad ***@***.***> wrote: ***@***.**** approved this pull request. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1998 (review)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAEF6WqnkTOpuwq4DIkbV8Sl03BRnUSlks5smn0XgaJpZM4Pg4vc> .

r2d4 · 2017-09-27T16:52:51Z

deploy/iso/minikube-iso/package/crio-bin/crio.conf

+image_volumes = "mkdir"
+
+# insecure_registries is used to skip TLS verification when pulling images.
+insecure_registries = [


We might need to write this file dynamically like we do the docker systemd unit to pass through things like insecure registries. I think this is fine for now though.

once i saw that there were dynamic service files generated, I thought the same thing.

r2d4 · 2017-09-27T17:05:38Z

Thanks for working through this!

vbatts · 2017-09-27T17:24:55Z

@r2d4 when I run:

kubectl run httpd --image=httpd
kubectl describe pod httpd

I see

[...]
    Container ID:       runc://1d753ae46099f0a2dca86b1d658d913bcabbc9d48c96b4d2bd56b8857481c413
[...]

vbatts · 2017-09-27T17:30:04Z

oh sorry, you said describe node, yes i see that same thing. I'll check into that.

rhatdan · 2017-09-27T17:49:18Z

👍

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Sep 22, 2017

vbatts mentioned this pull request Sep 22, 2017

support cri-o as a container-runtime minishift/minishift#1442

Open

vbatts force-pushed the crio-bin branch from 17efce6 to 0817c66 Compare September 22, 2017 17:39

vbatts force-pushed the crio-bin branch 2 times, most recently from d0110e6 to 71b5e6d Compare September 25, 2017 14:20

vbatts force-pushed the crio-bin branch 2 times, most recently from 508fd6f to 4d4331f Compare September 25, 2017 16:56

vbatts force-pushed the crio-bin branch from 4d4331f to a3edae6 Compare September 25, 2017 18:35

vbatts force-pushed the crio-bin branch 3 times, most recently from fe882ac to 55194b1 Compare September 26, 2017 14:32

vbatts force-pushed the crio-bin branch from 55194b1 to 987c40d Compare September 26, 2017 15:48

vbatts force-pushed the crio-bin branch from 987c40d to 2788714 Compare September 27, 2017 13:12

gbraad approved these changes Sep 27, 2017

View reviewed changes

r2d4 approved these changes Sep 27, 2017

View reviewed changes

r2d4 merged commit a90b6a4 into kubernetes:master Sep 27, 2017

vbatts deleted the crio-bin branch September 27, 2017 17:18

r2d4 mentioned this pull request Nov 2, 2017

Unable to build minikube ISO locally #1854

Closed

This was referenced Nov 18, 2017

Docs: Usage of the alternative CRI-O runtime #2207

Closed

Revamp rkt and add CRI-O as alternative runtime kubernetes/website#6371

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

minikube-iso: add cri-o runtime #1998

minikube-iso: add cri-o runtime #1998

vbatts commented Sep 22, 2017 •

edited

Loading

minikube-bot commented Sep 22, 2017

codecov-io commented Sep 22, 2017 •

edited

Loading

r2d4 commented Sep 22, 2017

vbatts commented Sep 22, 2017

r2d4 commented Sep 22, 2017

r2d4 commented Sep 23, 2017

vbatts commented Sep 25, 2017

vbatts commented Sep 25, 2017

vbatts commented Sep 25, 2017

r2d4 commented Sep 25, 2017

vbatts commented Sep 25, 2017

vbatts commented Sep 25, 2017

r2d4 commented Sep 25, 2017 •

edited

Loading

vbatts commented Sep 25, 2017

r2d4 commented Sep 25, 2017

vbatts commented Sep 25, 2017

vbatts commented Sep 25, 2017

r2d4 commented Sep 25, 2017

vbatts commented Sep 26, 2017

r2d4 commented Sep 26, 2017 •

edited

Loading

r2d4 commented Sep 26, 2017

vbatts commented Sep 26, 2017 via email •

edited by gbraad

Loading

vbatts commented Sep 27, 2017

r2d4 commented Sep 27, 2017

r2d4 commented Sep 27, 2017 •

edited

Loading

vbatts commented Sep 27, 2017 via email

r2d4 Sep 27, 2017

vbatts Sep 27, 2017

r2d4 commented Sep 27, 2017

vbatts commented Sep 27, 2017

vbatts commented Sep 27, 2017

rhatdan commented Sep 27, 2017

minikube-iso: add cri-o runtime #1998

minikube-iso: add cri-o runtime #1998

Conversation

vbatts commented Sep 22, 2017 • edited Loading

minikube-bot commented Sep 22, 2017

codecov-io commented Sep 22, 2017 • edited Loading

Codecov Report

r2d4 commented Sep 22, 2017

vbatts commented Sep 22, 2017

r2d4 commented Sep 22, 2017

r2d4 commented Sep 23, 2017

vbatts commented Sep 25, 2017

vbatts commented Sep 25, 2017

vbatts commented Sep 25, 2017

r2d4 commented Sep 25, 2017

vbatts commented Sep 25, 2017

vbatts commented Sep 25, 2017

r2d4 commented Sep 25, 2017 • edited Loading

vbatts commented Sep 25, 2017

r2d4 commented Sep 25, 2017

vbatts commented Sep 25, 2017

vbatts commented Sep 25, 2017

r2d4 commented Sep 25, 2017

vbatts commented Sep 26, 2017

r2d4 commented Sep 26, 2017 • edited Loading

r2d4 commented Sep 26, 2017

vbatts commented Sep 26, 2017 via email • edited by gbraad Loading

vbatts commented Sep 27, 2017

r2d4 commented Sep 27, 2017

r2d4 commented Sep 27, 2017 • edited Loading

vbatts commented Sep 27, 2017 via email

r2d4 Sep 27, 2017

Choose a reason for hiding this comment

vbatts Sep 27, 2017

Choose a reason for hiding this comment

r2d4 commented Sep 27, 2017

vbatts commented Sep 27, 2017

vbatts commented Sep 27, 2017

rhatdan commented Sep 27, 2017

vbatts commented Sep 22, 2017 •

edited

Loading

codecov-io commented Sep 22, 2017 •

edited

Loading

r2d4 commented Sep 25, 2017 •

edited

Loading

r2d4 commented Sep 26, 2017 •

edited

Loading

vbatts commented Sep 26, 2017 via email •

edited by gbraad

Loading

r2d4 commented Sep 27, 2017 •

edited

Loading