Skip to content
This repository has been archived by the owner on May 12, 2021. It is now read-only.

install: Add minikube instructions #445

Merged
merged 2 commits into from
Jul 19, 2019

Conversation

grahamwhaley
Copy link
Contributor

Add a doc on how to setup and install Kata into Minikube.

@grahamwhaley grahamwhaley requested a review from a team as a code owner April 24, 2019 17:20
@grahamwhaley
Copy link
Contributor Author

/cc @krsna1729

@grahamwhaley
Copy link
Contributor Author

@krsna1729 @GabyCT - feedback rolled in thanks. Updated.
but, when I did a dry run through with a fresh minikube install at my desk, launching the example deployment/pod gets stuck in the ContainerCreating status, and there is no qemu running on the node!
I wonder if something has changed maybe in the kata 1.6.2 release (which kata-deploy installed on my node) that is showing some incompatibility with minikube. I'm going to have to debug it (which will be good for my kata-k8s debug skillset).

@grahamwhaley
Copy link
Contributor Author

@jodh-intel @jcvenegas - oh, Travis gave me a:

ERROR: shell code in file 'install/minikube-installation-guide.md' is not valid

Hmm, shame it did not give me details - and, what are the rules - do all lines have to be executable in the CI/travis system? That no doubt won't work for this minikube code (no minikube installed for a start) - I'm prone to flip all the bash entries to sh entries to avoid the check/execution - unless you can see something obvious and easy to fix?

@jodh-intel
Copy link
Contributor

@grahamwhaley - The rules are in https://github.com/kata-containers/documentation/blob/master/Documentation-Requirements.md#code-blocks.

The problem with your file is that you are using bash code blocks and intermingling commands and command output. If you want this file to be executable, you will need to remove all command output from those blocks. If you really want to show commands and output (not generally recommended), use a ```sh code block instead as they will be ignored.

@grahamwhaley
Copy link
Contributor Author

Right, I don't think we can realistically execute these bash blocks unless we had some pre-scripts that set up the machine to have kvm and the docker block kvm2 driver installed and minikube itself, which this doc does not cover - I'll flip them all to sh blocks.

@jodh-intel
Copy link
Contributor

Ack. It's a shame though as I was hoping we could offer an automated install alternative to Docker. Maybe that can come later?

@grahamwhaley
Copy link
Contributor Author

It might. discussed with @egernst yesterday if we can add a minikube ci setup/test for kata-deploy testing at least - we might get the script to auto-install and setup minikube from that.

@jodh-intel
Copy link
Contributor

I'd almost prefer we wait until we can do that since once it's a valid auto-installation option, we can test (and continue to test) that doc to guarantee it doesn't go stale as these things have a habit of doing on their own ;)

@GabyCT
Copy link
Collaborator

GabyCT commented Apr 26, 2019

@grahamwhaley thanks for the changes

@gabibeyer
Copy link

gabibeyer commented Apr 30, 2019

When attempting to follow the instructions I came across this bug too. The pod was failing at sandbox creation, and when I ran a kubectl describe pods I got Failed create pod sandbox: rpc error: code = Unknown desc = container create failed: Failed to add filter for index 4 : no such file or directory".

Talked with @grahamwhaley and started doing a little bit of debugging. The first step was getting the last known good version up and running. So I did an scp of the artifacts from 1.6.0 (the version closest to the time it was last working 4/2/19) to minikube, and it worked. 1.6.1 started a container as well, so it seems to be something in 1.6.2 that is breaking it. Going to keep seeing what I can find, but in the meantime, should we open a new issue for this?

@amshinde
Copy link
Member

@gabibeyer Just happened to have a look at your error message, that does come from the network code:
https://github.com/kata-containers/runtime/blob/63e1c440a1a77d940af2b8b130249514657b90b3/virtcontainers/network.go#L846

@grahamwhaley Is that the same error you are seeing during the setup?

@wilsonianb
Copy link

I just so happened to be trying these instructions (from the gist) today and got the same error deploying the example:
Failed create pod sandbox: rpc error: code = Unknown desc = container create failed: Failed to add filter for index 4 : no such file or directory
Could this be the 1.6.2 change causing it?
kata-containers/runtime#1502

@grahamwhaley
Copy link
Contributor Author

@amshinde yep, same error. @wilsonianb could be! Minikube is running a v4.15 kernel and a rootfs constructed from a buildroot. It has a config.gz in the fs, which I had a look in - afaict it has all the netlink filtering enabled. @amshinde , any clues on what to look for then? I suspect an ip a from inside minikube might help?...

$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether f0:b9:23:8a:c2:18 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.207/24 brd 192.168.122.255 scope global dynamic eth0
       valid_lft 3359sec preferred_lft 3359sec
    inet6 fe80::f2b9:23ff:fe8a:c218/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether c0:23:26:2f:70:8a brd ff:ff:ff:ff:ff:ff
    inet 192.168.39.221/24 brd 192.168.39.255 scope global dynamic eth1
       valid_lft 3358sec preferred_lft 3358sec
    inet6 fe80::c223:26ff:fe2f:708a/64 scope link 
       valid_lft forever preferred_lft forever
4: sit0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
    link/sit 0.0.0.0 brd 0.0.0.0
5: mybridge: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1460 qdisc noqueue state UP group default qlen 1000
    link/ether 0a:58:0a:01:00:01 brd ff:ff:ff:ff:ff:ff
    inet 10.1.0.1/16 scope global mybridge
       valid_lft forever preferred_lft forever
    inet6 fe80::70e3:99ff:fe6d:d2ae/64 scope link 
       valid_lft forever preferred_lft forever
6: veth285ea917@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1460 qdisc noqueue master mybridge state UP group default 
    link/ether 26:29:dc:7b:2f:7c brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::2429:dcff:fe7b:2f7c/64 scope link 
       valid_lft forever preferred_lft forever
7: vethcf6c1fd5@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1460 qdisc noqueue master mybridge state UP group default 
    link/ether ee:3d:ff:66:a3:b1 brd ff:ff:ff:ff:ff:ff link-netnsid 1
    inet6 fe80::ec3d:ffff:fe66:a3b1/64 scope link 
       valid_lft forever preferred_lft forever
8: veth175a9042@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1460 qdisc noqueue master mybridge state UP group default 
    link/ether e2:e2:c6:a6:49:ab brd ff:ff:ff:ff:ff:ff link-netnsid 2
    inet6 fe80::e0e2:c6ff:fea6:49ab/64 scope link 
       valid_lft forever preferred_lft forever

Let us know what to look for (or, of course, you can make a minikube by following the doc in this PR ;-) ).

And then... to confirm - if you run up the minikube and kata, but before you launch the example pod/deployment, minikube ssh into minikube, sudo and go to /opt/kata/share/defaults/kata-containers and edit the configuration-qemu.toml. Flip the default networking from tcfilter to macvtap, and then you can launch the kata deployment/container! :-)

OK, let's see if we can figure out why tcfilter does not work for Minikube, and then either:

  • if we have a bug, fix it
  • or, maybe we can dynamically detect lack of tcfilter support and back out to use macvtap instead

@grahamwhaley
Copy link
Contributor Author

I'm sticking a DNM on here until we figure out what to do about the tc networking not working under minikube as is.

@grahamwhaley
Copy link
Contributor Author

@amshinde - I had a probe around (not that I'm sure what I'm looking for ;-) ) to see if anything obvious jumped out in the minikube about why tc filter add seems to be failing.

I did notice that minikube does not appear to have the macvtap or macvlan modules - do we need either/both of those to use tc?

I did also spot some error/warnings in the dmesg, but I'm not convinced they are the problem, but will post here just in case:

[ 2940.796857] eth0: Caught tx_queue_len zero misconfig
[ 2940.841446] mybridge: port 4(veth3039814e) entered disabled state

It might just be co-incidence that we get an error on index 4, and this is showing an error on port 4.
If there is anything you want me to try, let me know.
I might get the strength to see what it takes to build minikube, just in case ;-)

@grahamwhaley
Copy link
Contributor Author

@amshinde OK, I think the MACVTAP and other module missing may be red herring - I think they are not modules, but built into the minikube kernel.
Also, that kernel eth0 error looks like it is benignly fixed up in the kernel, so probably also not relevant.
I had a dig around, and it looks to me like our vendor of the netlink library we use is very very old. I may try a local update/build/test of that - I see there was a bug a while back on some of its internal functions that failed if there were many network items live (due to a buffer size restriction). Worth an update/spin I think for a start.

@grahamwhaley
Copy link
Contributor Author

@amshinde OK, I did a vendor update for the netlink module (if you do that, watch out for vishvananda/netlink#357 btw if you fail to compile...) - and the same fault still happens :-(

@grahamwhaley
Copy link
Contributor Author

@amshinde in case it helps, I added a few logger lines into network.go and dumped some of the attrs and tapattrs info, just in case anything leapt out. Sorry for the format, this is go's interpretation of the data structures:

time="2019-05-10T16:39:49Z" level=debug msg="setupTCFiltering()" arch=amd64 command=create container=2261fbf40e765e0d007c8297ee1980f6332bb3f86f77626104c4c2893dff700b name=kata-runtime pid=17175 source=virtcontainers subsystem=network
time="2019-05-10T16:39:49Z" level=info msg="TC: attrs" arch=amd64 attrs="&{4 1460 0 eth0 0a:58:0a:01:00:9c up|broadcast|multicast 69699 160 0 <nil>  0xc000443168 0 0xc00025ff00 ether <nil> up 0 0 0 65536 65535 []}" command=create container=2261fbf40e765e0d007c8297ee1980f6332bb3f86f77626104c4c2893dff700b name=kata-runtime pid=17175 source=virtcontainers subsystem=network
time="2019-05-10T16:39:49Z" level=info msg="TC: tapattrs" arch=amd64 command=create container=2261fbf40e765e0d007c8297ee1980f6332bb3f86f77626104c4c2893dff700b name=kata-runtime pid=17175 source=virtcontainers subsystem=network tapattrs="&{5 1500 1000 tap0_kata 7e:48:18:6c:7e:43 broadcast|multicast 4098 0 0 <nil>  0xc000442be4 0 0xc00025fe40 ether <nil> down 0 0 0 65536 65535 []}"
time="2019-05-10T16:39:49Z" level=debug msg="addRedirectTCFilter()" arch=amd64 command=create container=2261fbf40e765e0d007c8297ee1980f6332bb3f86f77626104c4c2893dff700b name=kata-runtime pid=17175 source=virtcontainers subsystem=network
time="2019-05-10T16:39:49Z" level=info msg="TC: src:dest" arch=amd64 command=create container=2261fbf40e765e0d007c8297ee1980f6332bb3f86f77626104c4c2893dff700b dest=5 name=kata-runtime pid=17175 source=virtcontainers src=4 subsystem=network
time="2019-05-10T16:39:49Z" level=error msg="Error bridging virtual endpoint" arch=amd64 command=create container=2261fbf40e765e0d007c8297ee1980f6332bb3f86f77626104c4c2893dff700b error="Failed to add filter for index 4 : no such file or directory" name=kata-runtime pid=17175 source=virtcontainers subsystem=network

If you need me to add more logging, let me know (or throw me a diff/patch)...

@amshinde
Copy link
Member

@grahamwhaley finally taking a look at this, this fell out of sight!
So the error is this "Failed to add filter for index 4 : no such file or directory"
I am inclined to think this is because of some missing kernel configs. Can you check if you have the kernel configs related to tc enabled in the minikube kernel? These would be CONFIG_NET_CLS_ACT, CONFIG_NET_INGRESS and CONFIG_NET_EGRESS.

@grahamwhaley
Copy link
Contributor Author

@amshinde - I will get to look at this, but in case you want to beat me to it, I think the minikube kernel config is over at https://github.com/kubernetes/minikube/blob/master/deploy/iso/minikube-iso/configs/minikube_defconfig

@grahamwhaley
Copy link
Contributor Author

Heh, oh, that link to the minikube repo is to the buildroot config... here is the one to the kernel config file:
https://github.com/kubernetes/minikube/blob/master/deploy/iso/minikube-iso/board/coreos/minikube/linux_defconfig
but, I'm not 100% sure that is the one that is used, and maybe it also picks up some default settings upon build?... So, logging into my local minikube, I can grab /proc/config.gz and have a look in there. I'm attaching a version here.
minikube_config.txt

From that file:

  • CONFIG_NET_CLS_ACT=y
  • CONFIG_NET_INGRESS=y
  • CONFIG_NET_OUTGRESS=y

All those look to be set @amshinde . Any more ideas? or, we can start trying to add more debug (but I have a feeling ultimately the bottom level netlink library we are using is just running into the default fairly anonymous not found error coming back out from the kernel :-(

Oh, if you want to re-create, just follow the minikube doc guide attached to this PR with any Kata > v1.6.2 (so the kata-deploy will pick that up for you and should fail as is today)

@GabyCT
Copy link
Collaborator

GabyCT commented Jun 4, 2019

/test

@grahamwhaley grahamwhaley changed the title install: Add minikube instructions WIP: install: Add minikube instructions Jun 4, 2019
@grahamwhaley
Copy link
Contributor Author

I've stuck a WIP in the title, as I have some edit/updates to do to this I believe before we land.
@jodh-intel @egernst - I did find though that there are no WIP or do not merge labels in the label list now we've enabled auto-magic-global-lable-thingy it seems. Is that intentional? Am I missing something?, as our WIP checker looks for those ;-)

@jodh-intel jodh-intel added the wip Work in Progress (PR incomplete - needs more work or rework) label Jun 5, 2019
@jodh-intel
Copy link
Contributor

@grahamwhaley - yep, we seemed to have lost those labels in the switch to the new set (amazing how these things pass through review eh? :)

I've now re-added those labels to all repos and raised the following to update the master database of labels:

Copy link
Contributor

@jodh-intel jodh-intel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @grahamwhaley. Some backticks changes required I think but also, it does look to me like all these commands could be made non-interactive. If so, we could make this document testable.


## Prerequisites

This installation guide has only been verified under a `Minikube` `Linux`* installation, using the
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not clear why we need backticks for these two? If they are project or product names, they can just be left capitalised I think, but use backticks for command names. Also, I guess drop the * as discussed on my runtime PR?

This installation guide may not work for `MacOS` installations of `Minikube`, due to the lack of
nested virtualization support on that platform.

This installation guide has not been tested on a `Windows` installation.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Againk, backticks?

This installation guide has only been verified under a `Minikube` `Linux`* installation, using the
[kvm2](https://github.com/kubernetes/minikube/blob/master/docs/drivers.md#kvm2-driver) driver.

This installation guide may not work for `MacOS` installations of `Minikube`, due to the lack of
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Backticks?


| what | why |
| ---- | --- |
| --vm-driver kvm2 | The host VM driver |
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please can you add backticks to all these options?

>
> | what | why |
> | ---- | --- |
> | --feature-gates=RuntimeClass=true | Kata needs to use the RuntimeClass k8s feature |
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing backticks.

Check you have virtualization enabled inside your `Minikube`:

```sh
minikube ssh
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • Missing $ prompt?
  • Could this become something like?:
    $ minikube ssh egrep 'vmx|svm' /proc/cpuinfo

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure. We have to add "'s or it goes horribly wrong though:

$ minikube ssh "egrep 'vmx|svm' /proc/cpuinfo"

I would have loved to add the --color option in there as well to make it pop out, but Minikube is based on busybox, so does not have it :-(



```sh
$ kubectl get nodes
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could this become something like?:

$ kubectl get nodes | grep -E "\<minikube\>" | grep -E "\<Ready\>"

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we were automating a spin wait then maybe, but I think minikube waits until the cluster is notionally up and Ready before it returns to the prompt anyway. Thus, if it failed to start and configure you'd get something like:

The connection to the server localhost:8080 was refused - did you specify the right host or port?

but when it works you see the fairly obvious:

NAME       STATUS   ROLES    AGE   VERSION
minikube   Ready    master   99m   v1.14.3

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, so kubectl will block on any command until it can talk to the cluster? OOI I wonder if there is either a formal is-the-cluster-up-yet command or a timeout / async option? If the cluster never comes up, does get nodes error / timeout or just hang forever?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

errors out immediately if the cluster is not up (as it cannot connect to the socket), or returns an answer (even if that is 'no nodes ready yet') otherwise afaik

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yuck. Oh well...

to check it is running:

```sh
$ kubectl get pods
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could this become something like?:

$ kubectl get pods | grep -- -apache-kata | grep -E "\<Running\>"

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sold. It looks OK on my machine as my distro by default aliases grep to use colour - but, I think having the whole output of kubectl get pods is more normal and useful in a k8s scenario, it's more what the user expects I would think.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Surely there must be some way to detect the pod is running without having to manually eyeball the output though? Maybe a --json option or something to make parsing reliable?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there is - I'll update to kubectl rollout status deployment php-apache-kata-qemu


```sh
$ minikube ssh
$ ps -ef | fgrep qemu-system
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could this and the uname command become two minikube ssh ... command instead?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sure. found a pgrep string command that does a nice job.

inside the `Kata` VM.

```sh
$ kubectl get pods
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could this become something like?:

$ pod=$(kubectl get pods | grep -E "\<Running\>" | grep kata- | head -1)
$ kernel=$(kubectl exec -ti "$pod" uname -r)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

similar to above, will replace with:

podname=$(kubectl get pods -o=name | fgrep php-apache-kata-qemu | sed 's?pod/??')
kubectl exec ${podname} -- uname -a

Copy link
Contributor Author

@grahamwhaley grahamwhaley left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated and pushed.
Updates applied.
I also tested the latest Minikube (v1.1.1) with the latest Kata from kata-deploy (v1.7.1), and it now works fine.

The full command is therefore:

```sh
$ minikube start --vm-driver kvm2 --memory 6144 --network-plugin=cni --enable-default-cni --container-runtime=cri-o --bootstrapper=kubeadm
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Explained in the table above - as Kata has 2G default pod RAM sizes, but the Minikube default is only 2G, so v.likely we cannot launch a single default Kata inside a default Minikube without extending the RAM allocation.

and the `kvm_intel` driver, you can perform the check by:

```sh
$ cat /sys/module/kvm_intel/parameters/nested
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the big issue with testability is our reliance on the external docker kvm2 driver (where we link out to the installation instructions), and the fact that drive has some issues as a native installation on some distros.
I'll clean up all the other bits and update the PR, and then see if I can work out how do do a 'docs execution' locally and peek at how hard it would be to make it runnable.
We would really like to have minikube in the CI, particuarly for kata-deploy, but I'm not sure if we can realistically use this doc to help with that.



```sh
$ kubectl get nodes
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we were automating a spin wait then maybe, but I think minikube waits until the cluster is notionally up and Ready before it returns to the prompt anyway. Thus, if it failed to start and configure you'd get something like:

The connection to the server localhost:8080 was refused - did you specify the right host or port?

but when it works you see the fairly obvious:

NAME       STATUS   ROLES    AGE   VERSION
minikube   Ready    master   99m   v1.14.3

Check you have virtualization enabled inside your `Minikube`:

```sh
minikube ssh
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure. We have to add "'s or it goes horribly wrong though:

$ minikube ssh "egrep 'vmx|svm' /proc/cpuinfo"

I would have loved to add the --color option in there as well to make it pop out, but Minikube is based on busybox, so does not have it :-(

```sh
$ minikube ssh
$ cd /opt/kata
# If the following directory is not yet created and populated, wait a short while and
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@krsna1729 heh, sleep 100. sigh. hmm, I'm not over the moon with that ;-)
Let me see if there is some way to check kata-deploy has entered its infinity sleep state (which is what it does when it completes). Maybe we need to add some magic to kata-deploy to create a 'I am done' marker at the end of its work loop...

```sh
$ minikube ssh
$ cd /opt/kata
# If the following directory is not yet created and populated, wait a short while and
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@krsna1729 @jodh-intel @amshinde @egernst Well, best I can come up with to check if kata-deploy has finished its work is:

podname=$(kubectl -n kube-system get pods -o=name | fgrep kata-deploy | sed 's?pod/??')
active=$(kubectl -n kube-system exec ${podname} -- ps -ef | fgrep infinity)

Better than guessing I guess - but, not great, and only really works in the single node case - does not scale out well.
I had a quick look at daemonsets - no obvious in-built way to manage 'status' - I think it would have to be built on some other infra method, and have the kata-deploy pods push status out to something else.

For the minikube docs, I'll update to the above ps sleep infinity hack, and note this only works for single node clusters.

to check it is running:

```sh
$ kubectl get pods
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sold. It looks OK on my machine as my distro by default aliases grep to use colour - but, I think having the whole output of kubectl get pods is more normal and useful in a k8s scenario, it's more what the user expects I would think.


```sh
$ minikube ssh
$ ps -ef | fgrep qemu-system
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sure. found a pgrep string command that does a nice job.

inside the `Kata` VM.

```sh
$ kubectl get pods
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

similar to above, will replace with:

podname=$(kubectl get pods -o=name | fgrep php-apache-kata-qemu | sed 's?pod/??')
kubectl exec ${podname} -- uname -a

@grahamwhaley
Copy link
Contributor Author

/test

@grahamwhaley grahamwhaley removed the wip Work in Progress (PR incomplete - needs more work or rework) label Jun 11, 2019
| what | why |
| ---- | --- |
| `--vm-driver kvm2` | The host VM driver |
| `--memory 6144` | Allocate sufficient memory, as Kata Containers default to 1 or 2Gb |
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: You could sort the options alphabetically.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I could, but I think as they are they keep related items (particularly the cni ones for instance) together. They also align somewhat with the order in the upstream Minikube docs iirc.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If they group logically, you could add a new column for that to make it clear (and then sort by that grouping column and sub-sort by the what column).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if the only way to get the PR landed is to sort them .... table now sorted.

This installation guide may not work for MacOS installations of Minikube, due to the lack of
nested virtualization support on that platform.

This installation guide has not been tested on a Windows installation.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

> **Note:** This example deploys Kata Containers using CRI-O. The same procedure should also work
> for a `containerd` based Kubernetes installation, but is not covered in this document.

> **Note:** Prior to Minikube/Kubernetes v1.14, the beta `RuntimeClass` feature also needed enabling with
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two notes next to each other, so I'd turn them into a list:

Notes:

  • item one.
  • item two.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. and for the new pair of notes earlier on. let's see if I got the formatting right..

$ minikube start --vm-driver kvm2 --memory 6144 --network-plugin=cni --enable-default-cni --container-runtime=cri-o --bootstrapper=kubeadm
```

> **Note:** For Kata Containers >v1.6.2, the now default `tcfilter` networking of Kata Containers
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • Can you expand > and < into words for clarity?
  • s/did not work for/does not work with/

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done



```sh
$ kubectl get nodes
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, so kubectl will block on any command until it can talk to the cluster? OOI I wonder if there is either a formal is-the-cluster-up-yet command or a timeout / async option? If the cluster never comes up, does get nodes error / timeout or just hang forever?

```sh
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
minikube Ready master 78m v1.13.4
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know you are trying to be helpful by providing command output throughout this doc. But I believe the best practice is to avoid showing any command output unless absolutely necessary; commands have a nasty habit of changing their output format so showing output that is different to what the user sees is worse than not showing output. I've tried to allude to that at the bottom of https://github.com/kata-containers/documentation/blob/master/Documentation-Requirements.md#code-blocks.

Maybe @klynnrif can comment on this.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jodh-intel I agree. In general, if you can avoid showing output you should avoid it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

dropped the output and added a relevant genericised sentence on what to look for.


```sh
$ podname=$(kubectl -n kube-system get pods -o=name | fgrep kata-deploy | sed 's?pod/??')
# If 'sleep infinity' is running, the pod has completed its work.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would probably be worth referencing https://github.com/kata-containers/packaging/blob/master/kata-deploy/scripts/kata-deploy.sh as that is what (currently) runs the sleep.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done. reworded. moved the comment out of the code block.

There are a couple of ways to verify it is running with Kata Containers. Nominally, it
should be hard to tell - the idea of Kata Containers is that your pod will run inside a VM,
but look and feel just like it would as a normal software pod. In
theory, you should not be able to tell your pod is running as a Kata Containers container.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These two sentences are essentially saying the same thing. I'd consider dropping the "Nominally, " sentence.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done


## Wrapping up

This guide should have shown a relatively easy way to get a Minikube up with Kata Containers installed.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could this be simplified to the following?:

This guide has shown an easy way to setup a Minikube with Kata Containers.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done


This guide should have shown a relatively easy way to get a Minikube up with Kata Containers installed.
Be aware, this is only a small single node Kubernetes cluster running under a nested virtualization setup.
It will have limitations, but, as a first introduction to Kata Containers, and how to install it under Kubernetes,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

s/It will have/It has/

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rephrased a bit

## Introduction

[Minikube](https://kubernetes.io/docs/setup/minikube/) is an easy way to try out a Kubernetes (k8s)
cluster locally. It utilises running a single node Kubernetes stack in a local VM.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

s/utilises running/creates/ ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

[Minikube](https://kubernetes.io/docs/setup/minikube/) is an easy way to try out a Kubernetes (k8s)
cluster locally. It utilises running a single node Kubernetes stack in a local VM.

[Kata Containers](https://github.com/kata-containers) can be installed into a Minikube cluster using
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe turn this around and make it prescriptive:

Install Kata Containers in a Minikube cluster using kata-deploy.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think prescriptive works in the Introduction - this is describing what can be done, not what to do or how to do it. Later in the 'installing Kata Containers' section is/can be prescriptive.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed.

install/minikube-installation-guide.md Outdated Show resolved Hide resolved
install/minikube-installation-guide.md Outdated Show resolved Hide resolved
The full command is therefore:

```sh
$ minikube start --vm-driver kvm2 --memory 6144 --network-plugin=cni --enable-default-cni --container-runtime=cri-o --bootstrapper=kubeadm
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm still not clear on this magic number - why/how is 6144 "sufficient"?

install/minikube-installation-guide.md Outdated Show resolved Hide resolved
install/minikube-installation-guide.md Outdated Show resolved Hide resolved
install/minikube-installation-guide.md Outdated Show resolved Hide resolved
@grahamwhaley grahamwhaley changed the title WIP: install: Add minikube instructions install: Add minikube instructions Jul 11, 2019
@grahamwhaley grahamwhaley added wip Work in Progress (PR incomplete - needs more work or rework) and removed wip Work in Progress (PR incomplete - needs more work or rework) labels Jul 11, 2019
Copy link
Contributor

@jodh-intel jodh-intel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for updating @grahamwhaley. A bunch more comments.

@klynnrif - ptal.

The full command is therefore:

```sh
$ minikube start --vm-driver kvm2 --memory 6144 --network-plugin=cni --enable-default-cni --container-runtime=cri-o --bootstrapper=kubeadm
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I still don't think this is explained clearly enough in the document but let's see what other think...



```sh
$ kubectl get nodes
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yuck. Oh well...


```sh
# No color option, as minikube is running busybox
$ minikube ssh "egrep 'vmx|svm' /proc/cpuinfo"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about a compromise:

$ egrep -o -m 1 'vmx|svm' /proc/cpuinfo

You can then explain that if the command above returns any output, the system is usable. That's better than a number and imho one heck of a lot better than the rest of the noise in that file which users shouldn't care about (and which just adds noise to our docs imho).

install/minikube-installation-guide.md Outdated Show resolved Hide resolved
Y
```

If your system does not report `Y` from the `nested` parameter, then details on how
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is perfect as the paragraph explains the output of the shell block above. But that means you can drop the command output since, as confirmed by the doc team, we should avoid displaying command output unless absolutely necessary (see https://github.com/kata-containers/documentation/blob/master/Documentation-Requirements.md#code-blocks).

There are still quite a few examples of command output in this doc.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤷‍♂️ nuked.

nested virtualization support on that platform.
> - This instal lation guide has not been tested on a Windows installation.

Before commencing installatin, it is strongly recommended you read the
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • typo: "installation".
  • nit: double space.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

| what | why |
| ---- | --- |
| `--vm-driver kvm2` | The host VM driver |
| `--memory 6144` | Allocate sufficient memory, as Kata Containers default to 1 or 2Gb |
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If they group logically, you could add a new column for that to make it clear (and then sort by that grouping column and sub-sort by the what column).

install/minikube-installation-guide.md Outdated Show resolved Hide resolved
install/minikube-installation-guide.md Outdated Show resolved Hide resolved
Copy link
Contributor Author

@grahamwhaley grahamwhaley left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

repushed

install/minikube-installation-guide.md Outdated Show resolved Hide resolved
install/minikube-installation-guide.md Outdated Show resolved Hide resolved

```sh
# No color option, as minikube is running busybox
$ minikube ssh "egrep 'vmx|svm' /proc/cpuinfo"
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

went with -c - I'd thought the count was just the retval, but is also on stdout, so is OK to explain.

install/minikube-installation-guide.md Outdated Show resolved Hide resolved
nested virtualization support on that platform.
> - This instal lation guide has not been tested on a Windows installation.

Before commencing installatin, it is strongly recommended you read the
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

| what | why |
| ---- | --- |
| `--vm-driver kvm2` | The host VM driver |
| `--memory 6144` | Allocate sufficient memory, as Kata Containers default to 1 or 2Gb |
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if the only way to get the PR landed is to sort them .... table now sorted.

The full command is therefore:

```sh
$ minikube start --vm-driver kvm2 --memory 6144 --network-plugin=cni --enable-default-cni --container-runtime=cri-o --bootstrapper=kubeadm
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've added an extra Note: to try and help.

Y
```

If your system does not report `Y` from the `nested` parameter, then details on how
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤷‍♂️ nuked.

Copy link

@klynnrif klynnrif left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scrubbed for grammar, structure, and flow. Sentence rewrites suggested to keep an active voice. Thanks!

install/minikube-installation-guide.md Outdated Show resolved Hide resolved
install/minikube-installation-guide.md Outdated Show resolved Hide resolved
install/minikube-installation-guide.md Outdated Show resolved Hide resolved

Alternatively, and for other architectures, the Kata Containers built in
[`kata-check`](https://github.com/kata-containers/runtime#hardware-requirements)
command can be used *inside Minikube* once Kata has been installed, to check for compatability.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Spelling error "compatability" => compatibility

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

install/minikube-installation-guide.md Outdated Show resolved Hide resolved
install/minikube-installation-guide.md Outdated Show resolved Hide resolved

## Enabling Kata Containers

> **Note:** This step is only required for Minikube/Kubernetes versions <= 1.13. Since version

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested rewrite to stay active:
Minikube/Kubernetes versions <= 1.13 only requires this step.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done, but I moved the 'only' earlier:

Only Minikube/Kubernetes versions <= 1.13 require this step.

install/minikube-installation-guide.md Show resolved Hide resolved
$ kubectl apply -f kata-qemu-runtimeClass.yaml
```

Kata Containers should now be installed and enabled in the Minikube cluster.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested rewrite to stay active:
The Kata Containers installation process should be complete and enabled in the Minikube cluster.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

install/minikube-installation-guide.md Outdated Show resolved Hide resolved
Copy link
Contributor Author

@grahamwhaley grahamwhaley left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@klynnrif - changes applied and pushed. thx

install/minikube-installation-guide.md Outdated Show resolved Hide resolved
install/minikube-installation-guide.md Outdated Show resolved Hide resolved
install/minikube-installation-guide.md Outdated Show resolved Hide resolved

Alternatively, and for other architectures, the Kata Containers built in
[`kata-check`](https://github.com/kata-containers/runtime#hardware-requirements)
command can be used *inside Minikube* once Kata has been installed, to check for compatability.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

## Setting up Minikube

To enable Kata Containers under Minikube, you need to add a few configuration options to the
default Minikube setup. This is easily accomplished, as Minikube supports them on the setup commandline.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

$ kubectl apply -f kata-qemu-runtimeClass.yaml
```

Kata Containers should now be installed and enabled in the Minikube cluster.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

install/minikube-installation-guide.md Outdated Show resolved Hide resolved

Launch a container that is defined to run on Kata Containers. The enabling is configured by
the following lines in the YAML file. See the Kubernetes
[Runtime Class Documentaiton](https://kubernetes.io/docs/concepts/containers/runtime-class/#usage)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

$ kubectl apply -f test-deploy-kata-qemu.yaml
```

An Apache PHP pod should be deployed with the Kata Containers runtime. This may take a few
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've moved the reference to Apache PHP to the previous sentence to avoid having to try and keep the post-tense reference.


An Apache PHP pod should be deployed with the Kata Containers runtime. This may take a few
moments if the container image needs to be pulled down into the cluster. Progress can be
checked using:
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

install/minikube-installation-guide.md Outdated Show resolved Hide resolved
install/minikube-installation-guide.md Outdated Show resolved Hide resolved
[kvm2](https://github.com/kubernetes/minikube/blob/master/docs/drivers.md#kvm2-driver) driver.

> **Notes:**
> - This installation guide may not work for MacOS installations of Minikube, due to the lack of
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"MacOS" isn't in the dictionary I'm afraid ;) To correct that, all you need to do is add MacOS/B (exactly as I've written it) in one of the spell checkers data files, say one of:

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so, open another PR in another repo, sigh....
and, looks like it is macOS (capitalisation).
OK, let me go make another PR...
Over at kata-containers/tests#1826
I'll see if that lands, or if we need to add a depends on this PR for it.

install/minikube-installation-guide.md Outdated Show resolved Hide resolved
install/minikube-installation-guide.md Outdated Show resolved Hide resolved
This guide has shown an easy way to setup Minikube with Kata Containers.
Be aware, this is only a small single node Kubernetes cluster running under a nested virtualization setup.
As such, it has limitations, but as a first introduction to Kata Containers, and how to install it under Kubernetes,
it should suffice for initial learning and experimentation.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice. Is there an obvious "next step" (or "further information") you could point users at?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm, not one that I can immediately think of. Next step would be to pick your route to a 'full k8s install', and you'd still use the above kube-deploy style install probably. I think there are too many possibilities to choose one to suggest.

[Minikube](https://kubernetes.io/docs/setup/minikube/) is an easy way to try out a Kubernetes (k8s)
cluster locally. It utilises running a single node Kubernetes stack in a local VM.

[Kata Containers](https://github.com/kata-containers) can be installed into a Minikube cluster using
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed.

Graham Whaley added 2 commits July 18, 2019 13:28
Add instructions of how to setup and install Kata under
Minikue.

Fixes: kata-containers#444

Signed-off-by: Graham Whaley <[email protected]>
Add Minikube to the list of 'cloud' providers installation instructions.
Whilst there, order the list alphabetically.

Signed-off-by: Graham Whaley <[email protected]>
Copy link
Contributor Author

@grahamwhaley grahamwhaley left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated, pushed.
We may want to wait for the 'macOS' spellcheck PR to land before we spin a CI again.

install/minikube-installation-guide.md Outdated Show resolved Hide resolved
install/minikube-installation-guide.md Outdated Show resolved Hide resolved
install/minikube-installation-guide.md Outdated Show resolved Hide resolved
[kvm2](https://github.com/kubernetes/minikube/blob/master/docs/drivers.md#kvm2-driver) driver.

> **Notes:**
> - This installation guide may not work for MacOS installations of Minikube, due to the lack of
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so, open another PR in another repo, sigh....
and, looks like it is macOS (capitalisation).
OK, let me go make another PR...
Over at kata-containers/tests#1826
I'll see if that lands, or if we need to add a depends on this PR for it.

install/minikube-installation-guide.md Outdated Show resolved Hide resolved
This guide has shown an easy way to setup Minikube with Kata Containers.
Be aware, this is only a small single node Kubernetes cluster running under a nested virtualization setup.
As such, it has limitations, but as a first introduction to Kata Containers, and how to install it under Kubernetes,
it should suffice for initial learning and experimentation.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm, not one that I can immediately think of. Next step would be to pick your route to a 'full k8s install', and you'd still use the above kube-deploy style install probably. I think there are too many possibilities to choose one to suggest.

Copy link
Contributor

@jodh-intel jodh-intel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @grahamwhaley.

Copy link

@klynnrif klynnrif left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm - thanks!

@egernst egernst merged commit 3655b4c into kata-containers:master Jul 19, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.