k8s: document how to choose which kernel/rootfs for which pod

[grahamwhaley]

Kata has the ability to select a different kernel/rootfs per-pod in k8s, but I can't find the documentation saying how to do that.
I suspect it could live in https://github.com/kata-containers/documentation/blob/master/how-to/how-to-use-k8s-with-cri-containerd-and-kata.md

Let's see if we already have that documented, and if not, add it. If so, maybe we can cross-ref the docs.

[grahamwhaley]

/cc @mythi , who was looking for this info...
/cc @sameo and @egernst who wrote and/or have set this up in the past.
I have a feeling the info might be living in a gist somewhere right now.

[grahamwhaley]

I think, for ref, the details may be in @sameo asciinema video at https://asciinema.org/a/146888
@sameo - can you confirm, will the details there still be valid you think?

[egernst]

/cc @eadamsintel -- do you have script from your demo?

[eadamsintel]

I do. It is in Google drive but once you have the kubernetes cluster setup and kata runtime added you need a yaml file annotation pointing to the custom kernel. See below.

apiVersion: v1
kind: Pod
metadata:
  annotations:
    com.github.containers.virtcontainers.KernelPath: "/home/kata/k8s/vmlinuz-4.14.61-7"
  name: nginx-kata-custom-kernel
spec:
  runtimeClassName: kata-runtime
  containers:
  - name: nginx
    image: sameo/nginx-kubecon:1.0
    ports:
    - containerPort: 8080

[mythi]

Cool, it seems the image path can also be set:

ImagePath = vcAnnotationsPrefix + "ImagePath"

(I was more interested in setting a custom image)