Add support for using the podman service within minikube

[afbjorklund]

Currently we offer support for using the docker daemon remotely, through docker-env.

https://minikube.sigs.k8s.io/docs/tasks/docker_daemon/

It is possible to offer the same kind of support for podman-remote, using podman-env.

https://github.com/containers/libpod/blob/master/docs/tutorials/remote_client.md

---

This uses the varlink protocol over ssh, in order to start a podman socket on the VM.

Unlike Docker, there is no need to start any long-running daemon before connecting...

Currently the only supported podman commands are: load, save, build, images

There are currently some issues with running them, so use Kubernetes for that instead.

---

Any images built are immediately available for the CRI-O runtime, for rapid turnaround.

When issuing the podman-remote build command, the build context is sent to the VM.

[afbjorklund]

Will need to add some minor documentation, on how to actually use this thing.

Maybe just some basic information, that would be rather similar to the above:

eval $(minikube podman-env)

podman-remote ps

---

Maybe one should should mention that podman and crio don't share containers:

$ podman-remote ps
CONTAINER ID  IMAGE  COMMAND  CREATED  STATUS  PORTS  NAMES

Or that the remote podman does not implement image listing, that only works locally:

$ podman-remote images
Error: unable to get images: org.varlink.service.MethodNotFound

But except for those little quirks, it works fine.

$ podman-remote version
Client:
Version:            1.7.0
RemoteAPI Version:  1
Go Version:         go1.13.6
Git Commit:         b7ce1157b00af09f4a09e39b377aa3abff46ee05
Built:              Sat Jan 18 15:26:34 2020
OS/Arch:            linux/amd64

Server:
Version:            1.6.4
RemoteAPI Version:  1
Go Version:         go1.11.13
Git Commit:         5cc92849f7fc9dd734ca2fd8f3ae8830b9a7eb26
Built:              Sat Jan  4 13:49:24 2020
OS/Arch:            linux/amd64

$ podman-remote build -t hello-node minikube
STEP 1: FROM node:6.14.2
Getting image source signatures
Copying blob sha256:0c5cf711b890ca3e9df1635796c969d71320570f97bdd8d1a9e857f85ec6f33c
Copying blob sha256:3d77ce4481b119f00e53bee9b4a443469c42c224db954ddaa2e6b74cd73cd5d0
Copying blob sha256:4e3b8a1eb9144b9f3d3e22adf51949bad358dd0497d5666e4149aef9cd26211d
Copying blob sha256:ddcf13cc195143149ccefecf62ef9b8d3536e62214d30853d5f4b8b356f22ac4
Copying blob sha256:9593dc852d6b21f5d3e612cb37150a661d2f5f5136a1276466984c46c8274951
Copying blob sha256:7d2f329349632fac6046432536420d15d630b68e7c536dcb28714fadb45a8a14
Copying blob sha256:2e460d114172e58ca4ed5b0ff392d786deabf4f849a37edf99af17e0b073d2f0
Copying blob sha256:d94b1226fbf2b110040b5350345884eee744ae7bc839faebaa4eac9c01c5bff2
Copying config sha256:00165cd5d0c00321af529a74915a9a7fe5cc9759ebca8e86ad38191933f551e8
Writing manifest to image destination
Storing signatures
STEP 2: EXPOSE 8080
0bc2b5d8aebc413ff81f04937f7b0aa47b2a53541e41bf2e98793cae4261ab22
STEP 3: COPY server.js .
42f85c036f1446c4ceb01c9133d730e9e35958ec2ad8c3a1327b6ba62b088dc0
STEP 4: CMD node server.js
STEP 5: COMMIT hello-node
eff766022cfcbf7e1636850646a869c30a42ca3209b321052a577639fa72974d

---

We are supposed to use the CRI anyway, and not those various runtime hacks.

$ minikube ssh -- sudo crictl ps
CONTAINER           IMAGE                                                              CREATED             STATE               NAME                      ATTEMPT             POD ID
1e7ae854f85f5       70f311871ae12c14bd0e02028f249f933f925e4370744e4e35f706da773a8f61   4 hours ago         Running             coredns                   0                   f6db6d14d1809
d3a789d441f44       70f311871ae12c14bd0e02028f249f933f925e4370744e4e35f706da773a8f61   4 hours ago         Running             coredns                   0                   552441b30d947
215a5a574800f       7d54289267dc5a115f940e8b1ea5c20483a5da5ae5bb3ad80107409ed1400f19   4 hours ago         Running             kube-proxy                0                   c4c93767e3dfb
2159bdfb101be       303ce5db0e90dab1c5728ec70d21091201a23cdf8aeca70ab54943bbaaf0833f   4 hours ago         Running             etcd                      0                   4b038141d53dd
4decf6551cd9d       78c190f736b115876724580513fdf37fa4c3984559dc9e90372b11c21b9cad28   4 hours ago         Running             kube-scheduler            0                   39ae45cbcbbde
f98f837157308       5eb3b7486872441e0943f6e14e9dd5cc1c70bc3047efacbc43d1aa9b7d5b3056   4 hours ago         Running             kube-controller-manager   0                   728f2b33a7786
8813f5985fcfa       0cae8d5cc64c7d8fbdf73ee2be36c77fdabd9e0c7d30da0c12aedf402730bbb2   4 hours ago         Running             kube-apiserver            0                   eb0b0f1d8f416

$ minikube ssh -- sudo crictl images
IMAGE                                     TAG                 IMAGE ID            SIZE
<none>                                    <none>              0bc2b5d8aebc4       683MB
<none>                                    <none>              42f85c036f144       683MB
docker.io/kubernetesui/dashboard          v2.0.0-beta8        eb51a35975256       91.2MB
docker.io/kubernetesui/metrics-scraper    v1.0.2              3b08661dc379d       40.1MB
docker.io/library/node                    6.14.2              00165cd5d0c00       683MB
gcr.io/k8s-minikube/storage-provisioner   v1.8.1              4689081edb103       80.8MB
k8s.gcr.io/coredns                        1.6.5               70f311871ae12       41.7MB
k8s.gcr.io/etcd                           3.4.3-0             303ce5db0e90d       290MB
k8s.gcr.io/kube-addon-manager             v9.0.2              bd12a212f9dcb       84.7MB
k8s.gcr.io/kube-apiserver                 v1.17.0             0cae8d5cc64c7       173MB
k8s.gcr.io/kube-controller-manager        v1.17.0             5eb3b74868724       162MB
k8s.gcr.io/kube-proxy                     v1.17.0             7d54289267dc5       118MB
k8s.gcr.io/kube-scheduler                 v1.17.0             78c190f736b11       96MB
k8s.gcr.io/pause                          3.1                 da86e6ba6ca19       747kB
localhost/hello-node                      latest              eff766022cfcb       683MB

[afbjorklund]

Seems like there is a difference in defaults, when it comes to the intermediate layers...
By default these are saved in podman (--layers=true) and hidden in podman images.

      --layers    cache intermediate layers dur
ing build. Use BUILDAH_LAYERS environment variable to override. (default true)

$ podman-remote build -t hello-node minikube
STEP 1: FROM node:6.14.2
STEP 2: EXPOSE 8080
fc5ced900f5f2f9667d7b5b373a9329e55788548221ffe98e62016482f3167b0
STEP 3: COPY server.js .
7c117a410de104cf6594ab9d250e85467d9dac559ea6cff96ce78d4d94b51d03
STEP 4: CMD node server.js
STEP 5: COMMIT hello-node
a0485d5eb2f7476b1885094b228da8f75cf9ca1c19bd0327b8a141fb114c665a

$ minikube ssh -- sudo podman images
REPOSITORY                                TAG            IMAGE ID       CREATED              SIZE
localhost/hello-node                      latest         a0485d5eb2f7   About a minute ago   683 MB

But they can still be seen by using the --all flag, and they also show when using crictl:
(The flag to images: -a, --all Show all images (default hides intermediate images))

$ minikube ssh -- sudo podman images --all
REPOSITORY                                TAG            IMAGE ID       CREATED          SIZE
localhost/hello-node                      latest         a0485d5eb2f7   10 seconds ago   683 MB
<none>                                    <none>         7c117a410de1   11 seconds ago   683 MB
<none>                                    <none>         fc5ced900f5f   12 seconds ago   683 MB

$ minikube ssh -- sudo crictl images
IMAGE                                     TAG                 IMAGE ID            SIZE
<none>                                    <none>              fc5ced900f5f2       683MB
<none>                                    <none>              7c117a410de10       683MB
localhost/hello-node                      latest              a0485d5eb2f74       683MB

---

So if wanting something more similar to docker build, use sudo podman build --layers=false.

$ podman-remote build --layers=false -t hello-node minikube
STEP 1: FROM node:6.14.2
STEP 2: EXPOSE 8080
STEP 3: COPY server.js .
STEP 4: CMD node server.js
STEP 5: COMMIT hello-node
Getting image source signatures
Copying blob sha256:2c833f307fd8f18a378b71d3c43c575fabdb88955a2198662938ac2a08a99928
Copying blob sha256:5f349fdc9028f7edde7f8d4c487d59b3e4b9d66a367dc85492fc7a81abf57b41
Copying blob sha256:bb6d734b467e06b65c8e1066c3ea953293615137992827f4004ee860397173f3
Copying blob sha256:f3693db46abb018f72d41856ae00c0a5ca73e46606f9c8b6b942e0da9354b22a
Copying blob sha256:bf769027dbbd15097c3762263f5ea2157185be40bbd4f93bd955eb7d43536415
Copying blob sha256:8c825a971eaf367c88f7d0590ee1261c8519b676e7e3aec69dd970725be377b4
Copying blob sha256:6e650662f0e303c7f94fc26378f323b81cbad56a8dc29e4800695b6534aeb916
Copying blob sha256:aeaa1edefd60fbeec38f6844814ba3be5262779b423136c30f7dfddf203fb542
Copying blob sha256:c36bb47185bd5a2e0afb9ee9f56b0cd91f699cec86a26f30da62f9b6534c3cdd
Copying config sha256:fe3314a71f7235ed2925c1ebaa78f8d6e2cf1fb01524d0f7f30dbb25519ce176
Writing manifest to image destination
Storing signatures
fe3314a71f7235ed2925c1ebaa78f8d6e2cf1fb01524d0f7f30dbb25519ce176

Then the intermediate containers are removed, just like they are when using docker.
EDIT: Seems that I was confused when I wrote this, mixing containers and images...

$ minikube ssh -- sudo crictl images
IMAGE                                     TAG                 IMAGE ID            SIZE
localhost/hello-node                      latest              fe3314a71f723       683MB

$ docker build -t hello-node minikube
Sending build context to Docker daemon  3.584kB
Step 1/4 : FROM node:6.14.2
 ---> 00165cd5d0c0
Step 2/4 : EXPOSE 8080
 ---> Running in 893d0ea2b0e7
Removing intermediate container 893d0ea2b0e7
 ---> 85fe45254671
Step 3/4 : COPY server.js .
 ---> c0d4d0653f57
Step 4/4 : CMD node server.js
 ---> Running in 4dd1264c118b
Removing intermediate container 4dd1264c118b
 ---> 02d53de8b6ff
Successfully built 02d53de8b6ff
Successfully tagged hello-node:latest

I removed the minikube containers (and node:6.14.2) in the above output, for brevity.

[afbjorklund]

Added some docs in https://deploy-preview-6351--kubernetes-sigs-minikube.netlify.com/docs/tasks/podman_service/ (same as docker daemon)

[afbjorklund]

After the initial confusion, it seems that there is a difference in the CRI when it comes to images.

$ eval $(minikube docker-env)
$ docker build -t hello-node minikube
Sending build context to Docker daemon  3.584kB
Step 1/4 : FROM node:6.14.2
 ---> 00165cd5d0c0
Step 2/4 : EXPOSE 8080
 ---> Running in 4e3f890a7f77
Removing intermediate container 4e3f890a7f77
 ---> bf51cad45209
Step 3/4 : COPY server.js .
 ---> 8b77a274c037
Step 4/4 : CMD node server.js
 ---> Running in 49ce5bb3434f
Removing intermediate container 49ce5bb3434f
 ---> e6287fe46af9
Successfully built e6287fe46af9
Successfully tagged hello-node:latest
$ docker images --all
REPOSITORY                                                       TAG                 IMAGE ID            CREATED             SIZE
<none>                                                           <none>              bf51cad45209        14 seconds ago      660MB
<none>                                                           <none>              8b77a274c037        14 seconds ago      660MB
hello-node                                                       latest              e6287fe46af9        14 seconds ago      660MB
$ minikube ssh -- sudo crictl images
IMAGE                                                            TAG                 IMAGE ID            SIZE
hello-node                                                       latest              e6287fe46af96       660MB

Probably because Docker does not use the CRI, so it is not the same Kubernetes code path.

runtime-endpoint: unix:///run/dockershim.sock
image-endpoint: unix:///run/dockershim.sock

[afbjorklund]

Seems to be working, running podman version 1.6.5