podman error executing kata-runtime

[sdeoras]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description
I am encountering error when running pod via kata-runtime.

Steps to reproduce the issue:

1. Make sure runc, runsc and kata-runtime are working well with either k8s/cri-o or k8s/containerd

2. Install podman version podman version 1.3.2-dev and define runtimes in libpod.conf and set cgroupfs as cgroup manager.

3. Run ubuntu image as sudo and test different runtimes.

Describe the results you received:

sudo podman --runtime=runc run --rm ubuntu date  # success
sudo podman --runtime=runsc run --rm ubuntu date # success
sudo podman --runtime=kata-runtime run --rm ubuntu date # fails with following error

Error: container create failed: rpc error: code = Internal desc = Could not run process: container_linux.go:345: starting container process caused "process_linux.go:424: container init caused \"rootfs_linux.go:58: mounting \\\"tmpfs\\\" to rootfs \\\"/run/kata-containers/shared/containers/0d4c7c15ce58a37d2f5f3ebc070b21704d861091812f0658bda8fd59f5b415e0/rootfs\\\" at \\\"/dev\\\" caused \\\"rootfs_linux.go:251: tmpcopyup: failed to setup tmpdir caused \\\\\\\"mkdir /tmp/runctop461258244: read-only file system\\\\\\\"\\\"\""
: internal libpod error

Describe the results you expected:
Expected no error with kata-runtime

Additional information you deem important (e.g. issue happens only occasionally):

kata-runtime --version
kata-runtime  : 1.7.0
   commit   : <<unknown>>
   OCI specs: 1.0.1-dev

podman version 0.12.1.2 works fine. The issue is seen with podman version 1.3.2-dev

Output of podman version:

podman version 1.3.2-dev

Output of podman info --debug:

debug:
  compiler: gc
  git commit: ""
  go version: go1.10.4
  podman version: 1.3.2-dev
host:
  BuildahVersion: 1.9.0-dev
  Conmon:
    package: 'conmon: /usr/libexec/crio/conmon'
    path: /usr/libexec/crio/conmon
    version: 'conmon version , commit: '
  Distribution:
    distribution: ubuntu
    version: "18.04"
  MemFree: 14088249344
  MemTotal: 15762862080
  OCIRuntime:
    package: 'cri-o-runc: /usr/bin/runc'
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc6
      spec: 1.0.1-dev
  SwapFree: 0
  SwapTotal: 0
  arch: amd64
  cpus: 4
  hostname: crio-1
  kernel: 4.15.0-1032-gcp
  os: linux
  rootless: true
  uptime: 1h 15m 53.93s (Approximately 0.04 days)
registries:
  blocked: null
  insecure: null
  search:
  - registry.access.redhat.com
  - registry.fedoraproject.org
  - quay.io
  - docker.io
store:
  ConfigFile: /home/sdeoras/.config/containers/storage.conf
  ContainerStore:
    number: 0
  GraphDriverName: vfs
  GraphOptions: null
  GraphRoot: /home/sdeoras/.local/share/containers/storage
  GraphStatus: {}
  ImageStore:
    number: 0
  RunRoot: /tmp/1002
  VolumePath: /home/sdeoras/.local/share/containers/storage/volumes

Additional environment details (AWS, VirtualBox, physical, etc.):
GCP nodes running nested virtualization enabled Ubuntu 18.04 image.

[amshinde]

This bug seems to have manifested in this commit: 70beb57

But its mainly caused due to tmpcopyup option passed for the /dev mount.
This is most likely due to a change on the kata side causing the VM rootfs to be passed read-only (we seemed to have missed handling the /tmp mount correctly). We need to fix this on the Kata side.

Question to the podman maintainers, why is tmpcopyup option passed for the /dev mount?
I would think this option would not be required for /dev.

[giuseppe]

@amshinde I guess that is done so that devices specified in the image are still present in the container. But I don't see any good reasons for having devices created in the image itself, so I think it should be fine if the runtime simply ignores it.

[sdeoras]

@amshinde, thank you. Should I file a new issue for kata-runtime and close this one?

[amshinde]

@sdeoras I have created an issue for this on Kata repo and linked it. Thanks!

[amshinde]

@giuseppe Kata needs to fix this for supporting the tmpcopyup option in any case.
But would it make sense for podman to not pass the tmpcopyup for /dev, like you mentioned, I dont see any reason for creating device files in the image.

[mheon]

This is definitely a bug on our end - we should not be setting tmpcopyup on /dev/. I'll special-case it.

[mheon]

#3238 to resolve

[amshinde]

Thanks @mheon !

[mheon]

Setting tmpcopyup there is a bug - I'll get a fix out

…

On Thu, May 30, 2019, 07:48 Saurabh Deoras ***@***.***> wrote: @amshinde <https://github.com/amshinde>, thank you. Should I file a new issue for kata-runtime and close this one? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#3229>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AB3AOCAP4QH6K6UDCUHNQ6TPX65JBANCNFSM4HQOVOLQ> .