Podman run oci-archive gives "Error: invalid reference format"

[gatoniel]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Hi,
I am supplying containers to users on a workstation. My current workflow is like this:

• Modify the Dockerfile: vim Dockerfile
• Build a container: podman build . -t myimage:1.0 --format=docker
• Then I push it to an archive in my user space: podman push myimage:1.0 oci-archive:/home/myuser/images/myimage:1.0
• Copy the archive to a readonly location for all users: sudo cp -r /home/myuser/images/myimage /mnt/datahdd/readonly/images/myimage
• a different user runs: podman run oci-archive:/mnt/datahdd/readonly/images/myimage:1.0

I am happy to hear about a better workflow, but this worked for me until today. Now the podman run gives an error: Error: invalid reference format. When I run podman run myimage:1.0 everything works fine. But podman run oci-archive:/home/myuser/images/myimage:1.0 gives me the same error.

I also tried pushing via podman push myimage:1.0 oci-archive:/home/myuser/images:myimage:1.0 and running via podman run oci-archive:/home/myuser/images:myimage:1.0, but the error stays. I also could not resolve this issue by removing the --format=docker statement in the build command.

As I said, this error occured just recently and this workflow seemed to work before. The only thing I recently changed is that I set up the gpg-agent. Might this disturb some processes? I also do not know, if I need to initialize the oci-archive path beforehand.

Steps to reproduce the issue:

1. podman pull registry.fedoraproject.org/f29/httpd

2. podman push registry.fedoraproject.org/f29/httpd oci-archive:/home/netter/administration/podman-images/httpd

3. podman run oci-archive:/home/netter/administration/podman-images/httpd

Describe the results you received:
I get:

Error: invalid reference format

Describe the results you expected:
The container should run, as if I was using the command podman run registry.fedoraproject.org/f29/httpd

Additional information you deem important (e.g. issue happens only occasionally):

I think the only thing I was changing in my setup was adding gpg-agent environment variables. In my envs I found this environment variable: OLDPWD=/home/netter/administration/podman-images

Output of podman version:

Version:      2.0.0
API Version:  1
Go Version:   go1.13.4
Built:        Thu Jan  1 01:00:00 1970
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.15.0
  cgroupVersion: v1
  conmon:
    package: conmon-2.0.18-1.el8.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.18, commit: 3f215ac67ed650ab6a44f8c036bcbd692afdc21e'
  cpus: 32
  distribution:
    distribution: '"centos"'
    version: "8"
  eventLogger: file
  hostname: ***
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 4.18.0-147.8.1.el8_1.x86_64
  linkmode: dynamic
  memFree: 141837963264
  memTotal: 270112145408
  ociRuntime:
    name: runc
    package: runc-1.0.0-15.5.el8.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc10
      commit: ffb456f24bd637cec53c611f309d89abb027b1d3
      spec: 1.0.1-dev
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  rootless: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.1-1.el8.x86_64
    version: |-
      slirp4netns version 1.1.1
      commit: bbf27c5acd4356edb97fa639b4e15e0cd56a39d5
      libslirp: 4.2.0
      SLIRP_CONFIG_VERSION_MAX: 2
  swapFree: 3010576384
  swapTotal: 4294963200
  uptime: 380h 4m 29.33s (Approximately 15.83 days)
registries: {}
store:
  configFile: /home/myuser/.config/containers/storage.conf
  containerStore:
    number: 5
    paused: 0
    running: 0
    stopped: 5
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.1.1-1.el8.x86_64
      Version: |-
        fusermount3 version: 3.2.1
        fuse-overlayfs: version 1.1.0
        FUSE library version 3.2.1
        using FUSE kernel interface version 7.26
  graphRoot: /home/myuser/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 146
  runRoot: /run/user/1000/containers
  volumePath: /home/myuser/.local/share/containers/storage/volumes
version:
  APIVersion: 1
  Built: 0
  BuiltTime: Thu Jan  1 01:00:00 1970
  GitCommit: ""
  GoVersion: go1.13.4
  OsArch: linux/amd64
  Version: 2.0.0

Package info (e.g. output of rpm -q podman or apt list podman):

Installierte Pakete
podman.x86_64        2.0.0-2.el8        @devel_kubic_libcontainers_stable
Verfügbare Pakete
podman.aarch64      2.0.0-2.el8        devel_kubic_libcontainers_stable
podman.src              2.0.0-2.el8        devel_kubic_libcontainers_stable

Additional environment details (AWS, VirtualBox, physical, etc.):
It is a physical workstation running Centos 8.

[vrothberg]

Thanks for opening the issue @gatoniel!

Podman can only create containers with images that are in the local container storage. This means, that it only supports the container-storage: storage which Podman implies if the transport isn't mentioned explicitly. Hence, oci-archive and other transports won't work.

Is your use case to set up a storage that multiple users can share? We support the concept of additional stores which @rhatdan wrote about in the following blot: https://developers.redhat.com/blog/2019/08/14/best-practices-for-running-buildah-in-a-container/

@rhatdan you wrote a more recent blog about additional stores explicitly but I fail to find it. Has it been released?

[gatoniel]

Hi,
thanks for the answers.

Podman can only create containers with images that are in the local container storage. This means, that it only supports the container-storage: storage which Podman implies if the transport isn't mentioned explicitly. Hence, oci-archive and other transports won't work.

Somehow this worked until some days ago. But then I will use a different setup.

Is your use case to set up a storage that multiple users can share? We support the concept of additional stores which @rhatdan wrote about in the following blot: https://developers.redhat.com/blog/2019/08/14/best-practices-for-running-buildah-in-a-container/

Yes, that is my use case. I read the article and will try to reproduce this on my system by adding

additionalimagestores = [
"/mnt/datahdd/readonly/container_store",
### is this safe???
"/home/myuser/.local/share/containers/storage",
]

to /etc/containers/storage.conf. But how do I now add images to the first store? Or would it also be safe, to just add my own user store? I think the workflow would be:

• podman build . -t myimage:tag
• skopeo copy containers-storage:localhost/myimage:tag containers-storage:/home/myuser/container_store/myimage:tag
• sudo cp -r /home/myuser/container_store /mnt/datahdd/readonly/container_store

I want to avoid running sudo together with skopeo or podman, to be sure that everything I run can also be run by other normal users. But the skopeo statement does not work and throws this error:

Invalid destination name containers-storage:/home/myuser/container_store/myimage:tag: error parsing named reference "/home/myuser/container_store/myimage:tag": invalid reference format

[mheon]

Can you pull the image with podman pull before running it? I suspect that we tightened the validation on the image name used by podman run, but podman pull should still work with oci-archive.

[gatoniel]

Can you pull the image with podman pull before running it? I suspect that we tightened the validation on the image name used by podman run, but podman pull should still work with oci-archive.

I could, but this would mean rewriting some other code, since the podman run command is executed within a Python tornado server (JupyterHub with podman spawner).

Another upside of the first mentioned solution with an additional storage is the disk usage, isnt it? When every user pulls the image, they need to save it in their own user storage under .local/share/containers . When I use the additional storage that would not be the case. This would save lots of disk space.

Best,
Niklas

[vrothberg]

Please scratch what I said before:

Podman can only create containers with images that are in the local container storage. This means, that it only supports the container-storage: storage which Podman implies if the transport isn't mentioned explicitly. Hence, oci-archive and other transports won't work.

For sure, Podman will pull images and given you report it has worked before, we may regressed here. I will investigate.

[vrothberg]

I can reproduce, thanks again!

[gatoniel]

Please scratch what I said before:

Podman can only create containers with images that are in the local container storage. This means, that it only supports the container-storage: storage which Podman implies if the transport isn't mentioned explicitly. Hence, oci-archive and other transports won't work.

For sure, Podman will pull images and given you report it has worked before, we may regressed here. I will investigate.

But as I understand the additional store is treated as a local container storage? I think the only problem I have is, how to copy/push from my local container storage to the additional storage.

[vrothberg]

Opened #6751 to fix the issue.

[vrothberg]

Please scratch what I said before:

Podman can only create containers with images that are in the local container storage. This means, that it only supports the container-storage: storage which Podman implies if the transport isn't mentioned explicitly. Hence, oci-archive and other transports won't work.

For sure, Podman will pull images and given you report it has worked before, we may regressed here. I will investigate.

But as I understand the additional store is treated as a local container storage? I think the only problem I have is, how to copy/push from my local container storage to the additional storage.

If our additional store is located /var/lib/mycontainers, we can populate it with the --root option of Podman, e.g., $ podman -- root /var/lib/mycontainers fedora:32. This way, the store will be mounted RW and allows for storing images.

[vrothberg]

Soon, a new blog from @rhatdan will be released on Red Hat SysAdmin which will cover this topic in greater detail.

[OZZlE]

had this error because of a trailing slash only, removed that and this error was gone :)