podman image prune should act recursively

[srcshelton]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

It's notable as other bugs have indicated that podman image ls can be very slow (although this is better nowadays). What is still noticeable is that image ls completes much faster with fewer images.

However, podman image prune appears to only remove images which were unreferenced when it was invoked, and doesn't recurse to also remove any images that the removal of the original images free up.

I wrote a quick script to look until no further images are freed, the output from a real-world run of which appears below:

Starting to prune podman images ...
1: Removed 1077 images on this pass...
2: Removed 35 images on this pass...
3: Removed 35 images on this pass...
4: Removed 35 images on this pass...
5: Removed 35 images on this pass...
6: Removed 35 images on this pass...
7: Removed 35 images on this pass...
8: Removed 35 images on this pass...
9: Removed 35 images on this pass...
10: Removed 35 images on this pass...
11: Removed 35 images on this pass...
12: Removed 35 images on this pass...
13: Removed 35 images on this pass...
14: Removed 35 images on this pass...
15: Removed 35 images on this pass...
16: Removed 35 images on this pass...
17: Removed 35 images on this pass...
18: Removed 34 images on this pass...
19: Removed 33 images on this pass...
20: Removed 33 images on this pass...
21: Removed 33 images on this pass...
22: Removed 33 images on this pass...
23: Removed 33 images on this pass...
24: Removed 33 images on this pass...
25: Removed 33 images on this pass...
26: Removed 0 images on this pass.

Removed 1902 images in total

... so, in this case, the initial podman image prune only removed about half of the prunable images. This surely isn't the intention?

Output of podman version:

Version:      2.1.1
API Version:  2.0.0
Go Version:   go1.14.7
Git Commit:   9f6d6ba0b314d86521b66183c9ce48eaa2da1de2
Built:        Tue Sep 29 16:34:31 2020
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.16.1
  cgroupManager: cgroupfs
  cgroupVersion: v2
  conmon:
    package: Unknown
    path: /usr/bin/conmon
    version: 'conmon version 2.0.21, commit: 35a2fa83022e56e18af7e6a865ba5d7165fa2a4a'
  cpus: 8
  distribution:
    distribution: gentoo
    version: unknown
  eventLogger: file
  hostname: dellr330
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 5.8.9-gentoo
  linkmode: dynamic
  memFree: 2493095936
  memTotal: 8063447040
  ociRuntime:
    name: crun
    package: Unknown
    path: /usr/bin/crun
    version: |-
      crun version 0.15-dirty
      commit: 56ca95e61639510c7dbd39ff512f80f626404969
      spec: 1.0.0
      +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /run/podman/podman.sock
  rootless: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 24785993728
  swapTotal: 25769787392
  uptime: 64h 27m 2.06s (Approximately 2.67 days)
registries:
  search:
  - docker.io
  - docker.pkg.github.com
  - quay.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 15
    paused: 0
    running: 11
    stopped: 4
  graphDriverName: overlay
  graphOptions:
    overlay.ignore_chown_errors: "false"
  graphRoot: /space/podman/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 180
  runRoot: /space/podman/run
  volumePath: /space/podman/volumes
version:
  APIVersion: 2.0.0
  Built: 1601393671
  BuiltTime: Tue Sep 29 16:34:31 2020
  GitCommit: 9f6d6ba0b314d86521b66183c9ce48eaa2da1de2
  GoVersion: go1.14.7
  OsArch: linux/amd64
  Version: 2.1.1

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

Yes

[vrothberg]

Thanks for opening the issue, @srcshelton!

I try to have a look tomorrow. There are certain code paths that have not yet been optimized but we plan to. The issue is a motivator to tackle prune :)

[vrothberg]

Removed the good-first issue label as I find it non-trivial to fix without knowing the code.

[vrothberg]

Opened #7887 to fix the issue. Note that it does not improve performance yet. There are some CPU cycles we can squeeze out but it would require some longer refactoring. It's still on the roadmap though, along with other code paths.

[srcshelton]

Just a quick note to say that while #7887 fixes this issue with podman image prune, the problem still seems to be apparent when pruning images during podman system prune!

Could this issue please be re-opened to consider this case also?

[vrothberg]

Can you share a reproducer?

The image-pruning code is shared, so it seems to be another issue.

[vrothberg]

I suggest opening a new issue, ideally with a reproducer.