podman system prune: error removing pod conmon cgroup: EBUSY

[edsantiago]

This is not a new flake, it dates back to June or earlier. Possibly I didn't file it because we have a ton of other EBUSY flakes and I thought this was the same one. Symptom seems to be:

time="2021-10-12T16:12:29Z" level=error msg="Deleting pod XXX 
    cgroup /libpod_parent/XXX: 
    remove /sys/fs/cgroup/misc/libpod_parent/XXX: 
    remove /sys/fs/cgroup/misc/libpod_parent/XXX/conmon:
    remove /sys/fs/cgroup/misc/libpod_parent/XXX/conmon: device or resource busy"

(newlines added for legibility).

Podman prune [It] podman system prune - pod,container stopped

• fedora-33 : int podman fedora-33 root container
  • PR Add note about empty fields and null values for API responses #11833
    • 10-01 14:07
    • 10-01 14:07
    • 10-01 14:07
  • PR Add guard for BuildOptions.CommonBuildOpts #11813
    • 09-30 18:20
    • 09-30 18:20
    • 09-30 18:20
  • PR pkg/specgen: cache image in generator #11808
    • 10-01 09:24
    • 10-01 09:24
    • 10-01 09:24
    • 10-01 08:17
    • 10-01 08:17
    • 10-01 08:17
  • PR machine: Info on successfully stopping qemu machine #11801
    • 09-30 07:32
    • 09-30 07:32
  • PR Add port configuration to first regular container #11761
    • 09-28 12:02
    • 09-28 12:02
    • 09-28 12:02
  • PR Makefile: use -ldflags/-gccgoflags depending on the go implemenatiton #11716
    • 09-23 11:29
    • 09-23 11:29
    • 09-23 11:29
  • PR Fix Error, empty output for info: 'VERSION' #11631
    • 09-21 13:15
    • 09-21 13:15
    • 09-21 13:15
  • PR Ignore mount errors except ErrContainerUnknown when cleaningup container  #11604
    • 09-22 16:22
    • 09-22 16:22
    • 09-22 16:22

The fact that it only seems to happen on f33 suggests a cgroups v1 problem.

[mheon]

It's all Cgroupsv1 podman-in-container.

We have a protection against this (setting parent cgroup max processes to 0, to prevent cloning, to prevent the cleanup process from launching; this is usually caused by the cleanup process still being active in the cgroup after the last container exits). Possible this doesn't work in a container?

[edsantiago]

Another one, f33 root again

[edsantiago]

And another

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[edsantiago]

In the past month:

Podman prune [It] podman system prune - pod,container stopped

• fedora-33 : int podman fedora-33 root container
  • PR Fix flake in upgrade tests #12260
    • 11-11 09:04
    • 11-11 09:04
    • 11-11 09:04
  • PR pod/container create: resolve conflicts of generated names #12137
    • 11-08 08:22
    • 11-08 08:22
    • 11-08 08:22
  • PR Set Checkpointed state to false after restore #12117
    • 10-27 10:30
    • 10-27 10:30
    • 10-27 10:30
  • PR Slirp4netns with ipv6 set net.ipv6.conf.default.accept_dad=0 #12098
    • 10-26 13:18
    • 10-26 13:18
    • 10-26 13:18
  • PR Record the image stream along with the path #12090
    • 10-26 17:33
    • 10-26 17:33
    • 10-26 17:33
    • 10-26 16:48
    • 10-26 16:48
    • 10-26 16:48

[edsantiago]

Still happening

Podman prune [It] podman system prune - pod,container stopped

• fedora-34 : int podman fedora-34 root container
  • PR specgen: honor empty args for entrypoint specified as --entrypoint "" #12541
    • 12-08 10:42
    • 12-08 10:42
    • 12-08 10:42

[vrothberg]

It's all Cgroupsv1 podman-in-container.

We have a protection against this (setting parent cgroup max processes to 0, to prevent cloning, to prevent the cleanup process from launching; this is usually caused by the cleanup process still being active in the cgroup after the last container exits). Possible this doesn't work in a container?

@giuseppe could you take a look at this one?

[giuseppe]

PR here: containers/common#858