Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

RUN fails in rootless podman build when cgroups is v1 #6982

Closed
csomh opened this issue Jul 15, 2020 · 21 comments · Fixed by containers/common#231
Closed

RUN fails in rootless podman build when cgroups is v1 #6982

csomh opened this issue Jul 15, 2020 · 21 comments · Fixed by containers/common#231
Labels
kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.

Comments

@csomh
Copy link
Contributor

csomh commented Jul 15, 2020

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

When cgroups is v1 podman build fails on RUN step with the following:

ERRO[0000] systemd cgroup flag passed, but systemd support for managing cgroups is not available 
systemd cgroup flag passed, but systemd support for managing cgroups is not available
error running container: error creating container for [/bin/sh -c touch /file]: : exit status 1
Error: error building at STEP "RUN touch /file": error while running runtime: exit status 1

Cannot reproduce with buildah bud, so filing it here.

Steps to reproduce the issue:

On a system running an up-to-date Fedora 32:

  1. Check that podman build is working okay:
$ podman build --no-cache -f -
FROM fedora:latest
RUN touch /file
STEP 1: FROM fedora:latest
STEP 2: RUN touch /file
STEP 3: COMMIT
--> 85f1922ea05
85f1922ea05bfe11e035caefa54d31f22a8c10559e7f4ca3fe9215ea2b31e163
  1. Configure system to use cgroups v1 and reboot.
sudo grubby --update-kernel=ALL --args="systemd.unified_cgroup_hierarchy=0"
  1. Run podman build again:
$ podman build --no-cache -f -
FROM fedora:latest                          
RUN touch /file
STEP 1: FROM fedora:latest
STEP 2: RUN touch /file
ERRO[0000] systemd cgroup flag passed, but systemd support for managing cgroups is not available 
systemd cgroup flag passed, but systemd support for managing cgroups is not available
error running container: error creating container for [/bin/sh -c touch /file]: : exit status 1
Error: error building at STEP "RUN touch /file": error while running runtime: exit status 1
  1. buildah bud works:
$ buildah bud --no-cache -f -
FROM fedora:latest
RUN touch /file
STEP 1: FROM fedora:latest
STEP 2: RUN touch /file
STEP 3: COMMIT
--> 03c30b816e2
03c30b816e266f69814b26c377a85c4419a4c26086b799f29c13bf79e6f4f527

Describe the results you received:

Build fails on RUN step.

Describe the results you expected:

Build should succeed.

Additional information you deem important (e.g. issue happens only occasionally):

Tested this on a fresh Fedora 32 installation.

The above is working with podman 1.8.2, but fails after upgrading to podman 2.0.2

Output of podman version:

Version:      2.0.2
API Version:  1
Go Version:   go1.14.3
Built:        Thu Jan  1 01:00:00 1970
OS/Arch:      linux/amd64

Output of podman info --debug:

When cgroups is v1

host:
  arch: amd64
  buildahVersion: 1.15.0
  cgroupVersion: v1
  conmon:
    package: conmon-2.0.18-1.fc32.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.18, commit: 6e8799f576f11f902cd8a8d8b45b2b2caf636a85'
  cpus: 8
  distribution:
    distribution: fedora
    version: "32"
  eventLogger: file
  hostname: localhost-live
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.7.8-200.fc32.x86_64
  linkmode: dynamic
  memFree: 2097209344
  memTotal: 4090630144
  ociRuntime:
    name: runc
    package: runc-1.0.0-144.dev.gite6555cc.fc32.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc10+dev
      commit: fbdbaf85ecbc0e077f336c03062710435607dbf1
      spec: 1.0.1-dev
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  rootless: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.1-1.fc32.x86_64
    version: |-
      slirp4netns version 1.1.1
      commit: bbf27c5acd4356edb97fa639b4e15e0cd56a39d5
      libslirp: 4.2.0
      SLIRP_CONFIG_VERSION_MAX: 2
  swapFree: 2147479552
  swapTotal: 2147479552
  uptime: 39.76s
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - docker.io
store:
  configFile: /home/csomh/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.1.2-1.fc32.x86_64
      Version: |-
        fusermount3 version: 3.9.1
        fuse-overlayfs: version 1.1.0
        FUSE library version 3.9.1
        using FUSE kernel interface version 7.31
  graphRoot: /home/csomh/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 2
  runRoot: /run/user/1000/containers
  volumePath: /home/csomh/.local/share/containers/storage/volumes
version:
  APIVersion: 1
  Built: 0
  BuiltTime: Thu Jan  1 01:00:00 1970
  GitCommit: ""
  GoVersion: go1.14.3
  OsArch: linux/amd64
  Version: 2.0.2

When cgroups is v2:

host:
  arch: amd64
  buildahVersion: 1.15.0
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.18-1.fc32.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.18, commit: 6e8799f576f11f902cd8a8d8b45b2b2caf636a85'
  cpus: 8
  distribution:
    distribution: fedora
    version: "32"
  eventLogger: file
  hostname: localhost-live
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.7.8-200.fc32.x86_64
  linkmode: dynamic
  memFree: 1938636800
  memTotal: 4090626048
  ociRuntime:
    name: crun
    package: crun-0.14.1-1.fc32.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.14.1
      commit: 598ea5e192ca12d4f6378217d3ab1415efeddefa
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  rootless: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.1-1.fc32.x86_64
    version: |-
      slirp4netns version 1.1.1
      commit: bbf27c5acd4356edb97fa639b4e15e0cd56a39d5
      libslirp: 4.2.0
      SLIRP_CONFIG_VERSION_MAX: 2
  swapFree: 2147479552
  swapTotal: 2147479552
  uptime: 1m 50.78s
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - docker.io
store:
  configFile: /home/csomh/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.1.2-1.fc32.x86_64
      Version: |-
        fusermount3 version: 3.9.1
        fuse-overlayfs: version 1.1.0
        FUSE library version 3.9.1
        using FUSE kernel interface version 7.31
  graphRoot: /home/csomh/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 0
  runRoot: /run/user/1000/containers
  volumePath: /home/csomh/.local/share/containers/storage/volumes
version:
  APIVersion: 1
  Built: 0
  BuiltTime: Thu Jan  1 01:00:00 1970
  GitCommit: ""
  GoVersion: go1.14.3
  OsArch: linux/amd64
  Version: 2.0.2

Package info (e.g. output of rpm -q podman or apt list podman):

$ rpm -q podman
podman-2.0.2-1.fc32.x86_64
$ rpm -q buildah
buildah-1.15.0-1.fc32.x86_64

Additional environment details (AWS, VirtualBox, physical, etc.):

Was able to reproduce both on physical machine as in VM.
@openshift-ci-robot openshift-ci-robot added the kind/bug Categorizes issue or PR as related to a bug. label Jul 15, 2020
@csomh csomh changed the title RUN fails in rootless podman build with "systemd support for managing cgroups is not available" when cgroups is v1 RUN fails in rootless podman build when cgroups is v1 Jul 15, 2020
@weatherfrog
Copy link

After upgrading podman to 2.0.2, we encounter the same problem on Ubuntu (podman seems to use cgroups v1 by default).

$ podman info --debug
host:
  arch: amd64
  buildahVersion: 1.15.0
  cgroupVersion: v1
  conmon:
    package: 'conmon: /usr/libexec/podman/conmon'
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.0.18, commit: '
  cpus: 16
  distribution:
    distribution: ubuntu
    version: "20.04"
  eventLogger: file
  hostname: xps15
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.4.0-40-generic
  linkmode: dynamic
  memFree: 8379371520
  memTotal: 33285279744
  ociRuntime:
    name: runc
    package: 'runc: /usr/sbin/runc'
    path: /usr/sbin/runc
    version: 'runc version spec: 1.0.1-dev'
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  rootless: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: 'slirp4netns: /usr/bin/slirp4netns'
    version: |-
      slirp4netns version 1.1.4
      commit: unknown
      libslirp: 4.2.0
      SLIRP_CONFIG_VERSION_MAX: 2
  swapFree: 999288832
  swapTotal: 1023406080
  uptime: 17h 32m 57.92s (Approximately 0.71 days)
registries:
  search:
  - docker.io
  - quay.io
store:
  configFile: /home/rg/.config/containers/storage.conf
  containerStore:
    number: 23
    paused: 0
    running: 0
    stopped: 23
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: 'fuse-overlayfs: /usr/bin/fuse-overlayfs'
      Version: |-
        fusermount3 version: 3.9.0
        fuse-overlayfs: version 0.7.6
        FUSE library version 3.9.0
        using FUSE kernel interface version 7.31
  graphRoot: /home/rg/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 174
  runRoot: /run/user/1000/containers
  volumePath: /home/rg/.local/share/containers/storage/volumes
version:
  APIVersion: 1
  Built: 0
  BuiltTime: Thu Jan  1 01:00:00 1970
  GitCommit: ""
  GoVersion: go1.14.4
  OsArch: linux/amd64
  Version: 2.0.2

@mheon mheon added the Buildah label Jul 15, 2020
@mheon
Copy link
Member

mheon commented Jul 15, 2020

@TomSweeneyRedHat @nalind PTAL

@csomh
Copy link
Contributor Author

csomh commented Jul 15, 2020

After some code digging the error seems to be due to adding --systemd-cgroup to the runtime. But not sure if the condition is wrong (would need to check for cgroupVersion) or cgroupManager is wrong when cgroup is v1.

Using the same flag with buildah bud will result in the same error:

$ buildah bud --runtime-flag systemd-cgroup --no-cache -f -
from fedora:latest
run touch /file

STEP 1: FROM fedora:latest
STEP 2: run touch /file
ERRO[0000] systemd cgroup flag passed, but systemd support for managing cgroups is not available 
systemd cgroup flag passed, but systemd support for managing cgroups is not available
error running container: error creating container for [/bin/sh -c touch /file]: : exit status 1
error building at STEP "RUN touch /file": error while running runtime: exit status 1
ERRO exit status 1

But anyways, this is just a hint, please take it with a grain of salt.

@csomh
Copy link
Contributor Author

csomh commented Jul 15, 2020

Setting cgroup_manager="cgroupfs" in libpod.conf solves the issue.

@mheon
Copy link
Member

mheon commented Jul 15, 2020

That should be automatically set for rootless cgroupsv1 systems

@edsantiago
Copy link
Member

Seeing this on RHEL8 gating tests.

@mheon
Copy link
Member

mheon commented Jul 15, 2020

Eeek. Alright, we're probably incorrectly setting systemd cgroups on v1 systemd, then.

@weatherfrog
Copy link

Setting cgroup_manager="cgroupfs" in libpod.conf solves the issue.

Confirmed, thank you.

@rhatdan
Copy link
Member

rhatdan commented Jul 16, 2020

Please do not use libpod.conf, Set it in containers.conf and remove libpod.conf.

@csomh
Copy link
Contributor Author

csomh commented Jul 17, 2020

Please do not use libpod.conf, Set it in containers.conf and remove libpod.conf.

Thanks @rhatdan! What is the reason for this? man podman is still referencing libpod.conf and barely mentions containers.conf. What am I missing?

@rhatdan
Copy link
Member

rhatdan commented Jul 17, 2020

Yikes that is a pretty serious omission.

@rhatdan rhatdan mentioned this issue Jul 17, 2020
Merged
@rhatdan
Copy link
Member

rhatdan commented Jul 17, 2020

@csomh PTAL #7009

@TomSweeneyRedHat
Copy link
Member

@mheon, did we ever have a spot where we set cgroup_manager="cgroupfs" for rootless folks? I took a quick look through the Podman code and didn't find that it was getting set to anything outside of the testing code.

@mheon
Copy link
Member

mheon commented Jul 17, 2020

It should be decided by c/common and containers.conf

@rhatdan
Copy link
Member

rhatdan commented Jul 20, 2020

https://github.com/containers/common/blob/master/pkg/config/config.go#L516

@mheon
Copy link
Member

mheon commented Jul 21, 2020

Confirmed, reproduced on a fresh RHEL8 install. It's not just Buildah, all podman run commands are also failing for me.

@mheon
Copy link
Member

mheon commented Jul 21, 2020

I think I know what's going on - that code for c/common seems bunk, it doesn't work on Cgroups v1 systems.

mheon added a commit to mheon/common that referenced this issue Jul 21, 2020
@mheon
Ensure that rootless cgroupsv1 will select cgroupfs
aea7bf8 
The current logic is that, if Podman was built with the systemd
build flag, we will always select systemd cgroups by default.
Then, if we detect no systemd dbus session, we will swap to
cgroupfs. Problem: there are cases where a systemd dbus session
is available, but systemd cgroups don't work - most notably,
rootless mode on cgroups v1 systems. Special-case this so that we
will not try to force systemd mode and break rootless containers.

Fixes containers/podman#6982

Signed-off-by: Matthew Heon <[email protected]>
@mheon mheon mentioned this issue Jul 21, 2020
Merged
@mheon
Copy link
Member

mheon commented Jul 21, 2020

containers/common#231 to fix
@edsantiago @vrothberg @rhatdan FYI - we'll need this landed and then backported so we can cut 2.0.3

mheon added a commit to mheon/common that referenced this issue Jul 21, 2020
@mheon
Ensure that rootless cgroupsv1 will select cgroupfs
a61c5e2 
The current logic is that, if Podman was built with the systemd
build flag, we will always select systemd cgroups by default.
Then, if we detect no systemd dbus session, we will swap to
cgroupfs. Problem: there are cases where a systemd dbus session
is available, but systemd cgroups don't work - most notably,
rootless mode on cgroups v1 systems. Special-case this so that we
will not try to force systemd mode and break rootless containers.

Fixes containers/podman#6982

Signed-off-by: Matthew Heon <[email protected]>
rhatdan pushed a commit to rhatdan/common that referenced this issue Jul 22, 2020
@mheon @rhatdan
Ensure that rootless cgroupsv1 will select cgroupfs
07ad863 
The current logic is that, if Podman was built with the systemd
build flag, we will always select systemd cgroups by default.
Then, if we detect no systemd dbus session, we will swap to
cgroupfs. Problem: there are cases where a systemd dbus session
is available, but systemd cgroups don't work - most notably,
rootless mode on cgroups v1 systems. Special-case this so that we
will not try to force systemd mode and break rootless containers.

Fixes containers/podman#6982

Signed-off-by: Matthew Heon <[email protected]>
@rhatdan rhatdan mentioned this issue Jul 22, 2020
Closed
vrothberg pushed a commit to vrothberg/common that referenced this issue Jul 22, 2020
@mheon @vrothberg
Ensure that rootless cgroupsv1 will select cgroupfs
73c5008 
The current logic is that, if Podman was built with the systemd
build flag, we will always select systemd cgroups by default.
Then, if we detect no systemd dbus session, we will swap to
cgroupfs. Problem: there are cases where a systemd dbus session
is available, but systemd cgroups don't work - most notably,
rootless mode on cgroups v1 systems. Special-case this so that we
will not try to force systemd mode and break rootless containers.

Fixes containers/podman#6982

Cherry-picked commit a61c5e2.

Signed-off-by: Matthew Heon <[email protected]>
Signed-off-by: Valentin Rothberg <[email protected]>
@vrothberg vrothberg mentioned this issue Jul 22, 2020
Merged
@vrothberg
Copy link
Member

containers/common#231 to fix
@edsantiago @vrothberg @rhatdan FYI - we'll need this landed and then backported so we can cut 2.0.3

containers/common#235

@weatherfrog
Copy link

👍 problem is gone with podman 2.0.3. Thank you guys!

@tisc0
Copy link

tisc0 commented Aug 13, 2020 •
edited
Loading

Seems I still have the problem with podman-2.0.4-1.fc31.x86_64 (but the fix of @weatherfrog in ~/.config/libpod.conf works).

If it's fine with a fresh install (which I haven't tested), maybe it would be efficient to fix existing config.

@github-actions github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 23, 2023
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Sep 23, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Ensure that rootless cgroupsv1 will select cgroupfs mheon/common
9 participants
@edsantiago @rhatdan @mheon @weatherfrog @vrothberg @csomh @tisc0 @TomSweeneyRedHat @openshift-ci-robot