[circleci local execute] error looking up cgroup: not implemented for cgroup v2 unified hierarchy #589
Comments
|
got the same issue. I have docker installed and working properly. |
|
Same for me |
|
Same issue I'm fairly certain this will only work if the kernel parameter |
|
Same here, guess everybody's on Arch. |
|
I get the same error, on Manjaro (an Arch derivative). |
|
I updated my Arch packages, annnnnnd here we are. |
|
Looks like this error from this library. These docs say that with a recent version of the library and with a recent Linux kernel, the kernel parameter It looks like
Edit: UpdateI pulled It looks like the latest |
|
Same issue on Arch Linux: |
|
I have more information about this problem that the CircleCI folks are conveniently ignoring. (Yes, I'm still using the AUR package, not the snap.) I wonder if this will become a bigger problem when Debian Bullseye gets released as stable. |
|
I can confirm that the fix mentioned by @dgoffredo works for Manjaro. How to fixsudo vim /etc/defaults/grub Change this line: GRUB_CMDLINE_LINUX="systemd.unified_cgroup_hierarchy=1"Update Grub: sudo update-grubReboot. |
|
I still get the following error when trying to locally run a CI job from a repository I work on: $ circleci local execute
Docker image digest: sha256:5bc33117a2ea3be8ed7cc8aab995fc69906a1112249e814fa8e7476b570a03a0
====>> Spin up environment
Build-agent version ()
System information:
Error determining system info
Error:
Unexpected environment preparation error: error looking up cgroup: not implemented for cgroup v2 unified hierarchy
Step failed
Task failed
Error: Unhandled prepare executor error: error looking up cgroup: not implemented for cgroup v2 unified hierarchyI see that the most recent $ docker pull circleci/picard
Using default tag: latest
latest: Pulling from circleci/picard
29291e31a76a: Pull complete
8afb3f12a298: Pull complete
f150479d1511: Pull complete
2e647f59c548: Pull complete
da193f18bce6: Pull complete
7c602d616b9c: Pull complete
c402e98532e9: Pull complete
Digest: sha256:5bc33117a2ea3be8ed7cc8aab995fc69906a1112249e814fa8e7476b570a03a0
Status: Downloaded newer image for circleci/picard:latest
docker.io/circleci/picard:latest
$ docker run --rm --interactive --tty --entrypoint=/bin/sh circleci/picard
/ # cd /opt/circleci/linux/amd64
/opt/circleci/linux/amd64 # ls
circleci-agent
/opt/circleci/linux/amd64 # grep 'runc@' circleci-agent
/home/circleci/go/pkg/mod/github.com/opencontainers/[email protected]/libcontainer/user/user.go
/home/circleci/go/pkg/mod/github.com/opencontainers/[email protected]/libcontainer/devices/device_unix.go
/home/circleci/go/pkg/mod/github.com/opencontainers/[email protected]/libcontainer/configs/namespaces_linux.go
/home/circleci/go/pkg/mod/github.com/opencontainers/[email protected]/libcontainer/configs/namespaces_syscall.go
/home/circleci/go/pkg/mod/github.com/opencontainers/[email protected]/libcontainer/userns/userns_linux.go
/home/circleci/go/pkg/mod/github.com/opencontainers/[email protected]/libcontainer/user/lookup_unix.go
/home/circleci/go/pkg/mod/github.com/opencontainers/[email protected]/libcontainer/cgroups/utils.go
/home/circleci/go/pkg/mod/github.com/opencontainers/[email protected]/libcontainer/cgroups/v1_utils.go
/home/circleci/go/pkg/mod/github.com/opencontainers/[email protected]/libcontainer/cgroups/file.go
/opt/circleci/linux/amd64 #
$I tried running all of cgroup v2 is enabled already: $ grep 'cgroup' /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="systemd.unified_cgroup_hierarchy=1"
# Enable cgroup v2. See <https://kind.sigs.k8s.io/docs/user/rootless/>.
GRUB_CMDLINE_LINUX="systemd.unified_cgroup_hierarchy=1"
$ |
|
FWIW, I think |
|
I'm experiencing this same issue on OSX Big Sur. |
|
My macbook is facing same error. |
|
Hitting this today on MacOS Monterey. $ circleci local execute --job build
Docker image digest: sha256:e025cba3f5236a7e3ed9b97dfadce44c824ea0cc1ae1427823ff4c1dd2f420c3
====>> Spin up environment
Build-agent version ()
System information:
Server Version: 20.10.11
Storage Driver: overlay2
Backing Filesystem: extfs
Cgroup Driver: cgroupfs
Cgroup Version: 2
Kernel Version: 5.10.76-linuxkit
Operating System: Docker Desktop
OSType: linux
Architecture: x86_64
Error:
Unexpected environment preparation error: error looking up cgroup: not implemented for cgroup v2 unified hierarchy
Step failed
Task failed
Error: Unhandled prepare executor error: error looking up cgroup: not implemented for cgroup v2 unified hierarchyThis is using the latest circleci/picard - released ~an hour ago - https://hub.docker.com/layers/circleci/picard/1.0.95690-99ec9ada/images/sha256-e025cba3f5236a7e3ed9b97dfadce44c824ea0cc1ae1427823ff4c1dd2f420c3?context=explore We can also see that docker run --rm --interactive --tty --entrypoint=/bin/sh circleci/picard
/ # cd /opt/circleci
/opt/circleci # grep 'runc@' ./linux/arm64/circleci-agent
/home/circleci/go/pkg/mod/github.com/opencontainers/[email protected]/libcontainer/user/user.go
/home/circleci/go/pkg/mod/github.com/opencontainers/[email protected]/libcontainer/devices/device_unix.go
/home/circleci/go/pkg/mod/github.com/opencontainers/[email protected]/libcontainer/configs/namespaces_linux.go
/home/circleci/go/pkg/mod/github.com/opencontainers/[email protected]/libcontainer/configs/namespaces_syscall.go
/home/circleci/go/pkg/mod/github.com/opencontainers/[email protected]/libcontainer/userns/userns_linux.go
/home/circleci/go/pkg/mod/github.com/opencontainers/[email protected]/libcontainer/user/lookup_unix.go
/home/circleci/go/pkg/mod/github.com/opencontainers/[email protected]/libcontainer/cgroups/utils.go
/home/circleci/go/pkg/mod/github.com/opencontainers/[email protected]/libcontainer/cgroups/v1_utils.go
/home/circleci/go/pkg/mod/github.com/opencontainers/[email protected]/libcontainer/cgroups/file.goThis is bothersome, since I can no longer test out circleci changes and have to use remote resources to debug. |
|
I'm seeing this also when attempting to test bioconda builds on a MacBook locally. |
|
Any update on this? circleci local execute --job test
Docker image digest: sha256:eb0c4688f843e6987c29f72c9fdf8c910d59b2a72c7e1559d46a3b1837038cdb
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
====>> Spin up environment
Build-agent version ()
System information:
Server Version: 20.10.11
Storage Driver: overlay2
Backing Filesystem: extfs
Cgroup Driver: cgroupfs
Cgroup Version: 2
Kernel Version: 5.10.76-linuxkit
Operating System: Docker Desktop
OSType: linux
Architecture: aarch64
Error:
Unexpected environment preparation error: error looking up cgroup: not implemented for cgroup v2 unified hierarchy
Step failed
Task failed
Error: Unhandled prepare executor error: error looking up cgroup: not implemented for cgroup v2 unified hierarchycircleci version
0.1.16535+5010eb7 (homebrew) |
|
Also seeing this with Docker Desktop 4.3.1 (72247) and a 2019 MacBook Pro running macOS 12.0.1 (21A559). It was fine with Docker Desktop 4.2.0. Running the latest CircleCI CLI (v0.1.16535). |
|
Running into the same issue with the latest CircleCI CLI (v0.1.16535) after upgrading the local docker desktop to the latest version 4.3.1 (72247) on Mac. |
|
Hey folks, I'm a product manager here at CircleCI. Thank you for making us aware of the problem and apologies for this loss of functionality. Our engineering team is investigating a fix. I unfortunately do not have an estimate at this time on when we'll get this fixed but as soon as I have more information I'll update this thread. We'll be updating the local CLI docs page to call out this limitation in the time being. Thanks all for your patience. |
|
Hi Sebastian, thanks a lot for working on this, I really appreciate it! |
|
We believe we have a fix for this issue in the CircleCI To accelerate your rollout of the agent, you can run: |
|
After deleting the build agent file and re-running, I'm now getting a different error on Manjaro: |
|
Seems like we have further debugging work to do on Linux. This was tested primarily on Docker Desktop for Mac. |
|
@mike-hearn could you post your version of: for comparison? |
|
(I'm interested in which |
|
Sure: |
|
I was specifically asked for feedback by Pete. Apologies if it's cluttering your inbox. |
|
Hey folks, thank you for letting us know that you're experiencing issues. We believe we have identified the root cause and are working on pushing a fix. Will update when we have more. cc @bjohnso5 @pete-woods |
|
We believe we have rolled out a fix for this recently reported issue. To accelerate your rollout of the agent, you can run: rm ~/.circleci/build_agent_settings.json to get the latest version. Please let us know if this still does not fix the issue for you. |
|
Can confirm, resolved for me. Thank you @sebastian-lerner. To make sure everyone is one the same page of instructions: $ rm ~/.circleci/build_agent_settings.json # this is important
$ circleci local execute --job "node/test"
Docker image digest: sha256:008ba7f4223f1e26c11df9575283491b620074fa96da6961e0dcde47fb757014
====>> Spin up environment
Build-agent version ()
System information:
Server Version: 20.10.11
Storage Driver: overlay2
Backing Filesystem: extfs
Cgroup Driver: cgroupfs
Cgroup Version: 2
Kernel Version: 5.10.76-linuxkit
Operating System: Docker Desktop
OSType: linux
Architecture: x86_64
Starting container cimg/node:16.13.2
Warning: No authentication provided, using CircleCI credentials for pulls from Docker Hub.
image is cached as cimg/node:16.13.2, but refreshing...
cimg/node:16.13.2:
using image cimg/node@sha256:0704990b98acbe821fafda00cd605a625598ac737e9b52df41713e2829cc5f78
pull stats: Image was already available so the image was not pulled
time to create container: 41ms
Creating Docker containers in parallel
16.13.2: Pulling from cimg/node
Digest: sha256:0704990b98acbe821fafda00cd605a625598ac737e9b52df41713e2829cc5f78
Status: Image is up to date for cimg/node:16.13.2
Time to upload agent and config: 2.026862226s
Time to start containers: 371.585657ms |
|
I've tried removing the build agent file already but am still getting the following error: |
|
+1 on issue: Running Catalina 10.15.7 and seeing the same error as @jmstone617 . I thought at first it was per my specific config.yml (I was still testing with my original file), but I duplicated the simple config file from @jtarchie, ran the identical commands to remove the build settings file and execute the job locally and still seeing the following error:
|
|
Exactly same error here. Running macOS Monterrey 12.2.1, installed CLI manually (v0.1.16947). Trace: |
|
The older I opened #676 since it seems to be a recurrence of a similar issue on a new image. |
|
How to change |
|
Okay. In my case, I changed |
|
@deokseong-kim-toss nice :) Yeah I put my steps on #676 - maybe it'd be good for you to +1 that issue. |
|
Hi folks, thank you for the detailed reports of what you are seeing. My team is actively investigating and we hope to have a resolution as soon as possible. I'll update this issue when I have more information. |
|
I have a very similar error: Steps to replicate:
|
|
@Kayden-lolasery see if the workaround in #676 works for you |
|
Hi, getting the following on Monterey 12.2, MacBook Pro 16-inch 2019 (intel) Output: |
FYI the workaround in #676 worked for me as well |
|
Can also confirm the workaround from @deokseong-kim-toss re: #676 solves the issue for me as well. Thank you for sharing that! |
|
@sebastian-lerner I'm having the same problem. MacOS: Monterey (12.3.1) The problem occurs with I've removed the build agent file, though the error I'm getting is consistently this:
|
|
@matschaffer Thank you! I have been able to get that to work. |
|
Hey folks, a quick update on this. We believe we've identified the root cause with the latest issue that has been reported and we think we've attributed it to a bug with Docker Desktop for Mac which we've filed here: docker/for-mac#6288 We're in the process of working with Docker to try and get this fixed. I'll update when I have more |
|
Hey folks, (I'm from CircleCI btw). The workaround that we described before, still continue to use that. We have raised the issue with Docker but they will take awhile to resolve the cgroupsv2 issue. We will update folks here on the thread once they have resolved the issue and thank you for your patience. |
|
FWIW, the issue still persists, even if cd ~/.circleci/
USER:.circleci USER$ ls
cli.yml update_check.ymlWould be nice to see if there's any workaround for this 🙏 |
|
We're still waiting to see if there is a workaround but at the moment, we're still investigating with folx. |
and others
Meta:
CircleCI CLI Version:
0.1.15195+595cc9e (release)
Operating System:
Arch Linux (latest)
CircleCI CLI Diagnostic:
Current behavior:
Expected behavior:
"circleci local execute" works
When did this begin?
Now
Was this previously working?:
No, never.
The text was updated successfully, but these errors were encountered: