Unable to build images through docker-compose #764

ryan-callahan-edo · 2023-08-06T05:17:17Z

Description

Love colima but I very suddenly started having issues a few hours ago. My workflow relies on vscode dev containers to run a compose file that builds an image for the current codebase and runs containers for dependencies like an rdbms.

I didn't change anything about the codebase, dockerfile, images, compose file, colima configs, or docker configs to trigger this, but after doing many other re-builds I started getting this error on the first command after pulling the base image:

runc run failed: unable to start container process: error during container init: error mounting "cgroup" to rootfs at "/sys/fs/cgroup": mount cgroup:/sys/fs/cgroup/openrc (via /proc/self/fd/6), flags: 0xf, data: openrc: invalid argument

Steps I've tried are:

resetting colima configs
restarting colima
deleting the lima instance and rebuilding
re-installing colima
restarting my laptop
adding cgroup: private and cgroup: host to my compose file
removing dependency containers from the compose file
different combinations of virtualization and mount drivers

My current workaround is to build the image with docker build then use the image in the compose file as that seems to work fine.

Version

Colima Version: 0.5.5
Lima Version: 0.17.0
Qemu Version: 8.0.3

Operating System

Output of `colima status`

INFO[0000] colima is running using macOS Virtualization.Framework
INFO[0000] arch: aarch64
INFO[0000] runtime: docker
INFO[0000] mountType: virtiofs
INFO[0000] socket: unix:///Users/ryancallahan/.colima/default/docker.sock

Reproduction Steps

run docker-compose up on a file that has a build section specified

example file:

version: "3.9"

services:
  app_service:
    build:
      context: ../../
      dockerfile: ./Docker/Development/Dockerfile
    container_name: app_service
    environment:
      ASPNETCORE_ENVIRONMENT: Development
    depends_on:
      pgdb:
        condition: service_healthy
    ports:
      - "43000:43000"
      - "44000:44000"
    volumes:
      - ../../:/app:cached
    command: sleep infinity

  pgdb:
    image: postgres:15.2-alpine3.17
    environment:
      POSTGRES_DB: gdb
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: postgres
    healthcheck:
      test: [ "CMD-SHELL", "pg_isready -U postgres" ]
      interval: 5s
      timeout: 5s
      retries: 10
    ports:
      - "5432:5432"
    volumes:
      - ../../.db:/var/lib/postgresql/data

  redis:
    image: redis:7-alpine
    ports:
      - '6379:6379'
    volumes:
      - ../../.redis:/data

Expected behaviour

build succeeds and containers start.

Additional context

Docker status dump:

Client:
 Version:    24.0.2-rd
 Context:    colima
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.11.0
    Path:     /Users/username/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  2.20.2
    Path:     /Users/username/.docker/cli-plugins/docker-compose

Server:
 Containers: 1
  Running: 1
  Paused: 0
  Stopped: 0
 Images: 1
 Server Version: 23.0.6
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 1fbd70374134b891f97ce19c70b6e50c7b9f4e0d
 runc version: 860f061b76bb4fc671f0f9e900f7d80ff93d4eb7
 init version:
 Security Options:
  seccomp
   Profile: builtin
 Kernel Version: 6.1.29-0-virt
 Operating System: Alpine Linux v3.18
 OSType: linux
 Architecture: aarch64
 CPUs: 6
 Total Memory: 7.754GiB
 Name: colima
 ID: 21e161e6-a6f7-4d42-8140-7c89b745cfd1
 Docker Root Dir: /var/lib/docker
 Debug Mode: true
  File Descriptors: 32
  Goroutines: 50
  System Time: 2023-08-06T05:15:27.816027873Z
  EventsListeners: 3
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

lima info dump:

{
    "version": "0.17.0",
    "templates": [
        {
            "name": "almalinux-8",
            "location": "/opt/homebrew/share/lima/templates/almalinux-8.yaml"
        },
        {
            "name": "almalinux-9",
            "location": "/opt/homebrew/share/lima/templates/almalinux-9.yaml"
        },
        {
            "name": "almalinux",
            "location": "/opt/homebrew/share/lima/templates/almalinux.yaml"
        },
        {
            "name": "alpine",
            "location": "/opt/homebrew/share/lima/templates/alpine.yaml"
        },
        {
            "name": "apptainer-rootful",
            "location": "/opt/homebrew/share/lima/templates/apptainer-rootful.yaml"
        },
        {
            "name": "apptainer",
            "location": "/opt/homebrew/share/lima/templates/apptainer.yaml"
        },
        {
            "name": "archlinux",
            "location": "/opt/homebrew/share/lima/templates/archlinux.yaml"
        },
        {
            "name": "buildkit",
            "location": "/opt/homebrew/share/lima/templates/buildkit.yaml"
        },
        {
            "name": "centos-stream-8",
            "location": "/opt/homebrew/share/lima/templates/centos-stream-8.yaml"
        },
        {
            "name": "centos-stream-9",
            "location": "/opt/homebrew/share/lima/templates/centos-stream-9.yaml"
        },
        {
            "name": "centos-stream",
            "location": "/opt/homebrew/share/lima/templates/centos-stream.yaml"
        },
        {
            "name": "debian-11",
            "location": "/opt/homebrew/share/lima/templates/debian-11.yaml"
        },
        {
            "name": "debian-12",
            "location": "/opt/homebrew/share/lima/templates/debian-12.yaml"
        },
        {
            "name": "debian",
            "location": "/opt/homebrew/share/lima/templates/debian.yaml"
        },
        {
            "name": "default",
            "location": "/opt/homebrew/share/lima/templates/default.yaml"
        },
        {
            "name": "deprecated/centos-7",
            "location": "/opt/homebrew/share/lima/templates/deprecated/centos-7.yaml"
        },
        {
            "name": "docker-rootful",
            "location": "/opt/homebrew/share/lima/templates/docker-rootful.yaml"
        },
        {
            "name": "docker",
            "location": "/opt/homebrew/share/lima/templates/docker.yaml"
        },
        {
            "name": "experimental/9p",
            "location": "/opt/homebrew/share/lima/templates/experimental/9p.yaml"
        },
        {
            "name": "experimental/armv7l",
            "location": "/opt/homebrew/share/lima/templates/experimental/armv7l.yaml"
        },
        {
            "name": "experimental/net-user-v2",
            "location": "/opt/homebrew/share/lima/templates/experimental/net-user-v2.yaml"
        },
        {
            "name": "experimental/opensuse-tumbleweed",
            "location": "/opt/homebrew/share/lima/templates/experimental/opensuse-tumbleweed.yaml"
        },
        {
            "name": "experimental/riscv64",
            "location": "/opt/homebrew/share/lima/templates/experimental/riscv64.yaml"
        },
        {
            "name": "experimental/rke2",
            "location": "/opt/homebrew/share/lima/templates/experimental/rke2.yaml"
        },
        {
            "name": "experimental/virtiofs-linux",
            "location": "/opt/homebrew/share/lima/templates/experimental/virtiofs-linux.yaml"
        },
        {
            "name": "experimental/vz",
            "location": "/opt/homebrew/share/lima/templates/experimental/vz.yaml"
        },
        {
            "name": "faasd",
            "location": "/opt/homebrew/share/lima/templates/faasd.yaml"
        },
        {
            "name": "fedora",
            "location": "/opt/homebrew/share/lima/templates/fedora.yaml"
        },
        {
            "name": "k3s",
            "location": "/opt/homebrew/share/lima/templates/k3s.yaml"
        },
        {
            "name": "k8s",
            "location": "/opt/homebrew/share/lima/templates/k8s.yaml"
        },
        {
            "name": "nomad",
            "location": "/opt/homebrew/share/lima/templates/nomad.yaml"
        },
        {
            "name": "opensuse",
            "location": "/opt/homebrew/share/lima/templates/opensuse.yaml"
        },
        {
            "name": "oraclelinux-8",
            "location": "/opt/homebrew/share/lima/templates/oraclelinux-8.yaml"
        },
        {
            "name": "oraclelinux-9",
            "location": "/opt/homebrew/share/lima/templates/oraclelinux-9.yaml"
        },
        {
            "name": "oraclelinux",
            "location": "/opt/homebrew/share/lima/templates/oraclelinux.yaml"
        },
        {
            "name": "podman-rootful",
            "location": "/opt/homebrew/share/lima/templates/podman-rootful.yaml"
        },
        {
            "name": "podman",
            "location": "/opt/homebrew/share/lima/templates/podman.yaml"
        },
        {
            "name": "rocky-8",
            "location": "/opt/homebrew/share/lima/templates/rocky-8.yaml"
        },
        {
            "name": "rocky-9",
            "location": "/opt/homebrew/share/lima/templates/rocky-9.yaml"
        },
        {
            "name": "rocky",
            "location": "/opt/homebrew/share/lima/templates/rocky.yaml"
        },
        {
            "name": "ubuntu-lts",
            "location": "/opt/homebrew/share/lima/templates/ubuntu-lts.yaml"
        },
        {
            "name": "ubuntu",
            "location": "/opt/homebrew/share/lima/templates/ubuntu.yaml"
        },
        {
            "name": "vmnet",
            "location": "/opt/homebrew/share/lima/templates/vmnet.yaml"
        }
    ],
    "defaultTemplate": {
        "vmType": "qemu",
        "arch": "aarch64",
        "images": [
            {
                "location": "https://cloud-images.ubuntu.com/releases/23.04/release-20230729/ubuntu-23.04-server-cloudimg-amd64.img",
                "arch": "x86_64",
                "digest": "sha256:300d765c17d047403e686c9513bcc71aecddbf4954750d109fb79634ebd69510"
            },
            {
                "location": "https://cloud-images.ubuntu.com/releases/23.04/release-20230729/ubuntu-23.04-server-cloudimg-arm64.img",
                "arch": "aarch64",
                "digest": "sha256:a4c5368e462bdbd3b317cc27aa2ac96c3c302813ec138ba307aa6f5f21286f48"
            },
            {
                "location": "https://cloud-images.ubuntu.com/releases/23.04/release/ubuntu-23.04-server-cloudimg-amd64.img",
                "arch": "x86_64"
            },
            {
                "location": "https://cloud-images.ubuntu.com/releases/23.04/release/ubuntu-23.04-server-cloudimg-arm64.img",
                "arch": "aarch64"
            }
        ],
        "cpuType": {
            "aarch64": "host",
            "armv7l": "cortex-a7",
            "riscv64": "rv64",
            "x86_64": "qemu64"
        },
        "cpus": 4,
        "memory": "4GiB",
        "disk": "100GiB",
        "mounts": [
            {
                "location": "~",
                "mountPoint": "~",
                "writable": false,
                "sshfs": {
                    "cache": true,
                    "followSymlinks": false,
                    "sftpDriver": ""
                },
                "9p": {
                    "securityModel": "none",
                    "protocolVersion": "9p2000.L",
                    "msize": "128KiB",
                    "cache": "fscache"
                },
                "virtiofs": {}
            },
            {
                "location": "/tmp/lima",
                "mountPoint": "/tmp/lima",
                "writable": true,
                "sshfs": {
                    "cache": true,
                    "followSymlinks": false,
                    "sftpDriver": ""
                },
                "9p": {
                    "securityModel": "none",
                    "protocolVersion": "9p2000.L",
                    "msize": "128KiB",
                    "cache": "mmap"
                },
                "virtiofs": {}
            }
        ],
        "mountType": "reverse-sshfs",
        "ssh": {
            "localPort": 0,
            "loadDotSSHPubKeys": true,
            "forwardAgent": false,
            "forwardX11": false,
            "forwardX11Trusted": false
        },
        "firmware": {
            "legacyBIOS": false
        },
        "audio": {
            "device": ""
        },
        "video": {
            "display": "none",
            "vnc": {
                "display": "127.0.0.1:0,to=9"
            }
        },
        "containerd": {
            "system": false,
            "user": true,
            "archives": [
                {
                    "location": "https://github.com/containerd/nerdctl/releases/download/v1.5.0/nerdctl-full-1.5.0-linux-amd64.tar.gz",
                    "arch": "x86_64",
                    "digest": "sha256:3f8c494e3c6a265fe2a3c41ef9d6bc859eeeb22095b3353d3558d8120833a23a"
                },
                {
                    "location": "https://github.com/containerd/nerdctl/releases/download/v1.5.0/nerdctl-full-1.5.0-linux-arm64.tar.gz",
                    "arch": "aarch64",
                    "digest": "sha256:32a2537e0a80e1493b5934ca56c3e237466606a1b720aef23b9c0a7fc3303bdb"
                }
            ]
        },
        "guestInstallPrefix": "/usr/local",
        "hostResolver": {
            "enabled": true,
            "ipv6": false
        },
        "propagateProxyEnv": true,
        "caCerts": {
            "removeDefaults": false
        },
        "rosetta": {
            "enabled": false,
            "binfmt": false
        }
    },
    "limaHome": "/Users/username/.lima",
    "vmTypes": [
        "qemu",
        "vz"
    ]
}

The text was updated successfully, but these errors were encountered:

chriswgerber · 2023-08-07T17:22:09Z

I've been having the same problem since last week (I think Friday) attempting to build with Buildkit 0.12.0 (running in Kubernetes pod) with the same setup running the same Docker server and runtime versions.

Additional info:

The error occurs for me at the point in a build when Buildkit attempts to execute the RUN ... commands.

Different versions of K3s produce the same error.
Tried running with an Ubuntu layer and it still had the same error.
Tried setting openrc rc_cgroup_mode="unified" and got same error.
Tried with sshfs and go the same error.

Edit:

Hopefully this is helpful in diagnosing, but I changed the Buildkit image to -rootless (specifically buildx-stable-1-rootless) and have been able to build successfully.

zhooda · 2023-08-09T03:36:09Z

I also ran into this issue with with the docker-container driver; managed to get it working again by using the moby/buildkit:v0.12.1-rootless image when creating the builder:

docker buildx create \
              --name fixed_builder \
              --driver-opt 'image=moby/buildkit:v0.12.1-rootless' \
              --bootstrap --use

andyli · 2023-08-09T18:15:27Z

May I know how to set rc_cgroup_mode="unified" in colima?
I'm getting a similar error when using earthly, and was suggested to try rc_cgroup_mode="unified".

onyxraven · 2023-08-11T17:57:01Z

I just ran into this with a buildx bake build, and the rootless driver worked for me. Thanks for the tip @zhooda - I'm going to look and see if theres a config to make that the default so I don't forget it.

that being said, is there something lima/colima need to do to make the 'default' buildx work? or is this a bug in buildkit? just wondering if we've got the right path forward :)

It may have to do with moby/buildkit#4003

uesleilima · 2023-08-18T10:39:01Z

Hey @onyxraven, any chance that you managed to set Colima's default buildx to set the rootless image?

onyxraven · 2023-08-18T15:34:20Z

Hey @onyxraven, any chance that you managed to set Colima's default buildx to set the rootless image?

I haven't had a chance to dig, but at first glance it doesn't look like theres a config file that is referenced for this https://docs.docker.com/engine/reference/commandline/buildx_create/ -- at least that is obvious. It looks like there are some structures for a .docker/buildx/ config location, so maybe there is one its just not in the main docs.

tian-ma · 2023-08-31T19:33:50Z

May I know how to set rc_cgroup_mode="unified" in colima? I'm getting a similar error when using earthly, and was suggested to try rc_cgroup_mode="unified".

colima ssh
sudo sed -E -i 's/#(rc_cgroup_mode).*/\1="unified"/' /etc/rc.conf
sudo service cgroups restart
exit

Jerome1337 · 2023-10-10T09:31:36Z

Same error here, using alpine images on M1Pro Macbook pro with buildx v0.11.2

Doing this 👇 fixed the problem

May I know how to set rc_cgroup_mode="unified" in colima? I'm getting a similar error when using earthly, and was suggested to try rc_cgroup_mode="unified".
colima ssh
sudo sed -E -i 's/#(rc_cgroup_mode).*/\1="unified"/' /etc/rc.conf
sudo service cgroups restart
exit

abiosoft · 2023-10-10T14:07:33Z

For latest version v0.5.6, there is a new --cgroups-v2 flag that should workaround the issue.

abiosoft · 2023-11-11T16:17:54Z

Just an FYI that this is getting fixed in the imminent v0.6.0 release.

BenTheElder mentioned this issue Aug 24, 2023

support default-cgroupns-mode=private #720

Closed

ngearhart mentioned this issue Aug 25, 2023

New BuildX Instance fails to execute, with cgroup errors. rancher-sandbox/rancher-desktop#5363

Closed

onyxraven mentioned this issue Sep 5, 2023

All builds fail with "runc run failed: unable to start container process" #792

Closed

5 tasks

Cryptophobia mentioned this issue Sep 5, 2023

ERROR: failed to solve: process "/bin/sh docker/buildx#1986

Open

3 tasks

chadlwilson mentioned this issue Sep 6, 2023

Build containers using buildkit in rootless mode gocd/gocd#11956

Merged

alexcb mentioned this issue Sep 22, 2023

Since earthly 0.7.14, runc run failed: openrc: invalid argument earthly/earthly#3167

Closed

abiosoft added a commit that referenced this issue Oct 5, 2023

vm: temporary workaround for #764

2887524

abiosoft mentioned this issue Oct 5, 2023

vm: temporary workaround for #764 #823

Merged

abiosoft added a commit that referenced this issue Oct 5, 2023

vm: temporary workaround for #764 (#823)

819ba26

abiosoft added this to the v0.6.0 milestone Nov 11, 2023

abiosoft mentioned this issue Nov 11, 2023

v0.6.0 refactor #848

Merged

abiosoft closed this as completed in #848 Nov 12, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unable to build images through docker-compose #764

Unable to build images through docker-compose #764

ryan-callahan-edo commented Aug 6, 2023 •

edited

Loading

chriswgerber commented Aug 7, 2023 •

edited

Loading

zhooda commented Aug 9, 2023

andyli commented Aug 9, 2023

onyxraven commented Aug 11, 2023 •

edited

Loading

uesleilima commented Aug 18, 2023 •

edited

Loading

onyxraven commented Aug 18, 2023

tian-ma commented Aug 31, 2023

Jerome1337 commented Oct 10, 2023

abiosoft commented Oct 10, 2023

abiosoft commented Nov 11, 2023

Unable to build images through docker-compose #764

Unable to build images through docker-compose #764

Comments

ryan-callahan-edo commented Aug 6, 2023 • edited Loading

Description

Version

Operating System

Output of colima status

Reproduction Steps

Expected behaviour

Additional context

chriswgerber commented Aug 7, 2023 • edited Loading

zhooda commented Aug 9, 2023

andyli commented Aug 9, 2023

onyxraven commented Aug 11, 2023 • edited Loading

uesleilima commented Aug 18, 2023 • edited Loading

onyxraven commented Aug 18, 2023

tian-ma commented Aug 31, 2023

Jerome1337 commented Oct 10, 2023

abiosoft commented Oct 10, 2023

abiosoft commented Nov 11, 2023

ryan-callahan-edo commented Aug 6, 2023 •

edited

Loading

Output of `colima status`

chriswgerber commented Aug 7, 2023 •

edited

Loading

onyxraven commented Aug 11, 2023 •

edited

Loading

uesleilima commented Aug 18, 2023 •

edited

Loading