Updating a containers DeviceCgroupRules not working

[agners]

Description
The Docker Engine API documents DeviceCgroupRules as possible properties in the payload to update a container. However, the Device CGroup rules do not get updated accordingly.

Steps to reproduce the issue:

# CONTAINER=$(docker run -d --rm alpine sleep 120)
# docker inspect --format="{{.HostConfig.DeviceCgroupRules}}" ${CONTAINER}
[]
# curl --unix-socket /var/run/docker.sock \
     -H "Content-Type: application/json" \
     -d '{ "DeviceCgroupRules": ["c 13:* rwm"] }' \
     "http://localhost/v1.41/containers/${CONTAINER}/update"
{"Warnings":null}
# docker inspect --format="{{.HostConfig.DeviceCgroupRules}}" ${CONTAINER}
[]

Describe the results you received:
No updated Device CGroup Rules (empty list)

Describe the results you expected:
The new list of Device CGroup as specified in the update API request.

Additional information you deem important (e.g. issue happens only occasionally):

Output of docker version:

$ docker version
Client:
 Version:           20.10.5
 API version:       1.41
 Go version:        go1.16
 Git commit:        55c4c88966
 Built:             Wed Mar  3 16:51:54 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server:
 Engine:
  Version:          20.10.5
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16
  Git commit:       363e9a88a1
  Built:            Wed Mar  3 16:51:28 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.4.4
  GitCommit:        05f951a3781f4f2c1911b05e61c160e9c30eaa8e.m
 runc:
  Version:          1.0.0-rc93
  GitCommit:        12644e614e25b05da6fd08a38ffa0cfe1903fdec
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Output of docker info:

$ docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.5.1-tp-docker)

Server:
 Containers: 40
  Running: 2
  Paused: 0
  Stopped: 38
 Images: 368
 Server Version: 20.10.5
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: false
 Logging Driver: journald
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 05f951a3781f4f2c1911b05e61c160e9c30eaa8e.m
 runc version: 12644e614e25b05da6fd08a38ffa0cfe1903fdec
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.11.10-arch1-1
 Operating System: Arch Linux
 OSType: linux
 Architecture: x86_64
 CPUs: 24
 Total Memory: 62.78GiB
 Name: allenwind
 ID: 4YKC:JPTF:AI6R:AV34:KFP6:PR3Y:J72G:OEHE:3ZH2:OM3Y:6M52:HUFJ
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Username: agners
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Additional environment details (AWS, VirtualBox, physical, etc.):
Same behavior with Docker 19.03.15.

[agners]

I did some more investigation using Docker 20.10.6 and cgroupv2 using the systemd driver. Also in this combination updating device cgroups seems not to work.

What I noticed is that Docker/containerd seems to duplicate the eBPF programs whenever a container update is executed:

# CONTAINER=$(docker run -d --rm alpine sleep 120)
# bpftool cgroup show /sys/fs/cgroup/system.slice/docker-${CONTAINER}.scope/
ID       AttachType      AttachFlags     Name           
91       device          multi                          
# bpftool prog
...
91: cgroup_device  tag 47d09a0f1e1202f4
        loaded_at 2021-04-20T16:53:04+0200  uid 0
        xlated 792B  jited 520B  memlock 4096B
# curl --unix-socket /var/run/docker.sock \
     -H "Content-Type: application/json" \
     -d '{ "DeviceCgroupRules": ["c 13:* rwm"] }' \
     "http://localhost/v1.41/containers/${CONTAINER}/update"
# bpftool prog
...
91: cgroup_device  tag 47d09a0f1e1202f4
        loaded_at 2021-04-20T16:53:04+0200  uid 0
        xlated 792B  jited 520B  memlock 4096B
93: cgroup_device  tag 47d09a0f1e1202f4
        loaded_at 2021-04-20T23:48:27+0200  uid 0
        xlated 792B  jited 520B  memlock 4096B
# bpftool cgroup show /sys/fs/cgroup/system.slice/docker-${CONTAINER}.scope/
ID       AttachType      AttachFlags     Name           
91       device          multi                          
93       device          multi

I guess that a new eBPF program gets created is expected, a bit unexpected that the old eBPF program still exists. Seems to be an entirely different bug.

But even the new eBPF program still has the same device restrictions.

Investigating a bit further revealed that the Docker daemon does not pass new device restrictions to containerd (in toContainerdResources at https://github.com/moby/moby/blob/master/daemon/update_linux.go#L11).

It seems that libcontainerdtypes.Resources does support device restrictions. But even when passing device restrictions it seems that containerd (currently) does not update the eBPF filters. So I guess this is a containerd limitation then?

The fact that containerd recreates the eBPF device filter programs on update seems to indicate that most of the infrastructure to adjust cgroup device restrictions dynamically is already present.

@AkihiroSuda maybe you have some pointers for me?

[AkihiroSuda]

maybe you have some pointers for me?

runc does not support updating devices for cgroup2
opencontainers/runc#2366

[ningfdx]

Yes, update only update some fields of container.Resource struct, doesn't include DeviceCgroupRules and other Device fields.

moby/container/container_unix.go

Line 278 in 7b9275c

func (container *Container) UpdateContainer(hostConfig *containertypes.HostConfig) error {

[agners]

Now with opencontainers/runc#2366 addressed by opencontainers/runc#2951, and release 1.0.0 released this should be one step closer. Is 20.10 going to use runc 1.0.0?

From what I understand libcontainerdtypes.Resources already support device restrictions so it should just be a moby change?