s390x cgroupv2 runc error

[ghost]

@AkihiroSuda @giuseppe @kolyshkin

Using the latest upstream kernel and runc. s390x could not start runc with cgroupv2.

runc spec
runc run root

WARN[0001] signal: killed
ERRO[0001] container_linux.go:349: starting container process caused "process_linux.go:439: container init caused "process_linux.go:405: setting cgroup config for procHooks process caused \"error while setting cgroup v2: [can't load program:\\n github.com/cilium/ebpf.newProgramWithBTF\\n /root/go/pkg/mod/github.com/cilium/[email protected]/prog.go:151\\n - permission denied: 0: (69) r1 = *(u16 *)(r2 +0)\\nR2 !read_ok\\nprocessed 1 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0]\"""
container_linux.go:349: starting container process caused "process_linux.go:439: container init caused "process_linux.go:405: setting cgroup config for procHooks process caused \"error while setting cgroup v2: [can't load program:\\n github.com/cilium/ebpf.newProgramWithBTF\\n /root/go/pkg/mod/github.com/cilium/[email protected]/prog.go:151\\n - permission denied: 0: (69) r1 = *(u16 *)(r2 +0)\\nR2 !read_ok\\nprocessed 1 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0]\"""

[AkihiroSuda]

Rootful or rootless?
Also make sure CONFIG_CGROUP_BPF=y CONFIG_CGROUP_DEVICE=y

[ghost]

Just root. Having all the configs. The thing is that it passed for all other arches here except s390x.

grep BPF linux/.config
CONFIG_CGROUP_BPF=y
CONFIG_BPF=y
CONFIG_BPF_SYSCALL=y

grep CGROUP linux/.config
CONFIG_CGROUPS=y
CONFIG_BLK_CGROUP=y
CONFIG_CGROUP_WRITEBACK=y
CONFIG_CGROUP_SCHED=y
CONFIG_CGROUP_PIDS=y
CONFIG_CGROUP_RDMA=y
CONFIG_CGROUP_FREEZER=y
CONFIG_CGROUP_HUGETLB=y
CONFIG_CGROUP_DEVICE=y
CONFIG_CGROUP_CPUACCT=y
CONFIG_CGROUP_PERF=y
CONFIG_CGROUP_BPF=y
CONFIG_CGROUP_DEBUG=y
CONFIG_SOCK_CGROUP_DATA=y

[AkihiroSuda]

cc @estesp

[estesp]

@alicefr do you know who might be able to look into this? @CAILCA what distro is running on your s390x partition?

[ghost]

@alicefr do you know who might be able to look into this? @CAILCA what distro is running on your s390x partition?

RHEL 8.1+ but with the latest upstream kernel and runc.

[alicefr]

Hi @estesp , I can try to reproduce it on my system. @CAILCA can you please describe more in detail:
1- Which kernel have you used and how did you install it?
2- Which kernel parameters have you used?
3- Which runc version?
4- If you can paste the entire kernel config

[giuseppe]

I think the issue is in the eBPF code.

runc probably needs the equivalent of: containers/crun@07bae05

[alicefr]

ouch @giuseppe good catch!I'll have a try

[alicefr]

I was able to reproduce the issue on a Fedora 31 with default kernel and upstream runc. I'll have a closer look

[alicefr]

@giuseppe I tried to apply the change you suggest:

git diff
diff --git a/libcontainer/cgroups/ebpf/devicefilter/devicefilter.go b/libcontainer/cgroups/ebpf/devicefilter/devicefilter.go
index 847ce8ef..18f7e4d1 100644
--- a/libcontainer/cgroups/ebpf/devicefilter/devicefilter.go
+++ b/libcontainer/cgroups/ebpf/devicefilter/devicefilter.go
@@ -49,7 +49,8 @@ func (p *program) init() {
        */
        // R2 <- type (lower 16 bit of u32 access_type at R1[0])
        p.insts = append(p.insts,
-               asm.LoadMem(asm.R2, asm.R1, 0, asm.Half))
+               asm.LoadMem(asm.R2, asm.R1, 0, asm.Word),
+               asm.And.Imm32(asm.R2, 0xFFFF))
 
        // R3 <- access (upper 16 bit of u32 access_type at R1[0])
        p.insts = append(p.insts,

but I still get the same behavior.

[giuseppe]

Are you able to run a different eBPF program on Fedora (e.g. yum install -y bcc-tools && /usr/share/bcc/tools/execsnoop)?

On x86_64 I had to disable secure boot for eBPF to work

[alicefr]

@giuseppe yes, I'm able to execute it without any further configurations.

[alicefr]

strace output
strace-output.txt

[alicefr]

I also tried to run crun and that works without any problems. So, it must be an issue with runc/ebpf go library

[alicefr]

it is an issue in the ebpf library:
I run a test:

go test -run TestProgramRun
--- FAIL: TestProgramRun (0.00s)
    prog_test.go:50:     0: LdXMemW dst: r2 src: r1 off: 4 imm: 0
             1: LdXMemW dst: r1 src: r1 off: 0 imm: 0
             2: MovReg dst: r3 src: r1
             3: AddImm dst: r3 imm: 14
             4: JGTReg dst: r3 off: -1 src: r2 <out>
             5: StMemB dst: r1 src: r0 off: 0 imm: 222
             6: StMemB dst: r1 src: r0 off: 1 imm: 173
             7: StMemB dst: r1 src: r0 off: 2 imm: 190
             8: StMemB dst: r1 src: r0 off: 3 imm: 239
        out:
             9: LdImmDW dst: r0 imm: 42
            11: Exit
        
    prog_test.go:59: can't load program: permission denied: 0: (61) r1 = *(u32 *)(r2 +4)
        R2 !read_ok
        processed 1 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0
FAIL
exit status 1
FAIL    github.com/cilium/ebpf  0.001s
Collapse

[alicefr]

Open an issue in the ebpf repository cilium/ebpf#86

[alicefr]

So, we have a fix for runc. This requires a fix in the clium/ebpf library (a PR will follow) and this fix. This PR #2381 fixes the same issues as @giuseppe already reported and it is based on his work

[alicefr]

Ebpf PR to fix registers on s390x: cilium/ebpf#91

[alicefr]

cilium/ebpf#91 was merged. #2389 update the github.com/cilium/ebpf package