validation: add cgroup devices validation

[zhouhao3]

On the one hand in order to achieve devices validation, on the other hand in order to achieve the following specerror：

DevicesApplyInOrder: The runtime MUST apply entries in the listed order.

But the test results are not the same as I expected, so I want to see what do you think.

➜  runtime-tools git:(devices-validation) ✗ sudo ./validation/linux_cgroups_devices.t
TAP version 13
ok 1 - find devices
ok 2 - get devices data
not ok 3 - devices c 10:229 rwm is set correctly
# expect: c 10:229 rwm, actual: a 0:0 rwm
not ok 4 - devices a 10:200 r is set correctly
# expect: a 10:200 r, actual: a 0:0 rwm
1..4

@wking @liangchenye @alban @dongsupark PTAL

Signed-off-by: Zhou Hao [email protected]

[dongsupark]

When creating a cgroup like this,

# mkdir /sys/fs/cgroup/devices/cgrouptest
# cat /sys/fs/cgroup/devices/cgrouptest/devices.list 
a *:* rwm

all devices are allowed by default, so that the major, minor numbers are always interpreted to 0:0. We would not want it to happen.

How about doing initialization by denying all devices like this, before running tests?

# echo "a *:* rwm" > /sys/fs/cgroup/devices/cgrouptest/devices.deny
# cat /sys/fs/cgroup/devices/cgrouptest/devices.list
#

[zhouhao3]

How about doing initialization by denying all devices like this, before running tests?

I think it's hard to confirm this time, devices are built in runc, and initialization should be done after creation.

But when I use runc alone to start a container, It will implement devices correctly.

➜  ub sudo runc run ub
# 
➜  ~ cd /sys/fs/cgroup/devices/cgroupstest 
➜  cgroupstest cat devices.list 
c 133:229 rw
b 8:0 r
c *:* m
b *:* m
c 1:3 rwm
c 1:8 rwm
c 1:7 rwm
c 5:0 rwm
c 1:5 rwm
c 1:9 rwm
c 5:1 rwm
c 136:* rwm
c 5:2 rwm
c 10:200 rwm
➜  cgroupstest 

[liangchenye]

But when I use runc alone to start a container, It will implement devices correctly.

How do you start it, runc run -b bundlerdir ID ?

[zhouhao3]

How do you start it, runc run -b bundlerdir ID ?

Just configured a runc can start the file, modify the config file, and then start directly.

➜  ls
config.json  rootfs
➜ sudo runc run ub

[dongsupark]

@q384566678 I've just looked into this PR again.
True, as you said, when running manually runc run mycontainer, the device cgroups list shows the full result correctly.

Apart from that, I think the line above g.AddLinuxResourcesDevice(true, "a", &major3, &minor3, "r") causes the issue of the wrong device list.
Think about a sequence of the following command lines.

# echo "c 10:229 rwm" > /sys/fs/cgroup/devices/cgrouptest/devices.allow
# cat /sys/fs/cgroup/devices/cgrouptest/devices.list   # This shows a correct result
c 10:229 rwm

# echo "a 10:200 r" > /sys/fs/cgroup/devices/cgrouptest/devices.allow
# cat /sys/fs/cgroup/devices/cgrouptest/devices.list   # an unexpected result. Existing devices are now gone
a *:* rwm

According to the Kernel cgroup v1 document, doing echo a > /sys/fs/cgroup/1/devices.allow will add the 'a : rwm' entry to the whitelist. Apparently allowing all devices results in wiping out existing entries and adding a single wildcard entry, even when the input is given by a specific pair of major/minor number.

When testing without the line for a, it shows a lot better result, doesn't it?

[wking]

# echo "a 10:200 r" > /sys/fs/cgroup/devices/cgrouptest/devices.allow
# cat /sys/fs/cgroup/devices/cgrouptest/devices.list # an unexpected result. Existing devices are now gone
a *:* rwm

That's surprising to me. In the wild, I expect few users to care about major/minor but not block/char. Still, does anyone have time to file a kernel patch to either respect the passed value or error out when major or minor are passed with a?

[zhouhao3]

updated, @dongsupark @liangchenye PTAL

[dongsupark]

@q384566678 Tested, and it works well. LGTM. 👍

[zhouhao3]

@liangchenye PTAL

[liangchenye]

LGTM

[liangchenye]

@q384566678 @dongsupark , as @wking mentioned, will you fire a bug to kernel?

[zhouhao3]

will you fire a bug to kernel?

I will try to do it when I have free time.