How to solve exec failed by upgrading to 1.1.4 without recreate the pod

[huweihuang]

backgroud:
runc 1.1.4
Fix failed exec after systemctl daemon-reload. Due to a regression
in v1.1.3, the DeviceAllow=char-pts rwm rule was no longer added and
was causing an error open /dev/pts/0: operation not permitted: unknown when systemd was reloaded

We can upgrade runc to version 1.1.4 and recreate the pod to solve this problem.
But how to solve an existing pod without re-create the pod. Because there are many pods created through runc1.1.3, the impact is large

[kolyshkin]

I think the best way is to manually add the rule to affected systemd units. This can be done via systemctl set-property.

[huweihuang]

Can you provide more detailed commands about the rule to affected systemd units.

[kolyshkin]

I can give you enough information to figure this out yourself, to prevent you from running some command someone gave you on the internet that you don't fully understand.

When runc is used with systemd cgroups driver (i.e. runc --systemd-cgroup), it uses systemd to manage container's cgroup. More details on this is https://github.com/opencontainers/runc/blob/main/docs/systemd.md). In particular, it creates a systemd unit (usually a scope) named based on either container name, or the value of linux.CgroupsPath (see the doc above for details).

Now, the runc bug is that it omits a specific DeviceAllow setting for this systemd unit (the subject of bug #3551). It is not a problem at first, because runc also sets the device rules directly in cgroupfs, on top of what systemd sets (questionable behavior BTW). Now, when you do systemctl daemon-reload, apparently systemd re-applies unit properties (including the device rules), and as a result, the container can't access pseudo-terminal devices (/dev/pts/*).

The way to fix this without restarting everything is to add the appropriate DeviceAllow rule to the appropriate systemd unit. To do that, you need to figure out the unit name (see above) and the needed DeviceAllow property value (for that, you can use create a new container using runc > 1.1.3 and see what the properties are), when use systemctl set-property.

[kolyshkin]

By the way, the author of the question refers to #3551, fixed by #3554

[huweihuang]

If I created a container through runc v1.1.3, but did not execute systemctl daemon-reload, and I upgrade to 1.1.4 now, will the next execution of systemctl daemon-reload command still cause the container created by runc v1.1.3 exec fail？

[kolyshkin]

Yes (mere upgrade of runc doesn't mean that the missing rule will somehow be added to all existing containers).