(regression in master) not ok 117 update devices [minimal transition rules]

[kolyshkin]

Got an error from CI for PR #3011, during test (1.16.x) integration test with systemd. Also reproducible on master and other recent PRs.

Quoting https://github.com/opencontainers/runc/runs/2787163520?check_suite_focus=true:

not ok 117 update devices [minimal transition rules]
# (in test file tests/integration/update.bats, line 625)
#   `[ -z "$(<"$CONTAINER_OUTPUT")" ]' failed
# runc spec (status=0):
# 
# runc run -d --console-socket /tmp/bats-run-25456/runc-tmp-recvtty.9wbHaU/console.sock test_update (status=0):
# 
# runc update --pids-limit 30 test_update (status=0):
# 
# runc update --pids-limit 30 test_update (status=0):
# 
# runc update --pids-limit 30 test_update (status=0):
# 
# runc update --pids-limit 30 test_update (status=0):
# 
# runc update --pids-limit 30 test_update (status=0):
# 
# runc update --pids-limit 30 test_update (status=0):
# 
# runc update --pids-limit 30 test_update (status=0):
# 
# runc update --pids-limit 30 test_update (status=0):
# 
# runc update --pids-limit 30 test_update (status=0):
# 
# runc update --pids-limit 30 test_update (status=0):
# 
# runc update --pids-limit 30 test_update (status=0):
# 
# runc update --pids-limit 30 test_update (status=0):
# 
# sh: can't create /dev/null: Operation not permitted
# /tmp/bats-run-25456/bats.37480.src: line 603: 39193 Killed                  ( "$RECVTTY" --no-stdin --pid-file "$TMP_RECVTTY_PID" --mode single "$TMP_CONSOLE_SOCKET" &> "$CONTAINER_OUTPUT" )

The test case was added in commit 4438eaa, PR #2391.

This sh: can't create /dev/null: Operation not permitted means somewhere during runc update the device access list might become empty for some time.

May be related to recent PR #2994 (although I do not see how). Definitely not related to PR #2986 as we have cgroup v1 here.

[kolyshkin]

Got it again on the same PR, the only difference is go version (1.15 this time). Hmm, maybe it is related to the changes I made in #3011.

From https://github.com/opencontainers/runc/runs/2787301089?check_suite_focus=true:

not ok 117 update devices [minimal transition rules]
# (in test file tests/integration/update.bats, line 625)
#   `[ -z "$(<"$CONTAINER_OUTPUT")" ]' failed
# runc spec (status=0):
# 
# runc run -d --console-socket /tmp/bats-run-25864/runc-tmp-recvtty.5CVnWw/console.sock test_update (status=0):
# 
# runc update --pids-limit 30 test_update (status=0):
# 
# runc update --pids-limit 30 test_update (status=0):
# 
# runc update --pids-limit 30 test_update (status=0):
# 
# runc update --pids-limit 30 test_update (status=0):
# 
# runc update --pids-limit 30 test_update (status=0):
# 
# runc update --pids-limit 30 test_update (status=0):
# 
# runc update --pids-limit 30 test_update (status=0):
# 
# runc update --pids-limit 30 test_update (status=0):
# 
# runc update --pids-limit 30 test_update (status=0):
# 
# runc update --pids-limit 30 test_update (status=0):
# 
# runc update --pids-limit 30 test_update (status=0):
# 
# runc update --pids-limit 30 test_update (status=0):
# 
# sh: can't create /dev/null: Operation not permitted
# /tmp/bats-run-25864/bats.38040.src: line 603: 39758 Killed                  ( "$RECVTTY" --no-stdin --pid-file "$TMP_RECVTTY_PID" --mode single "$TMP_CONSOLE_SOCKET" &> "$CONTAINER_OUTPUT" )

[kolyshkin]

Created #3015 to investigate

[kolyshkin]

Bad news -- I was able to reproduce this on master branch (see PR #3015), CI job https://github.com/opencontainers/runc/pull/3015/checks?check_run_id=2787947078.

Either we broke something recently, or it was always like that, I dunno.

[kolyshkin]

Adding 1.0.0 milestone just in case we did broke something recently (it's not yet clear).

[kolyshkin]

A few more failures recently with the same issue

• https://github.com/opencontainers/runc/runs/2783191713?check_suite_focus=true
• https://github.com/opencontainers/runc/runs/2783214868?check_suite_focus=true
• https://github.com/opencontainers/runc/runs/2762123551?check_suite_focus=true

The last one is the oldest, it failed 3 days ago.

[kolyshkin]

Shit, I think I did a very bad thing. We need to revert #2994.