Crashing Clang with Asan+Ubsan builds in 20240310.1.0 Ubuntu 22.04.

[igsilya]

Description

For some reason we have all the binaries built with clang -fsanitize=address,undefined crashing on the new 20240310.1.0 Ubuntu 22.04 images.

Examples:
https://github.com/ovsrobot/ovs/actions/runs/8237626849
https://github.com/igsilya/ovs/actions/runs/8238397886

Tests that happend to get 20240310.1.0 image are all crashing on startup:

#                             -*- compilation -*-
740. json.at:261: testing exponent must contain at least one digit (1) - C ...
../../../tests/json.at:261: printf %s "[1e]" > input
../../../tests/json.at:261: ovstest test-json  input
--- /dev/null	2024-03-11 19:04:04.198924076 +0000
+++ /home/runner/work/ovs/ovs/openvswitch-3.3.90/_build/sub/tests/testsuite.dir/at-groups/740/stderr	2024-03-11 19:13:51.408292432 +0000
@@ -0,0 +1 @@
+/home/runner/work/ovs/ovs/openvswitch-3.3.90/_build/sub/tests/testsuite.dir/at-groups/740/test-source: line 29: 66606 Segmentation fault      (core dumped) ovstest test-json input
stdout:
../../../tests/json.at:261: exit code was 139, expected 1
input:
> [1e]740. json.at:261: 740. exponent must contain at least one digit (1) - C (json.at:261): FAILED (json.at:261)

Above is a crash of the test with a very simple application.

Platforms affected

• Azure DevOps
• GitHub Actions - Standard Runners
• GitHub Actions - Larger Runners

Runner images affected

• Ubuntu 20.04
• Ubuntu 22.04
• macOS 11
• macOS 12
• macOS 13
• macOS 13 Arm64
• macOS 14
• macOS 14 Arm64
• Windows Server 2019
• Windows Server 2022

Image version and build link

Runner Image
Image: ubuntu-22.04
Version: 20240310.1.0

Is it regression?

Yes. 20240304.1.0 works just fine. For example another job from the same run: https://github.com/ovsrobot/ovs/actions/runs/8237626849/job/22527053666

Expected behavior

All the built binaries should not just crash.

Actual behavior

All the built binaries are crashing on start.

Repro steps

Run this workflow: https://github.com/openvswitch/ovs/actions/workflows/build-and-test.yml with openvswitch/ovs repository.

[igsilya]

Even simple ./configure sometimes crashes:

configure:4721: $? = 0
configure:4710: clang -V >&5
clang: error: argument to '-V' is missing (expected 1 value)
clang: error: no input files
configure:4721: $? = 1
configure:4710: clang -qversion >&5
clang: error: unknown argument '-qversion'; did you mean '--version'?
clang: error: no input files
configure:4721: $? = 1
configure:4710: clang -version >&5
clang: error: unknown argument '-version'; did you mean '--version'?
clang: error: no input files
configure:4721: $? = 1
configure:4741: checking whether the C compiler works
configure:4763: clang -g -O2 -Wno-error=unused-command-line-argument -fno-omit-frame-pointer -fno-common -fsanitize=address,undefined   conftest.c  >&5
configure:4767: $? = 0
configure:4817: result: yes
configure:4820: checking for C compiler default output file name
configure:4822: result: a.out
configure:4828: checking for suffix of executables
configure:4835: clang -o conftest -g -O2 -Wno-error=unused-command-line-argument -fno-omit-frame-pointer -fno-common -fsanitize=address,undefined   conftest.c  >&5
configure:4839: $? = 0
configure:4862: result: 
configure:4884: checking whether we are cross compiling
configure:4892: clang -o conftest -g -O2 -Wno-error=unused-command-line-argument -fno-omit-frame-pointer -fno-common -fsanitize=address,undefined   conftest.c  >&5
configure:4896: $? = 0
configure:4903: ./conftest
./configure: line 4905:  5356 Segmentation fault      (core dumped) ./conftest$ac_cv_exeext
configure:4907: $? = 139
configure:4914: error: in `/home/runner/work/ovs/ovs':
configure:4916: error: cannot run C compiled programs.

[igsilya]

The issue is caused by incompatibility between llvm 14 provided in ubuntu-22.04 image and the much newer kernel configured with high-entropy ASLR.

In 20240304.1.0:

$ sudo sysctl -a | grep vm.mmap.rnd
vm.mmap_rnd_bits = 28
vm.mmap_rnd_compat_bits = 8

In 20240310.1.0:

$ sudo sysctl -a | grep vm.mmap.rnd
vm.mmap_rnd_bits = 32
vm.mmap_rnd_compat_bits = 16

The isue was fixed in newer versions of llvm:
llvm/llvm-project@fb77ca0
https://reviews.llvm.org/D148280

So, we either need:

1. an updated version of llvm/clang that will be compatible with a new kernel.
2. or a kernel config change to reduce the entropy.
3. or a global sysctl change to set vm.mmap_rnd_bits = 28.

As a workaround the following addition to the workflow makes builds to work with a new image:

    - name: Fix kernel mmap rnd bits
      # Asan in llvm 14 provided in ubuntu 22.04 is incompatible with
      # high-entropy ASLR in much newer kernels that GitHub runners are
      # using leading to random crashes: https://reviews.llvm.org/D148280
      run: sudo sysctl vm.mmap_rnd_bits=28

[MaksimZhukov]

Hello @igsilya!
Thank you for reporting this!
We will investigate the issue and come back to you as soon as we have any updates

[mrc0mmand]

I can confirm that we (systemd) seem to suffer from the same issue. Some example runs:

• https://github.com/systemd/systemd/actions/runs/8248619545/job/22559325868
• https://github.com/systemd/systemd/actions/runs/8249434220/job/22561812118

Thanks a lot, @igsilya, for finding out the root cause and a potential workaround!

[mikhailkoliada]

Interestingly I always though that the bigger entropy cases a bit less problems than the smaller one, I guess we are going to tune kernel param for the time being, I'll prepare the changes.

[cmigliorini]

sudo sysctl vm.mmap_rnd_bits=28

We can't do this in a running container unless it's priviledged, right? I have an infinite "AddressSanitizer:DEADLYSIGNAL" loop while running gcc10 in a Focal container on a Jammy host (kernel 6.5), that is indeed solved by applying the above command to the host (vm.mmap_rnd_bits is 32 at boot)