LLD 21 crash on Linux 6.17 LTO build

[cgzones]

[...]
  LD [M]  drivers/gpu/drm/amd/amdgpu/amdgpu.o
  AR      drivers/gpu/built-in.a
  AR      drivers/built-in.a
  AR      built-in.a
  AR      vmlinux.a
  GEN     .tmp_initcalls.lds
  LD      vmlinux.o
inlinable function call in a function with debug info must have a !dbg location
  call void @__sanitizer_cov_stack_depth() #294
inlinable function call in a function with debug info must have a !dbg location
  call void @__sanitizer_cov_stack_depth() #294
inlinable function call in a function with debug info must have a !dbg location
  call void @__sanitizer_cov_stack_depth() #294
inlinable function call in a function with debug info must have a !dbg location
  call void @__sanitizer_cov_stack_depth() #294
inlinable function call in a function with debug info must have a !dbg location
  call void @__sanitizer_cov_stack_depth() #294
inlinable function call in a function with debug info must have a !dbg location
  call void @__sanitizer_cov_stack_depth() #294
inlinable function call in a function with debug info must have a !dbg location
  call void @__sanitizer_cov_stack_depth() #294
inlinable function call in a function with debug info must have a !dbg location
  call void @__sanitizer_cov_stack_depth() #294
inlinable function call in a function with debug info must have a !dbg location
  call void @__sanitizer_cov_stack_depth() #294
inlinable function call in a function with debug info must have a !dbg location
  call void @__sanitizer_cov_stack_depth() #294
inlinable function call in a function with debug info must have a !dbg location
  call void @__sanitizer_cov_stack_depth() #294
inlinable function call in a function with debug info must have a !dbg location
  call void @__sanitizer_cov_stack_depth() #294
inlinable function call in a function with debug info must have a !dbg location
  call void @__sanitizer_cov_stack_depth() #294
inlinable function call in a function with debug info must have a !dbg location
  call void @__sanitizer_cov_stack_depth() #294
inlinable function call in a function with debug info must have a !dbg location
  call void @__sanitizer_cov_stack_depth() #294
inlinable function call in a function with debug info must have a !dbg location
  call void @__sanitizer_cov_stack_depth() #294
inlinable function call in a function with debug info must have a !dbg location
  call void @__sanitizer_cov_stack_depth() #294
inlinable function call in a function with debug info must have a !dbg location
  call void @__sanitizer_cov_stack_depth() #294
inlinable function call in a function with debug info must have a !dbg location
  call void @__sanitizer_cov_stack_depth() #294
inlinable function call in a function with debug info must have a !dbg location
  call void @__sanitizer_cov_stack_depth() #294
inlinable function call in a function with debug info must have a !dbg location
  call void @__sanitizer_cov_stack_depth() #294
inlinable function call in a function with debug info must have a !dbg location
  call void @__sanitizer_cov_stack_depth() #294
LLVM ERROR: Broken module found, compilation aborted!
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
Stack dump:
0.      Program arguments: ld.lld-21 -m elf_x86_64 -mllvm -import-instr-limit=5 --compress-debug-sections=zstd -z noexecstack -r -o vmlinux.o -T .tmp_initcalls.lds --whole-archive vmlinux.a --no-whole-archive --start-group --end-group
1.      Running pass "verify" on module "ld-temp.o"
Stack dump without symbol names (ensure you have llvm-symbolizer in your PATH or set the environment var `LLVM_SYMBOLIZER_PATH` to point to it):
0  libLLVM.so.21.1 0x0000785eadc68c2a llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) + 58
1  libLLVM.so.21.1 0x0000785eadc66417 llvm::sys::RunSignalHandlers() + 135
2  libLLVM.so.21.1 0x0000785eadc699d4
3  libc.so.6       0x0000785ea913edf0
4  libc.so.6       0x0000785ea919395c
5  libc.so.6       0x0000785ea913ecc2 gsignal + 18
6  libc.so.6       0x0000785ea91274ac abort + 34
7  libLLVM.so.21.1 0x0000785eadbaf2ab llvm::report_fatal_error(llvm::Twine const&, bool) + 427
8  libLLVM.so.21.1 0x0000785eadbaf0fa
9  libLLVM.so.21.1 0x0000785eade468d1
10 libLLVM.so.21.1 0x0000785eaf634ac1
11 libLLVM.so.21.1 0x0000785eade11f17 llvm::PassManager<llvm::Module, llvm::AnalysisManager<llvm::Module>>::run(llvm::Module&, llvm::AnalysisManager<llvm::Module>&) + 327
12 libLLVM.so.21.1 0x0000785eaf62f908 llvm::lto::opt(llvm::lto::Config const&, llvm::TargetMachine*, unsigned int, llvm::Module&, bool, llvm::ModuleSummaryIndex*, llvm::ModuleSummaryIndex const*, std::vector<unsigned char, std::allocator<unsigned char>> const&) + 6232
13 libLLVM.so.21.1 0x0000785eaf63002b llvm::lto::backend(llvm::lto::Config const&, std::function<llvm::Expected<std::unique_ptr<llvm::CachedFileStream, std::default_delete<llvm::CachedFileStream>>> (unsigned int, llvm::Twine const&)>, unsigned int, llvm::Module&, llvm::ModuleSummaryIndex&) + 203
14 libLLVM.so.21.1 0x0000785eaf61ac6b llvm::lto::LTO::runRegularLTO(std::function<llvm::Expected<std::unique_ptr<llvm::CachedFileStream, std::default_delete<llvm::CachedFileStream>>> (unsigned int, llvm::Twine const&)>) + 1723
15 libLLVM.so.21.1 0x0000785eaf61a11a llvm::lto::LTO::run(std::function<llvm::Expected<std::unique_ptr<llvm::CachedFileStream, std::default_delete<llvm::CachedFileStream>>> (unsigned int, llvm::Twine const&)>, llvm::FileCache) + 634
16 ld.lld-21       0x00005dc0486281a5 lld::elf::BitcodeCompiler::compile() + 1781
17 ld.lld-21       0x00005dc048586c8c void lld::elf::LinkerDriver::compileBitcodeFiles<llvm::object::ELFType<(llvm::endianness)1, true>>(bool) + 220
18 ld.lld-21       0x00005dc04856c404 void lld::elf::LinkerDriver::link<llvm::object::ELFType<(llvm::endianness)1, true>>(llvm::opt::InputArgList&) + 5940
19 ld.lld-21       0x00005dc0485549d6 lld::elf::LinkerDriver::linkerMain(llvm::ArrayRef<char const*>) + 17334
20 ld.lld-21       0x00005dc0485505c1 lld::elf::link(llvm::ArrayRef<char const*>, llvm::raw_ostream&, llvm::raw_ostream&, bool, bool) + 721
21 ld.lld-21       0x00005dc0484782cc lld::unsafeLldMain(llvm::ArrayRef<char const*>, llvm::raw_ostream&, llvm::raw_ostream&, llvm::ArrayRef<lld::DriverDef>, bool) + 1724
22 ld.lld-21       0x00005dc048477512 lld_main(int, char**, llvm::ToolContext const&) + 306
23 ld.lld-21       0x00005dc048477bda main + 106
24 libc.so.6       0x0000785ea9128ca8
25 libc.so.6       0x0000785ea9128d65 __libc_start_main + 133
26 ld.lld-21       0x00005dc048477151 _start + 33
Aborted (core dumped)
make[6]: *** [/build/source-ro/scripts/Makefile.vmlinux_o:72: vmlinux.o] Error 134
make[5]: *** [/build/source-ro/Makefile:1225: vmlinux_o] Error 2
make[4]: *** [debian/rules:80: build-arch] Error 2
dpkg-buildpackage: error: make -f debian/rules binary subprocess returned exit status 2
make[3]: *** [/build/source-ro/scripts/Makefile.package:126: bindeb-pkg] Error 2
make[2]: *** [/build/source-ro/Makefile:1643: bindeb-pkg] Error 2
make[1]: *** [/build/source-ro/Makefile:248: __sub-make] Error 2
make[1]: Leaving directory '/tmp/output/build_dir'
make: *** [Makefile:248: __sub-make] Error 2

.config: config.txt

[nathanchance]

Thanks for the report! A simplified reproducer on top of defconfig:

$ cat kernel/configs/repro.config
CONFIG_LTO_NONE=n
CONFIG_LTO_CLANG_FULL=y
CONFIG_DEBUG_INFO_NONE=n
CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT=y
CONFIG_KSTACK_ERASE=y

$ make -skj"$(nproc)" ARCH=x86_64 LLVM=1 mrproper defconfig repro.config vmlinux
inlinable function call in a function with debug info must have a !dbg location
  call void @__sanitizer_cov_stack_depth() #265
inlinable function call in a function with debug info must have a !dbg location
  call void @__sanitizer_cov_stack_depth() #265
inlinable function call in a function with debug info must have a !dbg location
  call void @__sanitizer_cov_stack_depth() #265
inlinable function call in a function with debug info must have a !dbg location
  call void @__sanitizer_cov_stack_depth() #265
inlinable function call in a function with debug info must have a !dbg location
  call void @__sanitizer_cov_stack_depth() #265
inlinable function call in a function with debug info must have a !dbg location
  call void @__sanitizer_cov_stack_depth() #265
inlinable function call in a function with debug info must have a !dbg location
  call void @__sanitizer_cov_stack_depth() #265
inlinable function call in a function with debug info must have a !dbg location
  call void @__sanitizer_cov_stack_depth() #265
inlinable function call in a function with debug info must have a !dbg location
  call void @__sanitizer_cov_stack_depth() #265
inlinable function call in a function with debug info must have a !dbg location
  call void @__sanitizer_cov_stack_depth() #265
inlinable function call in a function with debug info must have a !dbg location
  call void @__sanitizer_cov_stack_depth() #265
inlinable function call in a function with debug info must have a !dbg location
  call void @__sanitizer_cov_stack_depth() #265
inlinable function call in a function with debug info must have a !dbg location
  call void @__sanitizer_cov_stack_depth() #265
inlinable function call in a function with debug info must have a !dbg location
  call void @__sanitizer_cov_stack_depth() #265
inlinable function call in a function with debug info must have a !dbg location
  call void @__sanitizer_cov_stack_depth() #265
LLVM ERROR: Broken module found, compilation aborted!
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
Stack dump:
0.      Program arguments: ld.lld -m elf_x86_64 -mllvm -import-instr-limit=5 --fatal-warnings -z noexecstack -r -o vmlinux.o -T .tmp_initcalls.lds --whole-archive vmlinux.a --no-whole-archive --start-group --end-group
1.      Running pass "verify" on module "ld-temp.o"
...

cc @kees, this seems related to llvm/llvm-project@c7b2d98? I am not sure why __sanitizer_cov_stack_depth() does not have a proper debug info location though... This has come up before and the fix has been to use InstrumentationIRBuilder, which this change appears to do?

llvm/llvm-project@913f7e9
llvm/llvm-project@4a8b124
llvm/llvm-project@4eef2e3

[kees]

Ah! Yeah, so it looks like I used the wrong IRBuilder? I will see what I need to do to get a proper test -- I assume it'll be close to the examples you gave (thank you!) :)

[nathanchance]

Yeah, so it looks like I used the wrong IRBuilder?

It seems like the path that you added does use InstrumentationIRBuilder but it also looks like the debug location gets set earlier in ModuleSanitizerCoverage::InjectCoverageAtBlock() so maybe that is related?

[kees]

0001-sancov-Fix-stack-depth-tracking-to-use-debug-locatio.patch This probably fixes it, but I'm fighting with my local builds at the moment. I'm about to be in hours of meetings, so I wanted to get this up in case someone else has a moment to test it before I can...

[kees]

<narrator>It did not solve the LTO problem</narrator> I'll keep investigating.

[kees]

I think I've fixed it, but I think LTO inlining the calls (which is what tripped the original bug here) is now tickling excessive stack usage. It looks like more of this bug llvm/llvm-project#38157
I think I'll need to disable the stack usage checking with kstack_erase enabled. I'm seeing:

ld.lld: warning: ../drivers/block/loop.c:1515:0: stack frame size (2136) exceeds limit (2048) in function 'lo_ioctl'