Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BOLT: clang segfault after instrumenting libclang-cpp.so or libLLVM.so when built with -DLLVM_LINK_{CLANG,LLVM}_DYLIB=ON #111530

Open
joebonrichie opened this issue Oct 8, 2024 · 2 comments
Labels

Comments

@joebonrichie
Copy link

joebonrichie commented Oct 8, 2024

Tested as of db1a762, the last worked was with llvm-17

Repro:

$ export LDFLAGS="-Wl,-O1 -Wl,-z,relro -Wl,-z,now -Wl,-z,max-page-size=0x1000 -Wl,-Bsymbolic-functions -Wl,--sort-common -Wl,-z,pack-relative-relocs -Wl,-q"
$ cmake -G Ninja -B stage1 -S llvm -DLLVM_TARGETS_TO_BUILD=X86 -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER=/usr/lib64/ccache/bin/clang -DCMAKE_CXX_COMPILER=/usr/lib64/ccache/bin/clang++ -DCMAKE_ASM_COMPILER=/usr/lib64/ccache/bin/clang -DLLVM_ENABLE_PROJECTS="clang;lld;bolt" -DLLVM_ENABLE_RUNTIMES="compiler-rt" -DCOMPILER_RT_BUILD_SANITIZERS=OFF -DCOMPILER_RT_BUILD_XRAY=OFF -DCOMPILER_RT_BUILD_LIBFUZZER=OFF -DCMAKE_INSTALL_PREFIX=stage1/install -DLLVM_LINK_LLVM_DYLIB=ON -DCLANG_LINK_CLANG_DYLIB=ON
$ ninja -C stage1 install
$ ./stage1/install/bin/llvm-bolt -instrument ./stage1/install/bin/../lib/libclang-cpp.so.20.0git -o ./stage1/install/bin/../lib/libclang-cpp.so.20.0git.instr
BOLT-INFO: shared object or position-independent executable detected
BOLT-INFO: Target architecture: x86_64
BOLT-INFO: BOLT version: db1a76206902e6926464e30cc47249e217405dab
BOLT-INFO: first alloc address is 0x0
BOLT-INFO: creating new program header table at address 0x4200000, offset 0x4200000
BOLT-WARNING: debug info will be stripped from the binary. Use -update-debug-sections to keep it.
BOLT-INFO: enabling relocation mode
BOLT-INFO: forcing -jump-tables=move for instrumentation
BOLT-INFO: enabling lite mode
BOLT-WARNING: Failed to analyze 16 relocations
BOLT-WARNING: 6 collisions detected while hashing binary objects. Use -v=1 to see the list.
BOLT-INFO: 0 out of 112356 functions in the binary (0.0%) have non-empty execution profile
BOLT-INSTRUMENTER: Number of indirect call site descriptors: 39341
BOLT-INSTRUMENTER: Number of indirect call target descriptors: 110954
BOLT-INSTRUMENTER: Number of function descriptors: 110954
BOLT-INSTRUMENTER: Number of branch counters: 1350926
BOLT-INSTRUMENTER: Number of ST leaf node counters: 656205
BOLT-INSTRUMENTER: Number of direct call counters: 0
BOLT-INSTRUMENTER: Total number of counters: 2007131
BOLT-INSTRUMENTER: Total size of counters: 16057048 bytes (static alloc memory)
BOLT-INSTRUMENTER: Total size of string table emitted: 16615228 bytes in file
BOLT-INSTRUMENTER: Total size of descriptors: 133046532 bytes in file
BOLT-INSTRUMENTER: Profile will be saved to file /tmp/prof.fdata
BOLT-INFO: 550535 instructions were shortened
BOLT-INFO: removed 4300 empty blocks
BOLT-INFO: UCE removed 80769 blocks and 4856578 bytes of code
BOLT-INFO: padding code to 0xac00000 to accommodate hot text
BOLT-INFO: output linked against instrumentation runtime library, lib entry point is 0xc93a7a0
BOLT-INFO: clear procedure is 0xc936340
BOLT-INFO: patched build-id (flipped last bit)
BOLT-INFO: setting __bolt_runtime_start to 0xc93a760
BOLT-INFO: setting __bolt_runtime_fini to 0xc93a7a0
BOLT-INFO: setting __hot_start to 0x4400000
BOLT-INFO: setting __hot_end to 0xab075d7
$ mv ./stage1/install/bin/../lib/libclang-cpp.so.20.0git.instr ./stage1/install/bin/../lib/libclang-cpp.so.20.0git
$ ./stage1/install/bin/clang -v
Segmentation fault (core dumped)

backstrace

                Stack trace of thread 307074:
                #0  0x00007fb7b7fd7043 elf_get_dynamic_info (ld-linux-x86-64.so.2 + 0x7043)
                #1  0x00007fb7b7fd7fe1 _dl_map_object (ld-linux-x86-64.so.2 + 0x7fe1)
                #2  0x00007fb7b7fd29b5 openaux (ld-linux-x86-64.so.2 + 0x29b5)
                #3  0x00007fb7b7fd14dd __GI__dl_catch_exception (ld-linux-x86-64.so.2 + 0x14dd)
                #4  0x00007fb7b7fd2d14 _dl_map_object_deps (ld-linux-x86-64.so.2 + 0x2d14)
                #5  0x00007fb7b7ff2a7f dl_main (ld-linux-x86-64.so.2 + 0x22a7f)
                #6  0x00007fb7b7fef343 _dl_sysdep_start (ld-linux-x86-64.so.2 + 0x1f343)
                #7  0x00007fb7b7ff0c7e _dl_start_final (ld-linux-x86-64.so.2 + 0x20c7e)
                #8  0x00007fb7b7fef988 _start (ld-linux-x86-64.so.2 + 0x1f988)
                ELF object binary architecture: AMD x86-64

Lemme know if I can provide any more details.

@github-actions github-actions bot added BOLT clang Clang issues not falling into any other category labels Oct 8, 2024
@EugeneZelenko EugeneZelenko removed the clang Clang issues not falling into any other category label Oct 8, 2024
@joebonrichie
Copy link
Author

bughunter

$ BOLT=./stage1/install/bin/llvm-bolt BOLT_OPTIONS="-v=1" INPUT_BINARY=./stage1/install/lib/libclang-cpp.so.20.0git bolt/utils/bughunter.sh 
Verify input binary passes
  INPUT_BINARY: : &&  ./stage1/install/lib/libclang-cpp.so.20.0git  |& cat >& /tmp/libclang-cpp.so.20.0git.3tX.bolt.out
bolt/utils/bughunter.sh: line 97: 698665 Segmentation fault      (core dumped) $TIMEOUT_OR_CMD $INPUT_BINARY $COMMAND_LINE 2>&1
     698666 Done                    | $POST_COMMAND >&$OUTPUT_FILE
  Warning: input binary failed
Verify optimized binary fails
  Bolt crashes while generating optimized binary.
egrep: warning: egrep is obsolescent; using grep -E
egrep: warning: egrep is obsolescent; using grep -E
Iteration 0, trying /tmp/func-names.eiR.txtaa / 56983 functions
  BOLT: ./stage1/install/bin/llvm-bolt -v=1 ./stage1/install/lib/libclang-cpp.so.20.0git -funcs-file-no-regex=/tmp/func-names.eiR.txtaa -o /tmp/libclang-cpp.so.20.0git.3tX.bolt >& /tmp/bolt3A4.log
  BOLT failure=1
Iteration 1, trying /tmp/func-names.rJG.txtaa / 28492 functions
  BOLT: ./stage1/install/bin/llvm-bolt -v=1 ./stage1/install/lib/libclang-cpp.so.20.0git -funcs-file-no-regex=/tmp/func-names.rJG.txtaa -o /tmp/libclang-cpp.so.20.0git.3tX.bolt >& /tmp/bolt3A4.log
  BOLT failure=1
Iteration 2, trying /tmp/func-names.9rf.txtaa / 14246 functions
  BOLT: ./stage1/install/bin/llvm-bolt -v=1 ./stage1/install/lib/libclang-cpp.so.20.0git -funcs-file-no-regex=/tmp/func-names.9rf.txtaa -o /tmp/libclang-cpp.so.20.0git.3tX.bolt >& /tmp/bolt3A4.log
  BOLT failure=1
Iteration 3, trying /tmp/func-names.OOh.txtaa / 7123 functions
  BOLT: ./stage1/install/bin/llvm-bolt -v=1 ./stage1/install/lib/libclang-cpp.so.20.0git -funcs-file-no-regex=/tmp/func-names.OOh.txtaa -o /tmp/libclang-cpp.so.20.0git.3tX.bolt >& /tmp/bolt3A4.log
  BOLT failure=1
Iteration 4, trying /tmp/func-names.Emo.txtaa / 3562 functions
  BOLT: ./stage1/install/bin/llvm-bolt -v=1 ./stage1/install/lib/libclang-cpp.so.20.0git -funcs-file-no-regex=/tmp/func-names.Emo.txtaa -o /tmp/libclang-cpp.so.20.0git.3tX.bolt >& /tmp/bolt3A4.log
  BOLT failure=1
Iteration 5, trying /tmp/func-names.Cc6.txtaa / 1781 functions
  BOLT: ./stage1/install/bin/llvm-bolt -v=1 ./stage1/install/lib/libclang-cpp.so.20.0git -funcs-file-no-regex=/tmp/func-names.Cc6.txtaa -o /tmp/libclang-cpp.so.20.0git.3tX.bolt >& /tmp/bolt3A4.log
  BOLT failure=1
Iteration 6, trying /tmp/func-names.yo9.txtaa / 891 functions
  BOLT: ./stage1/install/bin/llvm-bolt -v=1 ./stage1/install/lib/libclang-cpp.so.20.0git -funcs-file-no-regex=/tmp/func-names.yo9.txtaa -o /tmp/libclang-cpp.so.20.0git.3tX.bolt >& /tmp/bolt3A4.log
  BOLT failure=1
Iteration 7, trying /tmp/func-names.xZG.txtaa / 446 functions
  BOLT: ./stage1/install/bin/llvm-bolt -v=1 ./stage1/install/lib/libclang-cpp.so.20.0git -funcs-file-no-regex=/tmp/func-names.xZG.txtaa -o /tmp/libclang-cpp.so.20.0git.3tX.bolt >& /tmp/bolt3A4.log
  BOLT failure=1
Iteration 8, trying /tmp/func-names.ZjP.txtaa / 223 functions
  BOLT: ./stage1/install/bin/llvm-bolt -v=1 ./stage1/install/lib/libclang-cpp.so.20.0git -funcs-file-no-regex=/tmp/func-names.ZjP.txtaa -o /tmp/libclang-cpp.so.20.0git.3tX.bolt >& /tmp/bolt3A4.log
  BOLT failure=1
Iteration 9, trying /tmp/func-names.hOm.txtaa / 112 functions
  BOLT: ./stage1/install/bin/llvm-bolt -v=1 ./stage1/install/lib/libclang-cpp.so.20.0git -funcs-file-no-regex=/tmp/func-names.hOm.txtaa -o /tmp/libclang-cpp.so.20.0git.3tX.bolt >& /tmp/bolt3A4.log
  BOLT failure=1
Iteration 10, trying /tmp/func-names.6Gf.txtaa / 56 functions
  BOLT: ./stage1/install/bin/llvm-bolt -v=1 ./stage1/install/lib/libclang-cpp.so.20.0git -funcs-file-no-regex=/tmp/func-names.6Gf.txtaa -o /tmp/libclang-cpp.so.20.0git.3tX.bolt >& /tmp/bolt3A4.log
  BOLT failure=1
Iteration 11, trying /tmp/func-names.Wrg.txtaa / 28 functions
  BOLT: ./stage1/install/bin/llvm-bolt -v=1 ./stage1/install/lib/libclang-cpp.so.20.0git -funcs-file-no-regex=/tmp/func-names.Wrg.txtaa -o /tmp/libclang-cpp.so.20.0git.3tX.bolt >& /tmp/bolt3A4.log
  BOLT failure=1
Iteration 12, trying /tmp/func-names.NdQ.txtaa / 14 functions
  BOLT: ./stage1/install/bin/llvm-bolt -v=1 ./stage1/install/lib/libclang-cpp.so.20.0git -funcs-file-no-regex=/tmp/func-names.NdQ.txtaa -o /tmp/libclang-cpp.so.20.0git.3tX.bolt >& /tmp/bolt3A4.log
  BOLT failure=1
Iteration 13, trying /tmp/func-names.hPG.txtaa / 7 functions
  BOLT: ./stage1/install/bin/llvm-bolt -v=1 ./stage1/install/lib/libclang-cpp.so.20.0git -funcs-file-no-regex=/tmp/func-names.hPG.txtaa -o /tmp/libclang-cpp.so.20.0git.3tX.bolt >& /tmp/bolt3A4.log
  BOLT failure=1
Iteration 14, trying /tmp/func-names.2dz.txtaa / 4 functions
  BOLT: ./stage1/install/bin/llvm-bolt -v=1 ./stage1/install/lib/libclang-cpp.so.20.0git -funcs-file-no-regex=/tmp/func-names.2dz.txtaa -o /tmp/libclang-cpp.so.20.0git.3tX.bolt >& /tmp/bolt3A4.log
  BOLT failure=1
Iteration 15, trying /tmp/func-names.YxM.txtaa / 2 functions
  BOLT: ./stage1/install/bin/llvm-bolt -v=1 ./stage1/install/lib/libclang-cpp.so.20.0git -funcs-file-no-regex=/tmp/func-names.YxM.txtaa -o /tmp/libclang-cpp.so.20.0git.3tX.bolt >& /tmp/bolt3A4.log
  BOLT failure=1
Iteration 16, trying /tmp/func-names.Zte.txtaa / 1 functions
  BOLT: ./stage1/install/bin/llvm-bolt -v=1 ./stage1/install/lib/libclang-cpp.so.20.0git -funcs-file-no-regex=/tmp/func-names.Zte.txtaa -o /tmp/libclang-cpp.so.20.0git.3tX.bolt >& /tmp/bolt3A4.log
  BOLT failure=1
The function(s) that failed are in /tmp/func-names.xv2.txtaa
To reproduce, run: ./stage1/install/bin/llvm-bolt -v=1 ./stage1/install/lib/libclang-cpp.so.20.0git -funcs-file-no-regex=/tmp/func-names.Zte.txtaa -o /tmp/libclang-cpp.so.20.0git.3tX.bolt
rm: cannot remove '/tmp/libclang-cpp.so.20.0git.3tX.bolt': No such file or directory
rm: cannot remove '/tmp/libclang-cpp.so.20.0git.3tX.bolt.out': No such file or directory
[ble: exit 1]
 $ cat /tmp/func-names.xv2.txtaa -p
__bolt_fini_trampoline

@joebonrichie
Copy link
Author

joebonrichie commented Oct 15, 2024

bisected to 0053cb8

partial revert of 0053cb8 fixes clang segfaulting after libclang/libLLVM has been instrumented

diff --git a/bolt/lib/Rewrite/RewriteInstance.cpp b/bolt/lib/Rewrite/RewriteInstance.cpp
index 32ec7abe8b66..f7f63b2ea956 100644
--- a/bolt/lib/Rewrite/RewriteInstance.cpp
+++ b/bolt/lib/Rewrite/RewriteInstance.cpp
@@ -5068,11 +5068,9 @@ void RewriteInstance::patchELFAllocatableRelrSection(
   const uint8_t PSize = BC->AsmInfo->getCodePointerSize();
   const uint64_t MaxDelta = ((CHAR_BIT * DynamicRelrEntrySize) - 1) * PSize;
 
-  auto FixAddend = [&](const BinarySection &Section, const Relocation &Rel,
-                       uint64_t FileOffset) {
+  auto FixAddend = [&](const BinarySection &Section, const Relocation &Rel) {
     // Fix relocation symbol value in place if no static relocation found
-    // on the same address. We won't check the BF relocations here since it
-    // is rare case and no optimization is required.
+    // on the same address
     if (Section.getRelocationAt(Rel.Offset))
       return;
 
@@ -5081,6 +5079,10 @@ void RewriteInstance::patchELFAllocatableRelrSection(
     if (!Addend)
       return;
 
+    uint64_t FileOffset = Section.getOutputFileOffset();
+    if (!FileOffset)
+      FileOffset = Section.getInputFileOffset();
+    FileOffset += Rel.Offset;
     OS.pwrite(reinterpret_cast<const char *>(&Addend), PSize, FileOffset);
   };
 
@@ -5102,7 +5104,7 @@ void RewriteInstance::patchELFAllocatableRelrSection(
       RelOffset = RelOffset == 0 ? SectionAddress + Rel.Offset : RelOffset;
       assert((RelOffset & 1) == 0 && "Wrong relocation offset");
       RelOffsets.emplace(RelOffset);
-      FixAddend(Section, Rel, RelOffset);
+      FixAddend(Section, Rel);
     }
   }
 
-- 

@yota9 hopefully you don't mind the ping

joebonrichie added a commit to getsolus/packages that referenced this issue Oct 16, 2024
**Summary**
- llvm/llvm-project#111530
- This looks like a AArch64 specific fix and i'm assuming for now it's
  fine on X86.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants