Assertion `Data.isValidOffset(Offset) && "wrong LSDA address"' failed.

[derek-zhang]

Hi, Anyone can give some help ? Why LSDA address is wrong?
My binary was compiled by gcc8.2 on Linux,
I built the binary with '-Wl,--emit-relocs' and -fno-reorder-blocks-and-partition

PERF2BOLT: Starting data aggregation job for perf.data.bak
PERF2BOLT: spawning perf job to read branch events
PERF2BOLT: spawning perf job to read mem events
PERF2BOLT: spawning perf job to read process events
PERF2BOLT: spawning perf job to read task events
BOLT-INFO: Target architecture: x86_64
BOLT-INFO: first alloc address is 0x400000
BOLT-INFO: creating new program header table at address 0x3400000, offset 0x3000000
BOLT-INFO: enabling relocation mode
BOLT-WARNING: build-id will not be checked because we could not read one from input binary
BOLT-WARNING: split function detected on input : _ZN3rec5xxxxxx7XXXXXXX8instanceEv.cold.224/1. The support is limited in relocation mode.
PERF2BOLT: waiting for perf mmap events collection to finish...
PERF2BOLT: parsing perf-script mmap events output
PERF2BOLT: waiting for perf task events collection to finish...
PERF2BOLT: parsing perf-script task events output
PERF2BOLT: input binary is associated with 1 PID(s)
PERF2BOLT: waiting for perf events collection to finish...
PERF2BOLT: parse branch events...
PERF2BOLT: read 9910905 samples and 157521236 LBR entries
PERF2BOLT: 0 samples (0.0%) were ignored
PERF2BOLT: traces mismatching disassembled function contents: 7533595 (5.1%)
PERF2BOLT: out of range traces involving unknown regions: 60876241 (41.2%)
BOLT-INFO: forcing -jump-tables=move as PIC jump table was detected in function _ZN3xxx17xxxSSSS14calc_xxxxEmiRdS1_dRKSsiibii
perf2bolt: /ssd2/xxxx/tool/llvm/llvm/tools/llvm-bolt/src/Exceptions.cpp:141: void llvm::bolt::BinaryFunction::parseLSDA(llvm::ArrayRef, uint64_t): Assertion `Data.isValidOffset(Offset) && "wrong LSDA address"' failed.
#0 0x00000000014e1b18 llvm::sys::PrintStackTrace(llvm::raw_ostream&) /ssd2/xxxx/tool/llvm/llvm/lib/Support/Unix/Signals.inc:398:22
#1 0x00000000014e1bab PrintStackTraceSignalHandler(void*) /ssd2/xxxx/tool/llvm/llvm/lib/Support/Unix/Signals.inc:462:1
#2 0x00000000014e004a llvm::sys::RunSignalHandlers() /ssd2/xxxx/tool/llvm/llvm/lib/Support/Signals.cpp:49:19
#3 0x00000000014e148f SignalHandler(int) /ssd2/xxxx/tool/llvm/llvm/lib/Support/Unix/Signals.inc:252:1
#4 0x00007f89e6cb2fa0 __restore_rt (/opt/compiler/gcc-8.2/lib/libpthread.so.0+0x10fa0)
#5 0x00007f89e67fba48 __GI_raise /home/liruihao/mygcc82/glibc-2.21/signal/../sysdeps/unix/sysv/linux/raise.c:55:10
#6 0x00007f89e67e72d9 __GI_abort /home/liruihao/mygcc82/glibc-2.21/stdlib/abort.c:91:7
#7 0x00007f89e67e71af get_sysdep_segment_value /home/liruihao/mygcc82/glibc-2.21/intl/loadmsgcat.c:541:8
#8 0x00007f89e67e71af _nl_load_domain.cold.0 /home/liruihao/mygcc82/glibc-2.21/intl/loadmsgcat.c:1002:34
#9 0x00007f89e67f4992 (/opt/compiler/gcc-8.2/lib/libc.so.6+0x2d992)
#10 0x000000000057963f llvm::bolt::BinaryFunction::parseLSDA(llvm::ArrayRef, unsigned long) /ssd2/xxxxx/tool/llvm/llvm/tools/llvm-bolt/src/Exceptions.cpp:141:3
#11 0x00000000005be797 llvm::bolt::RewriteInstance::disassembleFunctions() /ssd2/xxxx/tool/llvm/llvm/tools/llvm-bolt/src/RewriteInstance.cpp:2765:25
#12 0x00000000005b2358 operator() /ssd2/xxxxx/tool/llvm/llvm/tools/llvm-bolt/src/RewriteInstance.cpp:1052:21
#13 0x00000000005b2358 llvm::bolt::RewriteInstance::run()::'lambda'(std::set<unsigned long, std::less, std::allocator > const&, bool)::operator()(std::set<unsigned long, std::less, std::allocator > const&, bool) const (./perf2bolt+0x5b2358)
#14 0x00000000005b25e3 llvm::bolt::RewriteInstance::run() /ssd2/xxxxx/tool/llvm/llvm/tools/llvm-bolt/src/RewriteInstance.cpp:1089:21
#15 0x000000000040fc6e main /ssd2/xxxxx/tool/llvm/llvm/tools/llvm-bolt/src/llvm-bolt.cpp:312:61
#16 0x00007f89e67e8b8e __libc_start_main /home/liruihao/mygcc82/glibc-2.21/csu/libc-start.c:289:16
#17 0x000000000040e489 _start /home/liruihao/mygcc82/glibc-2.21/csu/../sysdeps/x86_64/start.S:121:0

============

I tried to comment out LSDA code in bolt RewriteInstance.cpp. then I can get a perf2bolt result
But when I try to do llvm-bolt, an error occur as below.

===========

BOLT-WARNING: unable to disassemble instruction at offset 0x53 (address 0x1b459d3) in function K_XX_XX/sha1-x86_64.o/1(*2)
BOLT-WARNING: unable to disassemble instruction at offset 0x1 (address 0x1b46c81) in function K256/sha256-x86_64.o/1(*2)
BOLT-WARNING: unable to disassemble instruction at offset 0x15 (address 0x1b4b715) in function K512/sha512-x86_64.o/1(*2)
BOLT-ERROR: cannot process binaries with unmarked object in code at address 0x1b5bd80 belonging to section .text in relocation mode.

Any help is much appreciated

[maksfb]

Even though you compile with -fno-reorder-blocks-and-partition flag that prevents code splitting, your input binary contains a function _ZN3rec5xxxxxx7XXXXXXX8instanceEv that is split. Could you verify that the module containing this function is compiled with -fno-reorder-blocks-and-partition?

[derek-zhang]

Sure, I recompiled my binary and the depended modules with -fno-reorder-blocks-and-partition
But I still get an error:
BOLT-WARNING: split function detected on input : ssl_get_prev_session.cold.5/1. The support is limited in relocation mode.
I think ssl_get_prev_session is a system/public library openssl and I just link to it by using "-lssl".
So, is that means I have to recompile all the system/public libraries I used ? That's a lot of work.....

very appreciate your help!

PERF2BOLT: Starting data aggregation job for perf.data
PERF2BOLT: spawning perf job to read branch events
PERF2BOLT: spawning perf job to read mem events
PERF2BOLT: spawning perf job to read process events
PERF2BOLT: spawning perf job to read task events
BOLT-INFO: Target architecture: x86_64
BOLT-INFO: first alloc address is 0x400000
BOLT-INFO: creating new program header table at address 0x3400000, offset 0x3000000
BOLT-INFO: enabling relocation mode
BOLT-WARNING: build-id will not be checked because we could not read one from input binary
BOLT-WARNING: split function detected on input : ssl_get_prev_session.cold.5/1. The support is limited in relocation mode.
PERF2BOLT: waiting for perf mmap events collection to finish...
PERF2BOLT: parsing perf-script mmap events output
PERF2BOLT: waiting for perf task events collection to finish...
PERF2BOLT: parsing perf-script task events output
PERF2BOLT: input binary is associated with 1 PID(s)
PERF2BOLT: waiting for perf events collection to finish...
PERF2BOLT: parse branch events...
PERF2BOLT: read 539566 samples and 8301315 LBR entries
PERF2BOLT: 0 samples (0.0%) were ignored
PERF2BOLT: traces mismatching disassembled function contents: 393923 (5.1%)
PERF2BOLT: out of range traces involving unknown regions: 3139369 (40.4%)
BOLT-INFO: forcing -jump-tables=move as PIC jump table was detected in function _ZN3xxx17xxxxxxxx14xxxxxxxEmiRdS1_dRKSsiibii
BOLT-WARNING: 1 collisions detected while hashing binary objects. Use -v=1 to see the list.
PERF2BOLT: processing branch events...
PERF2BOLT: wrote 77376 objects and 0 memory objects to perf.fdata

BTW,

1. should I recompile my gcc8.2 with -fno-reorder-blocks-and-partition ?
2. I compiled my binary using gcc4.8.2 and still get the same LSDA error:

perf2bolt: /ssd2/xxx/tool/llvm/llvm/tools/llvm-bolt/src/Exceptions.cpp:141: void llvm::bolt::BinaryFunction::parseLSDA(llvm::ArrayRef, uint64_t): Assertion `Data.isValidOffset(Offset) && "wrong LSDA address"' failed.

[HShan886]

I get same an assert error, any help will be pleasure

[aaupov]

I get same an assert error, any help will be pleasure

We've recently added cold fragment support that would skip cases that BOLT can't process. So it's likely a different issue. Can you please provide the full log?

[HShan886]

My full log:
PERF2BOLT: Starting data aggregation job for perf.data
PERF2BOLT: spawning perf job to read branch events
PERF2BOLT: spawning perf job to read mem events
PERF2BOLT: spawning perf job to read process events
PERF2BOLT: spawning perf job to read task events
BOLT-INFO: Target architecture: x86_64
BOLT-INFO: BOLT version: 571d6a7120c2b637db2bb46fe3029f9d8576ab86
BOLT-INFO: first alloc address is 0x400000
BOLT-INFO: creating new program header table at address 0x9200000, offset 0x8e00000
BOLT-INFO: disabling -align-macro-fusion in non-relocation mode
BOLT-INFO: enabling lite mode
BOLT-WARNING: __stop_google_malloc (0x4c68aaf) does not have any section
BOLT-INFO: pre-processing profile using perf data aggregator
BOLT-WARNING: build-id will not be checked because we could not read one from input binary
PERF2BOLT: waiting for perf mmap events collection to finish...
PERF2BOLT: parsing perf-script mmap events output
PERF2BOLT: waiting for perf task events collection to finish...
PERF2BOLT: parsing perf-script task events output
PERF2BOLT: input binary is associated with 1 PID(s)
PERF2BOLT: waiting for perf events collection to finish...
PERF2BOLT: parse branch events...
PERF2BOLT: read 720642 samples and 11514877 LBR entries
PERF2BOLT: 0 samples (0.0%) were ignored
PERF2BOLT: traces mismatching disassembled function contents: 0 (0.0%)
PERF2BOLT: out of range traces involving unknown regions: 318112 (2.9%)
BOLT-WARNING: Ignored 0 functions due to cold fragments.
BOLT-INFO: forcing -jump-tables=move as PIC jump table was detected in function _ZN6localtime4yy11bs19ActiveSegment17CompleteEPNS2_15ContextE
perf2bolt: /opt/llvm-project/bolt/lib/Core/Exceptions.cpp:114: void llvm::bolt::BinaryFunction::parseLSDA(llvm::ArrayRef, uint64_t): Assertion `Data.isValidOffset(Offset) && "wrong LSDA address"' failed.
#0 0x00000000011a8845 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) /opt/llvm-project/llvm/lib/Support/Unix/Signals.inc:565:22
#1 0x00000000011a88fc PrintStackTraceSignalHandler(void*) /opt/llvm-project/llvm/lib/Support/Unix/Signals.inc:632:1
#2 0x00000000011a68d7 llvm::sys::RunSignalHandlers() /opt/llvm-project/llvm/lib/Support/Signals.cpp:97:20
#3 0x00000000011a828c SignalHandler(int) /opt/llvm-project/llvm/lib/Support/Unix/Signals.inc:407:1
#4 0x00007f48a5870620 __restore_rt sigaction.c:0:0
#5 0x00007f48a498f277 raise (/lib64/libc.so.6+0x36277)
#6 0x00007f48a4990968 abort (/lib64/libc.so.6+0x37968)
#7 0x00007f48a4988096 __assert_fail_base (/lib64/libc.so.6+0x2f096)
#8 0x00007f48a4988142 (/lib64/libc.so.6+0x2f142)
#9 0x00000000028163ee llvm::bolt::BinaryFunction::parseLSDA(llvm::ArrayRef, unsigned long) /opt/llvm-project/bolt/lib/Core/Exceptions.cpp:114:3
#10 0x0000000000d58448 llvm::bolt::RewriteInstance::disassembleFunctions() /opt/llvm-project/bolt/lib/Rewrite/RewriteInstance.cpp:2763:25
#11 0x0000000000d4c19b llvm::bolt::RewriteInstance::run() /opt/llvm-project/bolt/lib/Rewrite/RewriteInstance.cpp:765:27
#12 0x000000000040bcb9 main /opt/llvm-project/bolt/tools/driver/llvm-bolt.cpp:250:49
#13 0x00007f48a497b445 __libc_start_main (/lib64/libc.so.6+0x22445)
#14 0x000000000040a732 _start (/home/haishan.fhs/bin/llvm-bolt+0x40a732)
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
Stack dump:
0. Program arguments: perf2bolt -p perf.data -o perf.fdata pangu_blockserver --update-debug-sections

[HShan886]

@aaupov
that's great, the code is ready for upstream?

[aaupov]

@aaupov

that's great, the code is ready for upstream?

Hi @Haishan312, sorry for taking so long on this issue. It looks like a separate issue from cold fragment support, so it has not been fixed yet.