Skip to content
This repository has been archived by the owner on Jul 1, 2023. It is now read-only.

perf2bolt crashes with Assertion #10

Closed
suresh-srinivas opened this issue Jun 21, 2018 · 11 comments
Closed

perf2bolt crashes with Assertion #10

suresh-srinivas opened this issue Jun 21, 2018 · 11 comments
Assignees
@aaupov @maksfb

Comments

@suresh-srinivas
Copy link

suresh-srinivas commented Jun 21, 2018 •
edited
Loading

Thank you for open sourcing this.

Looks like I am missing something. I built the binary with '-Wl,--emit-relocs',

Software versions

gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.9) 
Ubuntu 16.04
labuser@y004:~/ssuresh/node$ uname -a
Linux y004 4.8.0-58-generic #63~16.04.1-Ubuntu SMP Mon Jun 26 18:08:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
labuser@y004:~/ssuresh/node$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 16.04.4 LTS
Release:        16.04
Codename:       xenial

I also have one function in a different section '.lpstub'.

labuser@y004:~/ssuresh$ perf2bolt -p perf.data -o perf.fdata node/node
PERF2BOLT: Starting data aggregation job for perf.data
PERF2BOLT: Spawning perf-script job to read branch events
PERF2BOLT: Spawning perf-script job to read mem events
PERF2BOLT: Spawning perf-script job to read tasks
BOLT-INFO: Target architecture: x86_64
BOLT-INFO: binary build-id is:     605ef65b6b623ad564a2cfd14f929c6c1b0cc223
PERF2BOLT: matched build-id and file name
BOLT-INFO: first alloc address is 0x400000
BOLT-INFO: creating new program header table at address 0x2600000, offset 0x2200000
BOLT-INFO: enabling relocation mode
perf2bolt: /home/labuser/ssuresh/llvm/tools/llvm-bolt/src/RewriteInstance.cpp:1381: void llvm::bolt::RewriteInstance::discoverFileObjects(): Assertion `Section && "section for functions must be registered."' failed.
LLVMSymbolizer: error reading file: No such file or directory
#0 0x000000000253f805 (perf2bolt+0x253f805)
#1 0x000000000253f898 (perf2bolt+0x253f898)
#2 0x000000000253da81 (perf2bolt+0x253da81)
#3 0x000000000253f07a (perf2bolt+0x253f07a)
#4 0x00007ffff7bcb390 __restore_rt (/lib/x86_64-linux-gnu/libpthread.so.0+0x11390)
#5 0x00007ffff693d428 gsignal /build/glibc-Cl5G7W/glibc-2.23/signal/../sysdeps/unix/sysv/linux/raise.c:54:0
#6 0x00007ffff693f02a abort /build/glibc-Cl5G7W/glibc-2.23/stdlib/abort.c:91:0
#7 0x00007ffff6935bd7 __assert_fail_base /build/glibc-Cl5G7W/glibc-2.23/assert/assert.c:92:0
#8 0x00007ffff6935c82 (/lib/x86_64-linux-gnu/libc.so.6+0x2dc82)
#9 0x0000000000584fa5 (perf2bolt+0x584fa5)
#10 0x000000000058199e (perf2bolt+0x58199e)
#11 0x0000000000581d09 (perf2bolt+0x581d09)
#12 0x0000000000412248 (perf2bolt+0x412248)
#13 0x00007ffff6928830 __libc_start_main /build/glibc-Cl5G7W/glibc-2.23/csu/../csu/libc-start.c:325:0
#14 0x0000000000410e49 (perf2bolt+0x410e49)
Stack dump:
0.      Program arguments: perf2bolt -p perf.data -o perf.fdata node/node 
Aborted (core dumped)

I have the following sections in the executable

    0x0000000000400270 - 0x000000000040028c is .interp
    0x000000000040028c - 0x00000000004002ac is .note.ABI-tag
    0x00000000004002ac - 0x00000000004002d0 is .note.gnu.build-id
    0x00000000004002d0 - 0x0000000000455774 is .gnu.hash
    0x0000000000455778 - 0x0000000000565320 is .dynsym
    0x0000000000565320 - 0x0000000000847760 is .dynstr
    0x0000000000847760 - 0x000000000085e1ae is .gnu.version
    0x000000000085e1b0 - 0x000000000085e3c0 is .gnu.version_r
    0x000000000085e3c0 - 0x0000000000860d48 is .rela.dyn
    0x0000000000860d48 - 0x0000000000860f28 is .rela.plt
    0x0000000000860f28 - 0x0000000000860f47 is .init
    0x0000000000860f50 - 0x00000000008610a0 is .plt
    0x00000000008610a0 - 0x0000000000861da8 is .plt.got
    0x0000000000a00000 - 0x0000000000a0018b is .lpstub
    0x0000000000a01000 - 0x00000000018cc669 is .text
    0x00000000018cc66c - 0x00000000018cc675 is .fini
    0x00000000018cc680 - 0x00000000020f7b88 is .rodata
    0x00000000020f7b88 - 0x000000000214c40c is .eh_frame_hdr
    0x000000000214c410 - 0x0000000002307a9c is .eh_frame
    0x0000000002507f60 - 0x0000000002507f64 is .tbss
    0x0000000002507f60 - 0x0000000002507fc8 is .init_array
    0x0000000002507fc8 - 0x0000000002507fe0 is .fini_array
    0x0000000002507fe0 - 0x0000000002507fe8 is .jcr
    0x0000000002507fe8 - 0x0000000002508238 is .dynamic
    0x0000000002508238 - 0x0000000002509000 is .got
    0x0000000002509000 - 0x000000000251f4f0 is .data
    0x000000000251f4f0 - 0x000000000251f4f0 is .tm_clone_table
    0x000000000251f500 - 0x0000000002537950 is .bss
@dstogov dstogov mentioned this issue Jun 21, 2018
Open
@suresh-srinivas
Copy link
Author

I built a version with out the .lpstub I am now getting a different error

PERF2BOLT: Starting data aggregation job for perf.data
PERF2BOLT: Spawning perf-script job to read branch events
PERF2BOLT: Spawning perf-script job to read mem events
PERF2BOLT: Spawning perf-script job to read tasks
BOLT-INFO: Target architecture: x86_64
BOLT-INFO: binary build-id is:     dd6035b881b5f47e1d349ed461bbea99d33db93d
PERF2BOLT: matched build-id and file name
BOLT-INFO: first alloc address is 0x400000
BOLT-INFO: creating new program header table at address 0x2400000, offset 0x2000000
BOLT-INFO: enabling relocation mode
BOLT-ERROR: cannot process binaries with unmarked object in code at address 0x1430380 belonging to section .text in relocation mode.

looking at the executable shows the following

        Entry point: 0x866de0
        0x0000000000400270 - 0x000000000040028c is .interp
        0x000000000040028c - 0x00000000004002ac is .note.ABI-tag
        0x00000000004002ac - 0x00000000004002d0 is .note.gnu.build-id
        0x00000000004002d0 - 0x0000000000455740 is .gnu.hash
        0x0000000000455740 - 0x00000000005651b0 is .dynsym
        0x00000000005651b0 - 0x00000000008473fc is .dynstr
        0x00000000008473fc - 0x000000000085de30 is .gnu.version
        0x000000000085de30 - 0x000000000085e040 is .gnu.version_r
        0x000000000085e040 - 0x0000000000860908 is .rela.dyn
        0x0000000000860908 - 0x0000000000860ad0 is .rela.plt
        0x0000000000860ad0 - 0x0000000000860aef is .init
        0x0000000000860af0 - 0x0000000000860c30 is .plt
        0x0000000000860c30 - 0x0000000000861908 is .plt.got
        0x0000000000862000 - 0x000000000172c669 is .text
        0x000000000172c66c - 0x000000000172c675 is .fini
        0x000000000172c680 - 0x0000000001f57a48 is .rodata
        0x0000000001f57a48 - 0x0000000001fac294 is .eh_frame_hdr
        0x0000000001fac298 - 0x0000000002167804 is .eh_frame
        0x0000000002367fa0 - 0x0000000002367fa4 is .tbss
        0x0000000002367fa0 - 0x0000000002368000 is .init_array
        0x0000000002368000 - 0x0000000002368018 is .fini_array
        0x0000000002368018 - 0x0000000002368020 is .jcr
        0x0000000002368020 - 0x0000000002368270 is .dynamic
        0x0000000002368270 - 0x0000000002369000 is .got
        0x0000000002369000 - 0x000000000237f4f0 is .data
        0x000000000237f4f0 - 0x000000000237f4f0 is .tm_clone_table
        0x000000000237f500 - 0x00000000023978d0 is .bss

@danielcdh danielcdh mentioned this issue Jun 21, 2018
Open
@suresh-srinivas
Copy link
Author

I also noticed that .rela.text is not present in my binary, only .rela.dyn and .rela.plt.

@maksfb maksfb self-assigned this Jun 21, 2018
@maksfb
Copy link
Contributor

maksfb commented Jun 21, 2018

I think the dump includes allocatable sections only, that's why.rela.text is missing. Based on the output, BOLT did detect .rela.text.

@suresh-srinivas
Copy link
Author

I was able to debug the second problem. BOLT stops on one of the assembly routines in openssl. I was able to solve the problem temporarily by using openssl without the assembly routines. perf2bolt and llvm-bolt suceeds. On on microbenchmark, I am getting 3% performance improvement. I will try other workloads and report. Let me also try the original problem which was the binary with 2 text segments.

@suresh-srinivas
Copy link
Author

I am leaving this bug open. The original problem of two text sections still seem to fail.

perf2bolt: /home/labuser/ssuresh/llvm/tools/llvm-bolt/src/RewriteInstance.cpp:1415: void llvm::bolt::RewriteInstance::discoverFileObjects(): Assertion `Section && "section for functions must be registered."' failed.
#0 0x0000000002563aa3 llvm::sys::PrintStackTrace(llvm::raw_ostream&) /home/labuser/ssuresh/llvm/lib/Support/Unix/Signals.inc:398:0
#1 0x0000000002563b36 PrintStackTraceSignalHandler(void*) /home/labuser/ssuresh/llvm/lib/Support/Unix/Signals.inc:462:0
#2 0x0000000002561d1f llvm::sys::RunSignalHandlers() /home/labuser/ssuresh/llvm/lib/Support/Signals.cpp:49:0
#3 0x0000000002563318 SignalHandler(int) /home/labuser/ssuresh/llvm/lib/Support/Unix/Signals.inc:252:0
#4 0x00007f76f27f7390 __restore_rt (/lib/x86_64-linux-gnu/libpthread.so.0+0x11390)
#5 0x00007f76f1569428 gsignal /build/glibc-Cl5G7W/glibc-2.23/signal/../sysdeps/unix/sysv/linux/raise.c:54:0
#6 0x00007f76f156b02a abort /build/glibc-Cl5G7W/glibc-2.23/stdlib/abort.c:91:0
#7 0x00007f76f1561bd7 __assert_fail_base /build/glibc-Cl5G7W/glibc-2.23/assert/assert.c:92:0
#8 0x00007f76f1561c82 (/lib/x86_64-linux-gnu/libc.so.6+0x2dc82)
#9 0x000000000059d0af llvm::bolt::RewriteInstance::discoverFileObjects() /home/labuser/ssuresh/llvm/tools/llvm-bolt/src/RewriteInstance.cpp:1416:0
#10 0x00000000005999b0 operator() /home/labuser/ssuresh/llvm/tools/llvm-bolt/src/RewriteInstance.cpp:988:0

This executable has two text sections .lpstub and .text

(gdb) info file
Symbols from "/home/labuser/ssuresh/node-js-large-pages/out/Release/node".
Local exec file:
        `/home/labuser/ssuresh/node-js-large-pages/out/Release/node', 
        file type elf64-x86-64.
        Entry point: 0xa062d0
        0x0000000000400270 - 0x000000000040028c is .interp
        0x000000000040028c - 0x00000000004002ac is .note.ABI-tag
        0x00000000004002ac - 0x00000000004002d0 is .note.gnu.build-id
        0x00000000004002d0 - 0x0000000000461090 is .gnu.hash
        0x0000000000461090 - 0x00000000005862e0 is .dynsym
        0x00000000005862e0 - 0x00000000008bebf1 is .dynstr
        0x00000000008bebf2 - 0x00000000008d72ce is .gnu.version
        0x00000000008d72d0 - 0x00000000008d7500 is .gnu.version_r
        0x00000000008d7500 - 0x00000000008d9fa8 is .rela.dyn
        0x00000000008d9fa8 - 0x00000000008da1a0 is .rela.plt
        0x00000000008da1a0 - 0x00000000008da1ba is .init
        0x00000000008da1c0 - 0x00000000008da320 is .plt
        0x00000000008da320 - 0x00000000008db088 is .plt.got
        0x0000000000a00000 - 0x0000000000a0020e is .lpstub
        0x0000000000a00210 - 0x000000000196a369 is .text
        0x000000000196a36c - 0x000000000196a375 is .fini
        0x000000000196a380 - 0x00000000021e35f8 is .rodata

@railkar railkar mentioned this issue Jul 22, 2019
Closed
@derek-zhang derek-zhang mentioned this issue Aug 1, 2019
Open
@derek-zhang
Copy link

Hi, I have a same bolt error:
BOLT-ERROR: cannot process binaries with unmarked object in code at address 0xa13b80 belonging to section .text in relocation mode.

and my binary does have .rela.text section
could you give some help ?
how did you find "BOLT stops on one of the assembly routines in openssl" ?

Thanks

@uttampawar uttampawar mentioned this issue Feb 14, 2020
Closed
@aaupov
Copy link
Contributor

aaupov commented Oct 8, 2020

Hi @suresh-srinivas, the problem no longer reproduces with the recent BOLT. I've managed to successfully optimize NodeJS with BOLT and posted my notes here: https://aaupov.github.io/blog/2020/10/08/bolt-nodejs.

I think we can close this issue. Please open a new one if you run into any further problems.
@maksfb

@suresh-srinivas
Copy link
Author

Thanks @aaupov . I will check it out. The original problem I had was with OpenSSL. Was your nodejs statically built?

@aaupov
Copy link
Contributor

aaupov commented Oct 8, 2020

@suresh-srinivas, no, I wasn't using static flags, let me double check.

@aaupov
Copy link
Contributor

aaupov commented Oct 13, 2020

I've checked static builds, seems like the original issue with OpenSSL doesn't reproduce with recent NodeJS and BOLT.
More details:

  • --fully-static node doesn't even build. There's a PR30199 with proposed fixes that solve the build problem. However, the resulting binary is not fully functional due to this glibc issue. Still, I was able to collect profile from this build and run it through perf2bolt successfully (with added -strict=0 option). Optimized binary crashes due to another issue, keeping track of it separately.
  • --partly-static node builds and runs tests successfully. perf2bolt works fine with this binary (also with -strict=0). Optimized binary works (passes make jstest).

@ghost ghost mentioned this issue Oct 18, 2020
Closed
@dongAxis dongAxis mentioned this issue Apr 20, 2021
Closed
@ccx1225 ccx1225 mentioned this issue Aug 24, 2021
Open
@HShan886 HShan886 mentioned this issue Aug 26, 2021
Closed
@mgrice mgrice mentioned this issue Sep 17, 2021
Open
@nadavrot nadavrot mentioned this issue Oct 27, 2021
Closed
@valuenumbering valuenumbering mentioned this issue Dec 13, 2021
Closed
@dafoerx dafoerx mentioned this issue Jan 4, 2022
Open
@DreamTeamWangbowen DreamTeamWangbowen mentioned this issue Mar 3, 2022
Open
@aaupov
Copy link
Contributor

aaupov commented Jan 19, 2023

Closing the issue as BOLT is able to process Node binary.
If you run into any other problem, please open a new issue on LLVM's issue tracker: https://github.com/llvm/llvm-project

@aaupov aaupov closed this as completed Jan 19, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@aaupov aaupov

@maksfb maksfb

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@aaupov @derek-zhang @maksfb @suresh-srinivas