Build: "--fully-static" configure option is broken

[emerzon]

Version

Affects all versions from 12.x to 17.3.1
The test below was done with 16.13.2.

Platform

Linux sagan-02 5.13.0-23-generic #23-Ubuntu SMP Fri Nov 26 11:41:15 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Subsystem

No response

What steps will reproduce the bug?

./configure --fully-static; make -j8

How often does it reproduce? Is there a required condition?

Always

What is the expected behavior?

No response

What do you see instead?

g++ -o /home/egomes/a/node-v16.13.2/out/Release/obj.target/libtest_crypto_engine.so -shared -pthread -rdynamic -m64  -Wl,-soname=libtest_crypto_engine.so -Wl,--whole-archive /home/egomes/a/node-v16.13.2/out/Release/obj.target/test_crypto_engine/test/fixtures/test_crypto_engine.o -Wl,--no-whole-archive -static
/usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/11/crtbeginT.o: relocation R_X86_64_32 against hidden symbol `__TMC_END__' can not be used when making a shared object
collect2: error: ld returned 1 exit status
make[1]: *** [test_crypto_engine.target.mk:123: /home/egomes/a/node-v16.13.2/out/Release/obj.target/libtest_crypto_engine.so] Error 1
make[1]: *** Waiting for unfinished jobs....
rm 9fc6b65b733c27999009f69898f8526185d4c971.intermediate
make: *** [Makefile:110: node] Error 2

Additional information

Somehow seems that the "-shared" flag gets out to the above line when it shouldn't.

[mscdex]

FWIW if you're using glibc, you'll never get a truly fully static build. You will need to build against a different libc like musl to get that. I've cross compiled a fully static node with musl in the past with success, but I have not tried with recent node branches.

[emerzon]

FWIW if you're using glibc, you'll never get a truly fully static build. You will need to build against a different libc like musl to get that. I've cross compiled a fully static node with musl in the past with success, but I have not tried with recent node branches.

That's not correct. I have managed to build fully static binaries with glibc using v11.15.0, for instance:

 ./configure --fully-static --with-intl=full-icu --download=all; make -j$(nproc);
...
egomes@sagan-02:~/node-v11.15.0/out/Release$ file node
node: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, BuildID[sha1]=276ec96b60a15310b903e5d7ae065d544e4ec003, for GNU/Linux 3.2.0, stripped

Apparently this feature broke on v12 and was never really fixed since then.

[emerzon]

Also, there is #30180 with fix #30199 which apparently was never merged for v13

[mscdex]

I have managed to build fully static binaries with glibc using v11.15.0, for instance:

What does ldd report for that binary though?

[emerzon]

I have managed to build fully static binaries with glibc using v11.15.0, for instance:

What does ldd report for that binary though?

egomes@sagan-02:~/node-v11.15.0/out/Release$ ldd node
     not a dynamic executable

[emerzon]

There's probably a second issue with the --fully-static switch. Running with make -j1 bypasses the relocation error above, but ends up failing on out/Debug/gen-regexp-special-case just as described in #30180

[dna2github]

current workaround: cd to out folder and manually run the line (remove -static) above failed line (LD_LIBRARY_PATH=/node/out/Release/lib.host:/node/out/Release/lib.target:$LD_LIBRARY_PATH; export LD_LIBRARY_PATH; cd ../tools/v8_gypfiles; mkdir -p /node/out/Release/obj/gen/src/regexp; python ../../deps/v8/tools/run.py "/node/out/Release/gen-regexp-special-case" "/node/out/Release/obj/gen/src/regexp/special-case.cc"); then back to src root and continue the build with make again ...

[emerzon]

current workaround: cd to out folder and manually run the line (remove -static) above failed line (LD_LIBRARY_PATH=/node/out/Release/lib.host:/node/out/Release/lib.target:$LD_LIBRARY_PATH; export LD_LIBRARY_PATH; cd ../tools/v8_gypfiles; mkdir -p /node/out/Release/obj/gen/src/regexp; python ../../deps/v8/tools/run.py "/node/out/Release/gen-regexp-special-case" "/node/out/Release/obj/gen/src/regexp/special-case.cc"); then back to src root and continue the build with make again ...

Thanks for the suggestion!
I am not quite sure which drawbacks this could have, but it did get me further in the build.

Now I am facing another (3rd!?) issue, probably related to the static-pie feature present on glibc versions => 2.27.

make -C out BUILDTYPE=Release V=0
  touch 0d400e6026d015f3b8ab3c23c0a6d3ab4b19fd98.intermediate
  LD_LIBRARY_PATH=/home/egomes/node-v17.3.1/out/Release/lib.host:/home/egomes/node-v17.3.1/out/Release/lib.target:$LD_LIBRARY_PATH; export LD_LIBRARY_PATH; cd ../.; mkdir -p /home/egomes/node-v17.3.1/out/Release/obj/gen/src/node/inspector/protocol; python tools/inspector_protocol/code_generator.py --jinja_dir tools/inspector_protocol --output_base "/home/egomes/node-v17.3.1/out/Release/obj/gen/src/" --config src/inspector/node_protocol_config.json
  touch 650511246299582ab8a88ef89d6e7eb0a60f7e14.intermediate
  LD_LIBRARY_PATH=/home/egomes/node-v17.3.1/out/Release/lib.host:/home/egomes/node-v17.3.1/out/Release/lib.target:$LD_LIBRARY_PATH; export LD_LIBRARY_PATH; cd ../tools/v8_gypfiles; mkdir -p /home/egomes/node-v17.3.1/out/Release/obj/gen/torque-generated/src/objects /home/egomes/node-v17.3.1/out/Release/obj/gen/torque-generated/src/wasm /home/egomes/node-v17.3.1/out/Release/obj/gen/torque-generated/src/debug /home/egomes/node-v17.3.1/out/Release/obj/gen/torque-generated/third_party/v8/builtins /home/egomes/node-v17.3.1/out/Release/obj/gen/torque-generated/test/torque /home/egomes/node-v17.3.1/out/Release/obj/gen/torque-generated/src/builtins /home/egomes/node-v17.3.1/out/Release/obj/gen/torque-generated/src/ic /home/egomes/node-v17.3.1/out/Release/obj/gen/torque-generated; "/home/egomes/node-v17.3.1/out/Release/torque" -o "/home/egomes/node-v17.3.1/out/Release/obj/gen/torque-generated" -v8-root ../../deps/v8 src/builtins/aggregate-error.tq src/builtins/array-at.tq src/builtins/array-concat.tq src/builtins/array-copywithin.tq src/builtins/array-every.tq src/builtins/array-filter.tq src/builtins/array-find.tq src/builtins/array-findindex.tq src/builtins/array-findlast.tq src/builtins/array-findlastindex.tq src/builtins/array-foreach.tq src/builtins/array-from.tq src/builtins/array-isarray.tq src/builtins/array-join.tq src/builtins/array-lastindexof.tq src/builtins/array-map.tq src/builtins/array-of.tq src/builtins/array-reduce-right.tq src/builtins/array-reduce.tq src/builtins/array-reverse.tq src/builtins/array-shift.tq src/builtins/array-slice.tq src/builtins/array-some.tq src/builtins/array-splice.tq src/builtins/array-unshift.tq src/builtins/array.tq src/builtins/arraybuffer.tq src/builtins/base.tq src/builtins/boolean.tq src/builtins/builtins-bigint.tq src/builtins/builtins-string.tq src/builtins/cast.tq src/builtins/collections.tq src/builtins/constructor.tq src/builtins/conversion.tq src/builtins/convert.tq src/builtins/console.tq src/builtins/data-view.tq src/builtins/finalization-registry.tq src/builtins/frames.tq src/builtins/frame-arguments.tq src/builtins/function.tq src/builtins/growable-fixed-array.tq src/builtins/ic-callable.tq src/builtins/ic-dynamic-check-maps.tq src/builtins/ic.tq src/builtins/internal-coverage.tq src/builtins/internal.tq src/builtins/iterator.tq src/builtins/math.tq src/builtins/number.tq src/builtins/object-fromentries.tq src/builtins/object.tq src/builtins/promise-abstract-operations.tq src/builtins/promise-all.tq src/builtins/promise-all-element-closure.tq src/builtins/promise-any.tq src/builtins/promise-constructor.tq src/builtins/promise-finally.tq src/builtins/promise-misc.tq src/builtins/promise-race.tq src/builtins/promise-reaction-job.tq src/builtins/promise-resolve.tq src/builtins/promise-then.tq src/builtins/promise-jobs.tq src/builtins/proxy-constructor.tq src/builtins/proxy-delete-property.tq src/builtins/proxy-get-property.tq src/builtins/proxy-get-prototype-of.tq src/builtins/proxy-has-property.tq src/builtins/proxy-is-extensible.tq src/builtins/proxy-prevent-extensions.tq src/builtins/proxy-revocable.tq src/builtins/proxy-revoke.tq src/builtins/proxy-set-property.tq src/builtins/proxy-set-prototype-of.tq src/builtins/proxy.tq src/builtins/reflect.tq src/builtins/regexp-exec.tq src/builtins/regexp-match-all.tq src/builtins/regexp-match.tq src/builtins/regexp-replace.tq src/builtins/regexp-search.tq src/builtins/regexp-source.tq src/builtins/regexp-split.tq src/builtins/regexp-test.tq src/builtins/regexp.tq src/builtins/string-at.tq src/builtins/string-endswith.tq src/builtins/string-html.tq src/builtins/string-includes.tq src/builtins/string-indexof.tq src/builtins/string-iterator.tq src/builtins/string-match-search.tq src/builtins/string-pad.tq src/builtins/string-repeat.tq src/builtins/string-replaceall.tq src/builtins/string-slice.tq src/builtins/string-startswith.tq src/builtins/string-substr.tq src/builtins/string-substring.tq src/builtins/string-trim.tq src/builtins/symbol.tq src/builtins/torque-internal.tq src/builtins/typed-array-at.tq src/builtins/typed-array-createtypedarray.tq src/builtins/typed-array-every.tq src/builtins/typed-array-entries.tq src/builtins/typed-array-filter.tq src/builtins/typed-array-find.tq src/builtins/typed-array-findindex.tq src/builtins/typed-array-findlast.tq src/builtins/typed-array-findlastindex.tq src/builtins/typed-array-foreach.tq src/builtins/typed-array-from.tq src/builtins/typed-array-keys.tq src/builtins/typed-array-of.tq src/builtins/typed-array-reduce.tq src/builtins/typed-array-reduceright.tq src/builtins/typed-array-set.tq src/builtins/typed-array-slice.tq src/builtins/typed-array-some.tq src/builtins/typed-array-sort.tq src/builtins/typed-array-subarray.tq src/builtins/typed-array-values.tq src/builtins/typed-array.tq src/builtins/weak-ref.tq src/ic/handler-configuration.tq src/objects/allocation-site.tq src/objects/api-callbacks.tq src/objects/arguments.tq src/objects/bigint.tq src/objects/cell.tq src/objects/code.tq src/objects/contexts.tq src/objects/data-handler.tq src/objects/debug-objects.tq src/objects/descriptor-array.tq src/objects/embedder-data-array.tq src/objects/feedback-cell.tq src/objects/feedback-vector.tq src/objects/fixed-array.tq src/objects/foreign.tq src/objects/free-space.tq src/objects/heap-number.tq src/objects/heap-object.tq src/objects/js-array-buffer.tq src/objects/js-array.tq src/objects/js-collection-iterator.tq src/objects/js-collection.tq src/objects/js-function.tq src/objects/js-generator.tq src/objects/js-objects.tq src/objects/js-promise.tq src/objects/js-proxy.tq src/objects/js-regexp-string-iterator.tq src/objects/js-regexp.tq src/objects/js-weak-refs.tq src/objects/literal-objects.tq src/objects/map.tq src/objects/megadom-handler.tq src/objects/microtask.tq src/objects/module.tq src/objects/name.tq src/objects/oddball.tq src/objects/ordered-hash-table.tq src/objects/primitive-heap-object.tq src/objects/promise.tq src/objects/property-array.tq src/objects/property-cell.tq src/objects/property-descriptor-object.tq src/objects/prototype-info.tq src/objects/regexp-match-info.tq src/objects/scope-info.tq src/objects/script.tq src/objects/shared-function-info.tq src/objects/source-text-module.tq src/objects/stack-frame-info.tq src/objects/string.tq src/objects/struct.tq src/objects/swiss-hash-table-helpers.tq src/objects/swiss-name-dictionary.tq src/objects/synthetic-module.tq src/objects/template-objects.tq src/objects/templates.tq src/objects/torque-defined-classes.tq test/torque/test-torque.tq third_party/v8/builtins/array-sort.tq src/objects/intl-objects.tq src/objects/js-break-iterator.tq src/objects/js-collator.tq src/objects/js-date-time-format.tq src/objects/js-display-names.tq src/objects/js-list-format.tq src/objects/js-locale.tq src/objects/js-number-format.tq src/objects/js-plural-rules.tq src/objects/js-relative-time-format.tq src/objects/js-segment-iterator.tq src/objects/js-segmenter.tq src/objects/js-segments.tq src/builtins/wasm.tq src/debug/debug-wasm-objects.tq src/wasm/wasm-objects.tq
  touch 15ebae13038868054c0f7f2ad81251b654d80ee3.intermediate
  LD_LIBRARY_PATH=/home/egomes/node-v17.3.1/out/Release/lib.host:/home/egomes/node-v17.3.1/out/Release/lib.target:$LD_LIBRARY_PATH; export LD_LIBRARY_PATH; cd ../tools/v8_gypfiles; mkdir -p /home/egomes/node-v17.3.1/out/Release/obj/gen/inspector-generated-output-root/include/inspector /home/egomes/node-v17.3.1/out/Release/obj/gen/inspector-generated-output-root/src/inspector/protocol; python ../../deps/v8/third_party/inspector_protocol/code_generator.py --jinja_dir ../../deps/v8/third_party --output_base "/home/egomes/node-v17.3.1/out/Release/obj/gen/inspector-generated-output-root/src/inspector" --config ../../deps/v8/src/inspector/inspector_protocol_config.json --inspector_protocol_dir ../../deps/v8/third_party/inspector_protocol
  clang++ -o /home/egomes/node-v17.3.1/out/Release/obj.target/libtest_crypto_engine.so -shared -pthread -rdynamic -m64  -Wl,-soname=libtest_crypto_engine.so -Wl,--whole-archive /home/egomes/node-v17.3.1/out/Release/obj.target/test_crypto_engine/test/fixtures/test_crypto_engine.o -Wl,--no-whole-archive -static -latomic
/usr/bin/ld: warning: cannot find entry symbol _start; defaulting to 00000000004010e0
/usr/bin/ld: /home/egomes/node-v17.3.1/out/Release/obj.target/test_crypto_engine/test/fixtures/test_crypto_engine.o: in function `bind':
test_crypto_engine.c:(.text+0x11): undefined reference to `ENGINE_set_id'
/usr/bin/ld: test_crypto_engine.c:(.text+0x24): undefined reference to `ENGINE_set_name'
/usr/bin/ld: /home/egomes/node-v17.3.1/out/Release/obj.target/test_crypto_engine/test/fixtures/test_crypto_engine.o: in function `bind_engine':
test_crypto_engine.c:(.text+0x8e): undefined reference to `ENGINE_get_static_state'
/usr/bin/ld: test_crypto_engine.c:(.text+0xa4): undefined reference to `CRYPTO_set_mem_functions'
/usr/bin/ld: test_crypto_engine.c:(.text+0xb3): undefined reference to `ENGINE_set_id'
/usr/bin/ld: test_crypto_engine.c:(.text+0xc6): undefined reference to `ENGINE_set_name'
/usr/bin/ld: /lib/x86_64-linux-gnu/libc.a(dl-reloc-static-pie.o): in function `_dl_relocate_static_pie':
(.text+0x1a): undefined reference to `_DYNAMIC'
/usr/bin/ld: (.text+0x39): undefined reference to `_DYNAMIC'
/usr/bin/ld: /home/egomes/node-v17.3.1/out/Release/obj.target/libtest_crypto_engine.so: hidden symbol `_DYNAMIC' isn't defined
/usr/bin/ld: final link failed: bad value
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[1]: *** [test_crypto_engine.target.mk:126: /home/egomes/node-v17.3.1/out/Release/obj.target/libtest_crypto_engine.so] Error 1
make[1]: *** Waiting for unfinished jobs....
rm 0d400e6026d015f3b8ab3c23c0a6d3ab4b19fd98.intermediate 15ebae13038868054c0f7f2ad81251b654d80ee3.intermediate 650511246299582ab8a88ef89d6e7eb0a60f7e14.intermediate
make: *** [Makefile:110: node] Error 2

[dna2github]

yes, you also need to fix out/test_crypto_engine.target.mk to remove -static; then you can get a static node

[dna2github]

and the manual run has no drawbacks since it just builds gen-regexp-special-case binary file and then the host will run it to generate a .cc source file; then its lifecycle is over ...

[emerzon]

and the manual run has no drawbacks since it just builds gen-regexp-special-case binary file and then the host will run it to generate a .cc source file; then its lifecycle is over ...

Thanks!!!

I have then written a scripted workaround for this build, which works for v16 and v17:

./configure --fully-static --enable-static --download=all --with-intl=full-icu;
for i in out/tools/v8_gypfiles/gen-regexp-special-case.target.mk out/test_crypto_engine.target.mk; do 
sed -i 's/\-static//g' $i || echo "nevermind"; done
make -j$(nproc)

Which did result in a fully-static binary:

egomes@sagan-02:~/node-v17.3.1$ ldd out/Release/node
        not a dynamic executable
egomes@sagan-02:~/node-v17.3.1$ file out/Release/node
out/Release/node: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, BuildID[sha1]=5242bc022d6a7ce2f2e9f363dfb42fc1c5827884, for GNU/Linux 3.2.0, with debug_info, not stripped

Since as you mentioned, these programs are only built to generate another .cpp file, therefore are not part of the actual build, there shouldn't be any drawbacks, however, I am not sure what could be a definitive fix to this.

Perhaps the build system needs to be patched in order not to pass "-static" (or any other 'runtime' flags) to the helper programs.

The other approach would be to investigate the segfault on such binaries when they are built with -static.
This issue seems to be related to symbol ordering issues, as explained in #30180 and also at https://stackoverflow.com/questions/35116327/when-g-static-link-pthread-cause-segmentation-fault-why.
Apparently, the workaround/solution provided by #30199 works only for older versions of glibc, and not the current one (2.34).

[alvarolb]

Thank you very much for the script @emerzon !

[bnoordhuis]

Closing, bug report against EOL version.