Building Godot for web with LTO fails as of Emscripten 3.1.52 (works in 3.1.39)

[haasal]

Tested versions

• Master branch godot (commit 5034478)
• emcc (Emscripten gcc/clang-like replacement + linker emulating GNU ld) 3.1.52 (fa478400df3351f7153c0279bc638784d3d90334)
• Python 3.10.12
• scons v4.0.1

System information

Ubuntu server 22 LTS

Issue description

When trying to compile godot from source for web with lto (emcc was activated and in PATH)

scons platform=web target=template_release lto=full

the project compiles fine till the linking step where em++ throws a massive error:

em++: error: '/home/REDACTED/emsdk/upstream/bin/wasm-ld -o bin/godot.web.template_release.wasm32.wasm -s platform/web/audio_driver_web.web.template_release.wasm32.o platform/web/display_server_web.web.template_release.wasm32.o platform/web/http_client_web.web.template_release.wasm32.o platform/web/javascript_bridge_singleton.web.template_release.wasm32.o platform/web/web_main.web.template_release.wasm32.o platform/web/os_web.web.template_release.wasm32.o platform/web/api/web_tools_editor_plugin.web.template_release.wasm32.o platform/web/web_runtime.web.template_release.wasm32.o main/libmain.web.template_release.wasm32.a modules/libmodules.web.template_release.wasm32.a modules/libmodule_ktx.web.template_release.wasm32.a modules/libmodule_msdfgen.web.template_release.wasm32.a modules/libmodule_theora.web.template_release.wasm32.a modules/libmodule_vorbis.web.template_release.wasm32.a modules/libmodule_basis_universal.web.template_release.wasm32.a modules/libmodule_bmp.web.template_release.wasm32.a modules/libmodule_csg.web.template_release.wasm32.a modules/libmodule_dds.web.template_release.wasm32.a modules/libmodule_enet.web.template_release.wasm32.a modules/libmodule_freetype.web.template_release.wasm32.a modules/libmodule_gdscript.web.template_release.wasm32.a modules/libmodule_gltf.web.template_release.wasm32.a modules/libmodule_gridmap.web.template_release.wasm32.a modules/libmodule_hdr.web.template_release.wasm32.a modules/libmodule_jpg.web.template_release.wasm32.a modules/libmodule_jsonrpc.web.template_release.wasm32.a modules/libmodule_mbedtls.web.template_release.wasm32.a modules/libmodule_meshoptimizer.web.template_release.wasm32.a modules/libmodule_minimp3.web.template_release.wasm32.a modules/libmodule_mobile_vr.web.template_release.wasm32.a modules/libmodule_multiplayer.web.template_release.wasm32.a modules/libmodule_navigation.web.template_release.wasm32.a modules/libmodule_noise.web.template_release.wasm32.a modules/libmodule_ogg.web.template_release.wasm32.a modules/libmodule_regex.web.template_release.wasm32.a modules/libmodule_squish.web.template_release.wasm32.a modules/libmodule_svg.web.template_release.wasm32.a modules/libmodule_text_server_adv.web.template_release.wasm32.a modules/libmodule_tga.web.template_release.wasm32.a modules/libmodule_upnp.web.template_release.wasm32.a modules/libmodule_vhacd.web.template_release.wasm32.a modules/libmodule_webp.web.template_release.wasm32.a modules/libmodule_webrtc.web.template_release.wasm32.a modules/libmodule_websocket.web.template_release.wasm32.a modules/libmodule_webxr.web.template_release.wasm32.a modules/libmodule_zip.web.template_release.wasm32.a platform/libplatform.web.template_release.wasm32.a drivers/libdrivers.web.template_release.wasm32.a scene/libscene.web.template_release.wasm32.a servers/libservers.web.template_release.wasm32.a core/libcore.web.template_release.wasm32.a modules/text_server_adv/libharfbuzz_builtin.web.template_release.wasm32.a modules/text_server_adv/libgraphite_builtin.web.template_release.wasm32.a modules/text_server_adv/libicu_builtin.web.template_release.wasm32.a modules/freetype/libfreetype_builtin.web.template_release.wasm32.a modules/msdfgen/libmsdfgen_builtin.web.template_release.wasm32.a -L/home/alex/code/projects/misc/emsdk/upstream/emscripten/cache/sysroot/lib/wasm32-emscripten/lto /home/alex/code/projects/misc/emsdk/upstream/emscripten/cache/sysroot/lib/wasm32-emscripten/lto/crtbegin.o -lGL-mt-webgl2-ofb-getprocaddr -lal -lhtml5 -lbulkmemory -lstubs-debug -lc-mt-debug -ldlmalloc-mt -lcompiler_rt-mt -lc++-mt-noexcept -lc++abi-debug-mt-noexcept -lsockets-mt --keep-section=target_features -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr /tmp/tmpfwiepyhplibemscripten_js_symbols.so --import-memory --shared-memory --strip-debug --export=_emscripten_thread_init --export=_emscripten_thread_exit --export=_emscripten_thread_crashed --export=_emscripten_tls_init --export=pthread_self --export=emscripten_stack_get_end --export=emscripten_stack_get_free --export=emscripten_stack_get_base --export=emscripten_stack_get_current --export=emscripten_stack_init --export=stackAlloc --export=stackSave --export=stackRestore --export=_emscripten_thread_free_data --export=emscripten_main_runtime_thread_id --export=emscripten_main_thread_process_queued_calls --export=_emscripten_run_on_main_thread_js --export=emscripten_stack_set_limits --export=__get_temp_ret --export=__set_temp_ret --export=__funcs_on_exit --export=__wasm_call_ctors --export=_emscripten_thread_init --export=_emscripten_thread_exit --export-if-defined=__start_em_asm --export-if-defined=__stop_em_asm --export-if-defined=__start_em_lib_deps --export-if-defined=__stop_em_lib_deps --export-if-defined=__start_em_js --export-if-defined=__stop_em_js --export-if-defined=main --export-if-defined=__main_argc_argv --export-if-defined=fflush --export-table -z stack-size=5242880 --initial-memory=33554432 --max-memory=2147483648 --no-entry --stack-first --table-base=1' failed (received SIGKILL (-9))

Also regular compiling without the LTO flag works without a problem

Steps to reproduce

• git clone https://github.com/godotengine/godot
• activate emcc
• compile

Minimal reproduction project (MRP)

...

[akien-mga]

I can reproduce the issue with Emscripten 3.1.52 (latest), but not with 3.1.39 (the version we use currently for official builds and CI).

So it's a recent change upstream that broke LTO (again...).

wasm-ld: error: /home/akien/Projects/godot/emscripten/emsdk/upstream/emscripten/cache/sysroot/lib/wasm32-emscripten/lto/libhtml5.a(callback.o): attempt to add bitcode file after LTO (_emscripten_run_callback_on_thread)
em++: error: '/home/akien/Projects/godot/emscripten/emsdk/upstream/bin/wasm-ld -o bin/godot.web.template_release.wasm32.wasm -s platform/web/audio_driver_web.web.template_release.wasm32.o platform/web/display_server_web.web.template_release.wasm32.o platform/web/http_client_web.web.template_release.wasm32.o platform/web/javascript_bridge_singleton.web.template_release.wasm32.o platform/web/web_main.web.template_release.wasm32.o platform/web/os_web.web.template_release.wasm32.o platform/web/api/web_tools_editor_plugin.web.template_release.wasm32.o platform/web/web_runtime.web.template_release.wasm32.o main/libmain.web.template_release.wasm32.a modules/libmodules.web.template_release.wasm32.a modules/libmodule_ktx.web.template_release.wasm32.a modules/libmodule_msdfgen.web.template_release.wasm32.a modules/libmodule_theora.web.template_release.wasm32.a modules/libmodule_vorbis.web.template_release.wasm32.a modules/libmodule_basis_universal.web.template_release.wasm32.a modules/libmodule_bmp.web.template_release.wasm32.a modules/libmodule_csg.web.template_release.wasm32.a modules/libmodule_dds.web.template_release.wasm32.a modules/libmodule_enet.web.template_release.wasm32.a modules/libmodule_freetype.web.template_release.wasm32.a modules/libmodule_gdscript.web.template_release.wasm32.a modules/libmodule_gltf.web.template_release.wasm32.a modules/libmodule_gridmap.web.template_release.wasm32.a modules/libmodule_hdr.web.template_release.wasm32.a modules/libmodule_jpg.web.template_release.wasm32.a modules/libmodule_jsonrpc.web.template_release.wasm32.a modules/libmodule_mbedtls.web.template_release.wasm32.a modules/libmodule_meshoptimizer.web.template_release.wasm32.a modules/libmodule_minimp3.web.template_release.wasm32.a modules/libmodule_mobile_vr.web.template_release.wasm32.a modules/libmodule_multiplayer.web.template_release.wasm32.a modules/libmodule_navigation.web.template_release.wasm32.a modules/libmodule_noise.web.template_release.wasm32.a modules/libmodule_ogg.web.template_release.wasm32.a modules/libmodule_regex.web.template_release.wasm32.a modules/libmodule_squish.web.template_release.wasm32.a modules/libmodule_svg.web.template_release.wasm32.a modules/libmodule_text_server_adv.web.template_release.wasm32.a modules/libmodule_tga.web.template_release.wasm32.a modules/libmodule_upnp.web.template_release.wasm32.a modules/libmodule_vhacd.web.template_release.wasm32.a modules/libmodule_webp.web.template_release.wasm32.a modules/libmodule_webrtc.web.template_release.wasm32.a modules/libmodule_websocket.web.template_release.wasm32.a modules/libmodule_webxr.web.template_release.wasm32.a modules/libmodule_zip.web.template_release.wasm32.a platform/libplatform.web.template_release.wasm32.a drivers/libdrivers.web.template_release.wasm32.a scene/libscene.web.template_release.wasm32.a servers/libservers.web.template_release.wasm32.a core/libcore.web.template_release.wasm32.a modules/text_server_adv/libharfbuzz_builtin.web.template_release.wasm32.a modules/text_server_adv/libgraphite_builtin.web.template_release.wasm32.a modules/text_server_adv/libicu_builtin.web.template_release.wasm32.a modules/freetype/libfreetype_builtin.web.template_release.wasm32.a modules/msdfgen/libmsdfgen_builtin.web.template_release.wasm32.a -L/home/akien/Projects/godot/emscripten/emsdk/upstream/emscripten/cache/sysroot/lib/wasm32-emscripten/lto /home/akien/Projects/godot/emscripten/emsdk/upstream/emscripten/cache/sysroot/lib/wasm32-emscripten/lto/crtbegin.o -lGL-mt-webgl2-ofb-getprocaddr -lal -lhtml5 -lbulkmemory -lstubs-debug -lc-mt-debug -ldlmalloc-mt -lcompiler_rt-mt -lc++-mt-noexcept -lc++abi-debug-mt-noexcept -lsockets-mt --keep-section=target_features -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr /tmp/tmpzjbym0dhlibemscripten_js_symbols.so --import-memory --shared-memory --strip-debug --export=_emscripten_thread_init --export=_emscripten_thread_exit --export=_emscripten_thread_crashed --export=_emscripten_tls_init --export=pthread_self --export=emscripten_stack_get_end --export=emscripten_stack_get_free --export=emscripten_stack_get_base --export=emscripten_stack_get_current --export=emscripten_stack_init --export=stackAlloc --export=stackSave --export=stackRestore --export=_emscripten_thread_free_data --export=emscripten_main_runtime_thread_id --export=emscripten_main_thread_process_queued_calls --export=_emscripten_run_on_main_thread_js --export=emscripten_stack_set_limits --export=__get_temp_ret --export=__set_temp_ret --export=__funcs_on_exit --export=__wasm_call_ctors --export=_emscripten_thread_init --export=_emscripten_thread_exit --export-if-defined=__start_em_asm --export-if-defined=__stop_em_asm --export-if-defined=__start_em_lib_deps --export-if-defined=__stop_em_lib_deps --export-if-defined=__start_em_js --export-if-defined=__stop_em_js --export-if-defined=main --export-if-defined=__main_argc_argv --export-if-defined=fflush --export-table -z stack-size=5242880 --initial-memory=33554432 --max-memory=2147483648 --no-entry --stack-first --table-base=1' failed (returned 1)

CC @adamscott

[haasal]

Ok this fixed that error but here is the next one:

scons platform=web target=template_release lto=full optimize=size module_text_server_adv_enabled=no module_text_server_fb_enabled=yes disable_3d=yes disable_advanced_gui=yes

wasm-ld: error: Invalid record (Producer: 'LLVM18.0.0git' Reader: 'LLVM 17.0.0git')

This doesn't seem to have anything to do with my local llvm version which is 16.0.6

Edit: Maybe this is because I tried to compile the 4.2.1 version (I'll try this again)

[AThousandShips]

You need to recompile stuff after changing version, wipe the output files

[Dinhero21]

Ok this fixed that error but here is the next one:

scons platform=web target=template_release lto=full optimize=size module_text_server_adv_enabled=no module_text_server_fb_enabled=yes disable_3d=yes disable_advanced_gui=yes

wasm-ld: error: Invalid record (Producer: 'LLVM18.0.0git' Reader: 'LLVM 17.0.0git')

This doesn't seem to have anything to do with my local llvm version which is 16.0.6

Edit: Maybe this is because I tried to compile the 4.2.1 version (I'll try this again)

I re-cloned the repository and it fixed that issue (probably only a single directory was needed but I couldn't find it)

[adamscott]

I re-cloned the repository and it fixed that issue (probably only a single directory was needed but I couldn't find it)

The lto cache is located in your emsdk or emscripten install. Mine is at <emsdk location>/upstream/emscripten/cache.

Edit: wait, it doesn't work.

[adamscott]

Ok this fixed that error but here is the next one:

scons platform=web target=template_release lto=full optimize=size module_text_server_adv_enabled=no module_text_server_fb_enabled=yes disable_3d=yes disable_advanced_gui=yes

wasm-ld: error: Invalid record (Producer: 'LLVM18.0.0git' Reader: 'LLVM 17.0.0git')

This doesn't seem to have anything to do with my local llvm version which is 16.0.6

Ok, I figured out on how to fix this error obtained when downgrading emsdk.

1. Install and activate a new version of emsdk

emsdk install 3.1.45
emsdk activate 3.1.45

2. Read the instructions on how to activate, or open a new tab if you activate it automatically, to force the refresh of the paths.
3. Run your scons build command with -c in order to clear the built files.
4. Finally, run your scons build command with --cache-disable in order to make sure that you don't use any cached files built with a newer version.

[adamscott]

So, I bisected emscripten versions and it's possible to compile Godot up to emscripten 3.1.46. I'm investigating why it changed in 3.1.47.

[adamscott]

Relates to godotengine/godot-docs#8589.