Skip to content

wasilibc: add flag for thread support; add extra test case#241739

Open
sternenseemann wants to merge 3 commits intoNixOS:masterfrom
sternenseemann:wasilibc-20
Open

wasilibc: add flag for thread support; add extra test case#241739
sternenseemann wants to merge 3 commits intoNixOS:masterfrom
sternenseemann:wasilibc-20

Conversation

@sternenseemann
Copy link
Member

@sternenseemann sternenseemann commented Jul 5, 2023

Description of changes

GHC's wasm backend needs LLVM 16 (due to new flags), bumping the default version for WASM seems like the simplest solution for this. While we're at it, update wasilibc which uses LLVM 16 in release 20 as well.

Things done
  • Built on platform(s)
    • x86_64-linux
    • aarch64-linux
    • x86_64-darwin
    • aarch64-darwin
  • For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
  • Tested, as applicable:
  • Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
  • Tested basic functionality of all binary files (usually in ./result/bin/)
  • 23.11 Release Notes (or backporting 23.05 Release notes)
    • (Package updates) Added a release notes entry if the change is major or breaking
    • (Module updates) Added a release notes entry if the change is significant
    • (Module addition) Added a release notes entry if adding a new NixOS module
  • Fits CONTRIBUTING.md.

@ofborg ofborg bot added 11.by: package-maintainer This PR was created by a maintainer of all the package it changes. 10.rebuild-darwin: 11-100 This PR causes between 11 and 100 packages to rebuild on Darwin. 10.rebuild-linux: 11-100 This PR causes between 11 and 100 packages to rebuild on Linux. labels Jul 5, 2023
@sternenseemann
Copy link
Member Author

firefox-esr now fails with 010bce9: error: unexpected opcode: 0xfc 0xa

/build/firefox-102.12.0/media/ffvpx/config_unix64.h:219:9: warning: 'HAVE_LINUX_PERF_EVENT_H' macro redefined [-Wmacro-redefined]
#define HAVE_LINUX_PERF_EVENT_H 0
        ^
/build/firefox-102.12.0/mozobj/mozilla-config.h:56:9: note: previous definition is here
#define HAVE_LINUX_PERF_EVENT_H 1
        ^
In file included from <built-in>:2:
In file included from /build/firefox-102.12.0/media/ffvpx/libavutil_visibility.h:16:
In file included from /build/firefox-102.12.0/media/ffvpx/libavutil/log.h:25:
In file included from /build/firefox-102.12.0/media/ffvpx/libavutil/avutil.h:296:
In file included from /build/firefox-102.12.0/media/ffvpx/libavutil/common.h:114:
In file included from /build/firefox-102.12.0/media/ffvpx/config.h:52:
In file included from /build/firefox-102.12.0/media/ffvpx/config_common.h:3:
/build/firefox-102.12.0/media/ffvpx/defaults_disabled.h:232:9: warning: 'CONFIG_AV1_PARSER' macro redefined [-Wmacro-redefined]
#define CONFIG_AV1_PARSER 0
        ^
/build/firefox-102.12.0/media/ffvpx/config_unix64.h:681:9: note: previous definition is here
#define CONFIG_AV1_PARSER 1
        ^
In file included from <built-in>:2:
In file included from /build/firefox-102.12.0/media/ffvpx/libavutil_visibility.h:16:
In file included from /build/firefox-102.12.0/media/ffvpx/libavutil/log.h:25:
In file included from /build/firefox-102.12.0/media/ffvpx/libavutil/avutil.h:296:
In file included from /build/firefox-102.12.0/media/ffvpx/libavutil/common.h:114:
In file included from /build/firefox-102.12.0/media/ffvpx/config.h:52:
In file included from /build/firefox-102.12.0/media/ffvpx/config_common.h:3:
/build/firefox-102.12.0/media/ffvpx/defaults_disabled.h:233:9: warning: 'CONFIG_AV1_DECODER' macro redefined [-Wmacro-redefined]
#define CONFIG_AV1_DECODER 0
        ^
/build/firefox-102.12.0/media/ffvpx/config_unix64.h:676:9: note: previous definition is here
#define CONFIG_AV1_DECODER 1
        ^
In file included from <built-in>:2:
In file included from /build/firefox-102.12.0/media/ffvpx/libavutil_visibility.h:16:
In file included from /build/firefox-102.12.0/media/ffvpx/libavutil/log.h:25:
In file included from /build/firefox-102.12.0/media/ffvpx/libavutil/avutil.h:296:
In file included from /build/firefox-102.12.0/media/ffvpx/libavutil/common.h:114:
In file included from /build/firefox-102.12.0/media/ffvpx/config.h:45:
/build/firefox-102.12.0/media/ffvpx/config_unix64.h:219:9: warning: 'HAVE_LINUX_PERF_EVENT_H' macro redefined [-Wmacro-redefined]
#define HAVE_LINUX_PERF_EVENT_H 0
        ^
/build/firefox-102.12.0/mozobj/mozilla-config.h:56:9: note: previous definition is here
#define HAVE_LINUX_PERF_EVENT_H 1
        ^
In file included from <built-in>:2:
In file included from /build/firefox-102.12.0/media/ffvpx/libavutil_visibility.h:16:
In file included from /build/firefox-102.12.0/media/ffvpx/libavutil/log.h:25:
In file included from /build/firefox-102.12.0/media/ffvpx/libavutil/avutil.h:296:
In file included from /build/firefox-102.12.0/media/ffvpx/libavutil/common.h:114:
In file included from /build/firefox-102.12.0/media/ffvpx/config.h:52:
In file included from /build/firefox-102.12.0/media/ffvpx/config_common.h:3:
/build/firefox-102.12.0/media/ffvpx/defaults_disabled.h:232:9: warning: 'CONFIG_AV1_PARSER' macro redefined [-Wmacro-redefined]
#define CONFIG_AV1_PARSER 0
        ^
/build/firefox-102.12.0/media/ffvpx/config_unix64.h:681:9: note: previous definition is here
#define CONFIG_AV1_PARSER 1
        ^
In file included from <built-in>:2:
In file included from /build/firefox-102.12.0/media/ffvpx/libavutil_visibility.h:16:
In file included from /build/firefox-102.12.0/media/ffvpx/libavutil/log.h:25:
In file included from /build/firefox-102.12.0/media/ffvpx/libavutil/avutil.h:296:
In file included from /build/firefox-102.12.0/media/ffvpx/libavutil/common.h:114:
In file included from /build/firefox-102.12.0/media/ffvpx/config.h:52:
In file included from /build/firefox-102.12.0/media/ffvpx/config_common.h:3:
/build/firefox-102.12.0/media/ffvpx/defaults_disabled.h:233:9: warning: 'CONFIG_AV1_DECODER' macro redefined [-Wmacro-redefined]
#define CONFIG_AV1_DECODER 0
        ^
/build/firefox-102.12.0/media/ffvpx/config_unix64.h:676:9: note: previous definition is here
#define CONFIG_AV1_DECODER 1
        ^
3 warnings generated.
3 warnings generated.
3 warnings generated.
In file included from <built-in>:2:
In file included from /build/firefox-102.12.0/media/ffvpx/libavutil_visibility.h:16:
In file included from /build/firefox-102.12.0/media/ffvpx/libavutil/log.h:25:
In file included from /build/firefox-102.12.0/media/ffvpx/libavutil/avutil.h:296:
In file included from /build/firefox-102.12.0/media/ffvpx/libavutil/common.h:114:
In file included from /build/firefox-102.12.0/media/ffvpx/config.h:45:
/build/firefox-102.12.0/media/ffvpx/config_unix64.h:219:9: warning: 'HAVE_LINUX_PERF_EVENT_H' macro redefined [-Wmacro-redefined]
#define HAVE_LINUX_PERF_EVENT_H 0
        ^
/build/firefox-102.12.0/mozobj/mozilla-config.h:56:9: note: previous definition is here
#define HAVE_LINUX_PERF_EVENT_H 1
        ^
In file included from <built-in>:2In file included from <built-in>:2:
In file included from /build/firefox-102.12.0/media/ffvpx/libavutil_visibility.h:16:
In file included from /build/firefox-102.12.0/media/ffvpx/libavutil/log.h:25:
In file included from /build/firefox-102.12.0/media/ffvpx/libavutil/avutil.h:296:
In file included from /build/firefox-102.12.0/media/ffvpx/libavutil/common.h:114:
In file included from /build/firefox-102.12.0/media/ffvpx/config.h:45:
/build/firefox-102.12.0/media/ffvpx/config_unix64.h:219:9: warning: 'HAVE_LINUX_PERF_EVENT_H' macro redefined [-Wmacro-redefined]
#define HAVE_LINUX_PERF_EVENT_H 0
        ^
/build/firefox-102.12.0/mozobj/mozilla-config.h:56:9: note: previous definition is here
#define HAVE_LINUX_PERF_EVENT_H 1
        ^
:
In file included from /build/firefox-102.12.0/media/ffvpx/libavutil_visibility.h:16:
In file included from /build/firefox-102.12.0/media/ffvpx/libavutil/log.h:25:
In file included from /build/firefox-102.12.0/media/ffvpx/libavutil/avutil.h:296:
In file included from /build/firefox-102.12.0/media/ffvpx/libavutil/common.h:114:
In file included from /build/firefox-102.12.0/media/ffvpx/config.hIn file included from <built-in>:2::
In file included from /build/firefox-102.12.0/media/ffvpx/libavutil_visibility.h:16:
In file included from /build/firefox-102.12.0/media/ffvpx/libavutil/log.h:25:
In file included from /build/firefox-102.12.0/media/ffvpx/libavutil/avutil.h:296:
In file included from /build/firefox-102.12.0/media/ffvpx/libavutil/common.h:114:
In file included from /build/firefox-102.12.0/media/ffvpx/config.h:52:
In file included from /build/firefox-102.12.0/media/ffvpx/config_common.h:3:
/build/firefox-102.12.0/media/ffvpx/defaults_disabled.h:232:9: warning: 'CONFIG_AV1_PARSER' macro redefined [-Wmacro-redefined]
#define CONFIG_AV1_PARSER 0
        ^
/build/firefox-102.12.0/media/ffvpx/config_unix64.h:681:9: note: previous definition is here
#define CONFIG_AV1_PARSER 1
        ^
In file included from 52<built-in>:2:
In file included from /build/firefox-102.12.0/media/ffvpx/libavutil_visibility.h:16:
In file included from /build/firefox-102.12.0/media/ffvpx/libavutil/log.h:25:
In file included from /build/firefox-102.12.0/media/ffvpx/libavutil/avutil.h:296:
In file included from /build/firefox-102.12.0/media/ffvpx/libavutil/common.h:114:
In file included from /build/firefox-102.12.0/media/ffvpx/config.h:52:
In file included from /build/firefox-102.12.0/media/ffvpx/config_common.h:3:
/build/firefox-102.12.0/media/ffvpx/defaults_disabled.h:233:9: warning: 'CONFIG_AV1_DECODER' macro redefined [-Wmacro-redefined]
#define CONFIG_AV1_DECODER 0
        ^
/build/firefox-102.12.0/media/ffvpx/config_unix64.h:676:9: note: previous definition is here
#define CONFIG_AV1_DECODER 1
        ^
:
In file included from /build/firefox-102.12.0/media/ffvpx/config_common.h:3:
/build/firefox-102.12.0/media/ffvpx/defaults_disabled.h:232:9: warning: 'CONFIG_AV1_PARSER' macro redefined [-Wmacro-redefined]
#define CONFIG_AV1_PARSER 0
        ^
/build/firefox-102.12.0/media/ffvpx/config_unix64.h:681:9: note: previous definition is here
#define CONFIG_AV1_PARSER 1
        ^
In file included from <built-in>:2:
In file included from /build/firefox-102.12.0/media/ffvpx/libavutil_visibility.h:16:
In file included from /build/firefox-102.12.0/media/ffvpx/libavutil/log.h:25:
In file included from /build/firefox-102.12.0/media/ffvpx/libavutil/avutil.h:296:
In file included from /build/firefox-102.12.0/media/ffvpx/libavutil/common.h:114:
In file included from /build/firefox-102.12.0/media/ffvpx/config.h:52:
In file included from /build/firefox-102.12.0/media/ffvpx/config_common.h:3:
/build/firefox-102.12.0/media/ffvpx/defaults_disabled.h:233:9: warning: 'CONFIG_AV1_DECODER' macro redefined [-Wmacro-redefined]
#define CONFIG_AV1_DECODER 0
        ^
/build/firefox-102.12.0/media/ffvpx/config_unix64.h:676:9: note: previous definition is here
#define CONFIG_AV1_DECODER 1
        ^
3 warnings generated.
3 warnings generated.
3 warnings generated.
3 warnings generated.
make[3]: Leaving directory '/build/firefox-102.12.0/mozobj/media/libdav1d'
make[3]: Leaving directory '/build/firefox-102.12.0/mozobj/media/ffvpx/libavcodec/x86'
3 warnings generated.
3 warnings generated.
3 warnings generated.
3 warnings generated.
make[3]: Leaving directory '/build/firefox-102.12.0/mozobj/toolkit/components/places'
rm -f rlbox.wasm
/nix/store/7k0niyhlmadcs30pcb462diwkdb9kadc-wasm32-unknown-wasi-clang-wrapper-16.0.6/bin/wasm32-unknown-wasi-c++ --sysroot=/nix/store/33cpnqkbwy26jdd88gjx67d828d4j9hi-wasi-sysroot -o rlbox.wasm -Wl,--export-all -Wl,--stack-first -Wl,-z,stack-size=262144 -Wl,--no-entry -Wl,--growable-table ogg_alloc.wasm ogg_bitwise.wasm ogg_framing.wasm xmlparse.wasm xmlrole.wasm xmltok.wasm wasm2c_sandbox_wrapper.wasm mozHunspellRLBoxSandbox.wasm affentry.wasm affixmgr.wasm csutil.wasm hashmgr.wasm hunspell.wasm phonet.wasm replist.wasm suggestmgr.wasm GraphiteExtra.wasm CmapCache.wasm Code.wasm Collider.wasm Decompressor.wasm Face.wasm FeatureMap.wasm FileFace.wasm Font.wasm GlyphCache.wasm GlyphFace.wasm Intervals.wasm Justifier.wasm NameTable.wasm Pass.wasm Position.wasm Segment.wasm Silf.wasm Slot.wasm Sparse.wasm TtfUtil.wasm UtfCodec.wasm call_machine.wasm gr_char_info.wasm gr_face.wasm gr_features.wasm gr_font.wasm gr_logging.wasm gr_segment.wasm gr_slot.wasm json.wasm RLBoxWOFF2Sandbox.wasm table_tags.wasm variable_length.wasm woff2_common.wasm woff2_dec.wasm woff2_out.wasm -lwasi-emulated-process-clocks
../../dist/host/bin/wasm2c -o rlbox.wasm.c rlbox.wasm
3 warnings generated.
010bce9: error: unexpected opcode: 0xfc 0xa
make[3]: *** [/build/firefox-102.12.0/config/rules.mk:510: rlbox.wasm.c] Error 1
make[3]: Leaving directory '/build/firefox-102.12.0/mozobj/security/rlbox'
make[2]: *** [/build/firefox-102.12.0/config/recurse.mk:72: security/rlbox/target-objects] Error 2
make[2]: *** Waiting for unfinished jobs....
3 warnings generated.
3 warnings generated.
make[3]: Leaving directory '/build/firefox-102.12.0/mozobj/tools/profiler'
make[3]: Leaving directory '/build/firefox-102.12.0/mozobj/js/src/gc'
make[3]: Leaving directory '/build/firefox-102.12.0/mozobj/netwerk/dns'
3 warnings generated.
make[3]: Leaving directory '/build/firefox-102.12.0/mozobj/widget'
make[3]: Leaving directory '/build/firefox-102.12.0/mozobj/toolkit/components/url-classifier'
3 warnings generated.
make[3]: Leaving directory '/build/firefox-102.12.0/mozobj/toolkit/components/reputationservice'
3 warnings generated.
3 warnings generated.
make[3]: Leaving directory '/build/firefox-102.12.0/mozobj/security/manager/ssl'
make[3]: Leaving directory '/build/firefox-102.12.0/mozobj/toolkit/components/telemetry'
3 warnings generated.
make[3]: Leaving directory '/build/firefox-102.12.0/mozobj/media/ffvpx/libavcodec'
make[3]: Leaving directory '/build/firefox-102.12.0/mozobj/media/libdav1d/asm'
make[3]: Leaving directory '/build/firefox-102.12.0/mozobj/netwerk/base'
make[3]: Leaving directory '/build/firefox-102.12.0/mozobj/layout/style'
warning: `webrender` (lib) generated 1 warning (run `cargo fix --lib -p webrender` to apply 1 suggestion)
warning: `style` (lib) generated 1 warning
   Compiling gkrust v0.1.0 (/build/firefox-102.12.0/toolkit/library/rust)
    Finished release [optimized] target(s) in 19m 54s
warning: the following packages contain code that will be rejected by a future version of Rust: bindgen v0.56.0 (/build/firefox-102.12.0/third_party/rust/bindgen), cose v0.1.4, nom v5.1.2
note: to see what the problems were, use the option `--future-incompat-report`, or run `cargo report future-incompatibilities --id 1`
make[3]: Leaving directory '/build/firefox-102.12.0/mozobj/toolkit/library/rust'
make[2]: Leaving directory '/build/firefox-102.12.0/mozobj'
make[1]: *** [/build/firefox-102.12.0/config/recurse.mk:34: compile] Error 2
make[1]: Leaving directory '/build/firefox-102.12.0/mozobj'
make: *** [/build/firefox-102.12.0/config/rules.mk:361: default] Error 2
builder for '/nix/store/i2m2akx5zf77cp6zmk2b9y6drks8gxcf-firefox-esr-102-unwrapped-102.12.0esr.drv' failed with exit code 2

@mweinelt
Copy link
Member

Firefox 102 ESR is gone!

@sternenseemann sternenseemann marked this pull request as draft November 8, 2023 14:54
@mweinelt
Copy link
Member

mweinelt commented Nov 8, 2023

Wasilibc 20 is currently on staging-next

This is an experimental new feature in wasilibc which also allows
enabling threads in libcxx* as a consequence. This is wired up using
passthru.hasThreads. It is disabled by default (as it is experimental
still), but overlaying wasilibc to turn on the flag should allow using
it.

See also:
- WebAssembly/wasi-sdk#274
- WebAssembly/wasi-sdk#301
- WebAssembly/wasi-sdk#314

Flags for libc++abi are copied from wasi-sdk.

The BULK_MEMORY_SOURCES is dropped, as it no longer needs to be set to
be empty to compile firefox. With it set, the build of wasilibc would
fail if enableThreads is true.

Test using

  (import ./. {
    overlays = [
      (self: super: {
        wasilibc = super.wasilibc.override { enableThreads = true; };
      })
    ];
  }).wasilibc.tests.simple-c-cxx-binaries
This is a quicker way than building firefox to see if linking works.
@ofborg ofborg bot added 10.rebuild-darwin: 1-10 This PR causes between 1 and 10 packages to rebuild on Darwin. 10.rebuild-darwin: 1 This PR causes 1 package to rebuild on Darwin. and removed 10.rebuild-darwin: 11-100 This PR causes between 11 and 100 packages to rebuild on Darwin. labels Nov 19, 2023
@sternenseemann sternenseemann changed the title wasilibc: 19 -> 20 wasilibc: add flag for thread support; add extra test case Nov 20, 2023
@sternenseemann sternenseemann marked this pull request as ready for review November 20, 2023 12:55
@sternenseemann
Copy link
Member Author

Remaining changes are an additional test case and optional, off by default support for the new pthread API in wasilibc.

@wegank wegank added the 2.status: merge conflict This PR has merge conflicts with the target branch label Apr 5, 2024
@wegank wegank added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Jul 4, 2024
@ghost ghost closed this Dec 19, 2024
@mweinelt
Copy link
Member

This looks like it is still applicable and just needs a rebase.

@ghost ghost reopened this Dec 19, 2024
@stale stale bot removed the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Dec 19, 2024
@github-actions github-actions bot added the 6.topic: llvm/clang Issues related to llvmPackages, clangStdenv and related label Dec 19, 2024
@nix-owners nix-owners bot requested a review from RossComputerGuy December 19, 2024 17:28
"-DLIBCXX_ADDITIONAL_LIBRARIES=unwind"
] ++ lib.optionals stdenv.hostPlatform.isWasm [
"-DLIBCXX_ENABLE_THREADS=OFF"
"-DLIBCXX_ENABLE_THREADS=${if stdenv.cc.libc.hasThreads then "ON" else "OFF"}"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please use lib.cmakeBool

@wegank wegank added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Jan 2, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

2.status: merge conflict This PR has merge conflicts with the target branch 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md 6.topic: llvm/clang Issues related to llvmPackages, clangStdenv and related 10.rebuild-darwin: 1-10 This PR causes between 1 and 10 packages to rebuild on Darwin. 10.rebuild-darwin: 1 This PR causes 1 package to rebuild on Darwin. 10.rebuild-linux: 11-100 This PR causes between 11 and 100 packages to rebuild on Linux. 11.by: package-maintainer This PR was created by a maintainer of all the package it changes.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants