SIGBUS and SIGSEGV a when compiling a modified version of php_cgi with cranelift

[Michael-F-Bryan]

Describe the bug

I am running into segfaults (SIGSEGV and SIGBUS) during the compilation of a particular WebAssembly file.

This happens on both debug and release versions of the wasix branch (commit d34ce88) on a M1 Mac, although the release binary triggers a SIGSEGV while the debug binary triggers a SIGBUS.

$  ~/.cargo/bin/wasmer --version --verbose
wasmer 3.1.0 (d34ce 2023-01-10)
binary: wasmer-cli
commit-hash: d34ce882e5e90ef500b51a2caacd5939e3397c82
commit-date: 2023-01-10
host: aarch64-apple-darwin
compiler: singlepass,cranelift

$ rustc --version --verbose
rustc 1.68.0-nightly (659e169d3 2023-01-04)
binary: rustc
commit-hash: 659e169d37990b9c730a59a96081f2ef7afbe8f1
commit-date: 2023-01-04
host: aarch64-apple-darwin
release: 1.68.0-nightly
LLVM version: 15.0.6

$ uname -a
Darwin mbp 22.2.0 Darwin Kernel Version 22.2.0: Fri Nov 11 02:03:51 PST 2022; root:xnu-8792.61.2~4/RELEASE_ARM64_T6000 arm64

Steps to reproduce

First, install the wasmer CLI from the wasix branch.

$ git checkout d34ce882e5e90ef500b51a2caacd5939e3397c82
$ cargo install --features cranelift,webc_runner,wasi,singlepass --path lib/cli --debug

Then, extract the modified.wasm file from input.zip.

Now you can try to run it with wasmer run.

$ ~/.cargo/bin/wasmer run --cranelift modified.wasm
[1]    48340 bus error  ~/.cargo/bin/wasmer run --cranelift modified.wasm

Running LLDB on the debug binary gave the following backtrace:

(lldb) run rename-import-namespace/modified.wasm
Process 45804 launched: '/Users/work/.cargo/bin/wasmer' (arm64)
Process 45804 stopped

* thread #1, name = 'main', queue = 'com.apple.main-thread', stop reason = signal SIGBUS
    frame #0: 0x0000000163160b04
->  0x163160b04: udf    #0x0
    0x163160b08: udf    #0x0
    0x163160b0c: udf    #0x0
    0x163160b10: udf    #0x0
Target 0: (wasmer) stopped.

(lldb) bt
* thread #1, name = 'main', queue = 'com.apple.main-thread', stop reason = signal SIGBUS
  * frame #0: 0x0000000163160b04
    frame #1: 0x0000000101f59f38 wasmer`wasmer_types::compilation::relocation::_::_$LT$impl$u20$rkyv..Deserialize$LT$wasmer_types..compilation..relocation..Relocation$C$__D$GT$$u20$for$u20$$LT$wasmer_types..compilation..relocation..Relocation$u20$as$u20$rkyv..Archive$GT$..Archived$GT$::deserialize::h6f2a1cd23db4d61c(self=0x000000010b57fa78, deserializer=0x000000016fdf68f8) at relocation.rs:86:25
    frame #2: 0x0000000101f598f8 wasmer`rkyv::impls::core::_$LT$impl$u20$rkyv..DeserializeUnsized$LT$$u5b$U$u5d$$C$D$GT$$u20$for$u20$$u5b$T$u5d$$GT$::deserialize_unsized::h871818887bcc9ea5(self=&[wasmer_types::compilation::relocation::ArchivedRelocation] @ 0x000000016fdf49d0, deserializer=0x000000016fdf68f8, alloc={closure_env#0}<wasmer_types::compilation::relocation::Relocation, rkyv::de::deserializers::alloc::SharedDeserializeMap> @ 0x000000016fdf48c7) at mod.rs:269:41
    frame #3: 0x0000000101f578d0 wasmer`rkyv::impls::alloc::vec::_$LT$impl$u20$rkyv..Deserialize$LT$alloc..vec..Vec$LT$T$GT$$C$D$GT$$u20$for$u20$rkyv..vec..ArchivedVec$LT$$LT$T$u20$as$u20$rkyv..Archive$GT$..Archived$GT$$GT$::deserialize::h7d9c6fe00795a30e(self=0x000000010bfe38a8, deserializer=0x000000016fdf68f8) at vec.rs:64:32
    frame #4: 0x0000000101f565b8 wasmer`rkyv::impls::core::_$LT$impl$u20$rkyv..DeserializeUnsized$LT$$u5b$U$u5d$$C$D$GT$$u20$for$u20$$u5b$T$u5d$$GT$::deserialize_unsized::h87b38d6fae1c802b(self=&[rkyv::vec::ArchivedVec<wasmer_types::compilation::relocation::ArchivedRelocation>] @ 0x000000016fdf4d90, deserializer=0x000000016fdf68f8, alloc={closure_env#0}<alloc::vec::Vec<wasmer_types::compilation::relocation::Relocation, alloc::alloc::Global>, rkyv::de::deserializers::alloc::SharedDeserializeMap> @ 0x000000016fdf4c87) at mod.rs:269:41
    frame #5: 0x0000000101f56da8 wasmer`rkyv::impls::alloc::vec::_$LT$impl$u20$rkyv..Deserialize$LT$alloc..vec..Vec$LT$T$GT$$C$D$GT$$u20$for$u20$rkyv..vec..ArchivedVec$LT$$LT$T$u20$as$u20$rkyv..Archive$GT$..Archived$GT$$GT$::deserialize::h1740544bb7abf877(self=0x000000010c730c90, deserializer=0x000000016fdf68f8) at vec.rs:64:32
    frame #6: 0x0000000101e94234 wasmer`wasmer_types::entity::primary_map::_::_$LT$impl$u20$rkyv..Deserialize$LT$wasmer_types..entity..primary_map..PrimaryMap$LT$K$C$V$GT$$C$__D$GT$$u20$for$u20$$LT$wasmer_types..entity..primary_map..PrimaryMap$LT$K$C$V$GT$$u20$as$u20$rkyv..Archive$GT$..Archived$GT$::deserialize::hbd1486be8bcf50a7(self=0x000000010c730c90, deserializer=0x000000016fdf68f8) at primary_map.rs:36:25
    frame #7: 0x0000000101eaad4c wasmer`wasmer_types::serialize::_::_$LT$impl$u20$rkyv..Deserialize$LT$wasmer_types..serialize..SerializableCompilation$C$__D$GT$$u20$for$u20$$LT$wasmer_types..serialize..SerializableCompilation$u20$as$u20$rkyv..Archive$GT$..Archived$GT$::deserialize::h5a1fe0a54de13018(self=0x000000010c730c88, deserializer=0x000000016fdf68f8) at serialize.rs:20:28
    frame #8: 0x0000000101eac01c wasmer`wasmer_types::serialize::_::_$LT$impl$u20$rkyv..Deserialize$LT$wasmer_types..serialize..SerializableModule$C$__D$GT$$u20$for$u20$$LT$wasmer_types..serialize..SerializableModule$u20$as$u20$rkyv..Archive$GT$..Archived$GT$::deserialize::hb4072f80f08997f5(self=0x000000010c730c80, deserializer=0x000000016fdf68f8) at serialize.rs:54:19
    frame #9: 0x0000000101eda4f0 wasmer`wasmer_types::serialize::SerializableModule::deserialize_from_archive::hb46a4276ad1795ff(archived=0x000000010c730c80) at serialize.rs:129:9
    frame #10: 0x0000000101eda270 wasmer`wasmer_types::serialize::SerializableModule::deserialize::h9ba5a038bcc7a58b(metadata_slice=(data_ptr = "\xfd{\xbf\xa9\xfd\U00000003", length = 27463048)) at serialize.rs:100:9
    frame #11: 0x0000000101c6884c wasmer`wasmer_compiler::engine::artifact::Artifact::deserialize::h94bd1f3f23a1c707(engine=0x000000016fdfc030, bytes=(data_ptr = "wasmer-universalWASMER", length = 27463080)) at artifact.rs:127:28
    frame #12: 0x0000000101c8d0e8 wasmer`wasmer_compiler::engine::inner::Engine::deserialize::hc46f4ee5af3ee3a2(self=0x000000016fdfc030, bytes=(data_ptr = "wasmer-universalWASMER", length = 27463080)) at inner.rs:184:21
    frame #13: 0x0000000101c8d35c wasmer`wasmer_compiler::engine::inner::Engine::deserialize_from_file::h6df3ebfdd80d0510(self=0x000000016fdfc030, file_ref=&std::path::Path @ 0x000000016fdfb148) at inner.rs:199:9
    frame #14: 0x00000001000c2ce8 wasmer`wasmer::sys::module::Module::deserialize_from_file::h7f358b3d4da7bb59(engine=0x000000016fdfc028, path=PathBuf @ 0x000000016fdfb510) at module.rs:300:24
    frame #15: 0x00000001000ea4b8 wasmer`_$LT$wasmer_cache..filesystem..FileSystemCache$u20$as$u20$wasmer_cache..cache..Cache$GT$::load::h26fe395c4f7654d8(self=0x000000016fdfb630, engine=0x000000016fdfc028, key=(__0 = "\x8cZMfj <\x99\xbcM\xf7?\U0000001dsڤ\xef\xfe\xbe\U00000011Z\x8a\xff\xfb\xfa\xa2\x968\xd8T@\xce")) at filesystem.rs:105:19
    frame #16: 0x00000001000a5658 wasmer`wasmer_cli::commands::run::RunWithPathBuf::get_module_from_cache::h91d07df44eec21f8(self=0x000000016fdfddc0, store=0x000000016fdfc028, contents=(data_ptr = "", length = 11051999), compiler_type=0x000000016fdfc077) at run.rs:428:24
    frame #17: 0x00000001000a52a8 wasmer`wasmer_cli::commands::run::RunWithPathBuf::get_store_module::h0ecf3b64161b20ad(self=0x000000016fdfddc0) at run.rs:390:13
    frame #18: 0x00000001000a36ac wasmer`wasmer_cli::commands::run::RunWithPathBuf::inner_execute::hc40448fa6d2140a3(self=0x000000016fdfddc0) at run.rs:206:35
    frame #19: 0x00000001000a2c38 wasmer`wasmer_cli::commands::run::RunWithPathBuf::execute::h6f9ea7deb73ffa7d(self=0x000000016fdfe970) at run.rs:150:9
    frame #20: 0x00000001000a5ed8 wasmer`wasmer_cli::commands::run::Run::execute::hb837ece6caafe0a1(self=0x000000016fdfee60) at run.rs:570:9
    frame #21: 0x00000001000fd15c wasmer`wasmer_cli::cli::WasmerCLIOptions::execute::h30ba4c54da3f80ec(self=0x000000016fdfee58) at cli.rs:164:35
    frame #22: 0x00000001000fdb9c wasmer`wasmer_cli::cli::wasmer_main_inner::h5bff492c7fade00b at cli.rs:254:5
    frame #23: 0x00000001000fd30c wasmer`wasmer_cli::cli::wasmer_main::h00a46ef7f9f6a084 at cli.rs:196:25
    frame #24: 0x0000000100000c68 wasmer`wasmer::main::h6761ffe90c4dedfb at wasmer.rs:9:5
    frame #25: 0x0000000100000ddc wasmer`core::ops::function::FnOnce::call_once::h3b7122b4878c5869((null)=(wasmer`wasmer::main::h6761ffe90c4dedfb at wasmer.rs:8), (null)=<unavailable>) at function.rs:248:5
    frame #26: 0x0000000100000cb4 wasmer`std::sys_common::backtrace::__rust_begin_short_backtrace::h7ed23e9bf0b924ad(f=(wasmer`wasmer::main::h6761ffe90c4dedfb at wasmer.rs:8)) at backtrace.rs:122:18
    frame #27: 0x0000000100000d64 wasmer`std::rt::lang_start::_$u7b$$u7b$closure$u7d$$u7d$::he556e00b175bd0de at rt.rs:145:18
    frame #28: 0x0000000102e090e4 wasmer`std::rt::lang_start_internal::hef2161f9571a51d7 [inlined] core::ops::function::impls::_$LT$impl$u20$core..ops..function..FnOnce$LT$A$GT$$u20$for$u20$$RF$F$GT$::call_once::h7cd8ae72620b0d1f at function.rs:280:13 [opt]
    frame #29: 0x0000000102e090d8 wasmer`std::rt::lang_start_internal::hef2161f9571a51d7 [inlined] std::panicking::try::do_call::h0bbb0f423dd9d86c at panicking.rs:492:40 [opt]
    frame #30: 0x0000000102e090d8 wasmer`std::rt::lang_start_internal::hef2161f9571a51d7 [inlined] std::panicking::try::hdaabe7e5908702af at panicking.rs:456:19 [opt]
    frame #31: 0x0000000102e090d8 wasmer`std::rt::lang_start_internal::hef2161f9571a51d7 [inlined] std::panic::catch_unwind::h7ee653eae81d0a43 at panic.rs:137:14 [opt]
    frame #32: 0x0000000102e090d8 wasmer`std::rt::lang_start_internal::hef2161f9571a51d7 [inlined] std::rt::lang_start_internal::_$u7b$$u7b$closure$u7d$$u7d$::he727754da11a45c1 at rt.rs:128:48 [opt]
    frame #33: 0x0000000102e090d8 wasmer`std::rt::lang_start_internal::hef2161f9571a51d7 [inlined] std::panicking::try::do_call::h0171064c04d908b7 at panicking.rs:492:40 [opt]
    frame #34: 0x0000000102e090d8 wasmer`std::rt::lang_start_internal::hef2161f9571a51d7 [inlined] std::panicking::try::h9c521838fe914345 at panicking.rs:456:19 [opt]
    frame #35: 0x0000000102e090d8 wasmer`std::rt::lang_start_internal::hef2161f9571a51d7 [inlined] std::panic::catch_unwind::hac847f960377a9be at panic.rs:137:14 [opt]
    frame #36: 0x0000000102e090d8 wasmer`std::rt::lang_start_internal::hef2161f9571a51d7 at rt.rs:128:20 [opt]
    frame #37: 0x0000000100000d2c wasmer`std::rt::lang_start::hab5ee70cafa638f8(main=(wasmer`wasmer::main::h6761ffe90c4dedfb at wasmer.rs:8), argc=2, argv=0x000000016fdff6b0) at rt.rs:144:17
    frame #38: 0x0000000100000c94 wasmer`main + 32
    frame #39: 0x00000001b0f7fe50 dyld`start + 2544

The backtrace from the release binary looks to be the same, plus/minus some inlining.

(lldb) run --cranelift modified.wasm
Process 53494 launched: '/Users/work/.cargo/bin/wasmer' (arm64)
Process 53494 stopped

* thread #1, name = 'main', queue = 'com.apple.main-thread', stop reason = signal SIGSEGV
    frame #0: 0x0000000100e32878 wasmer`rkyv::impls::core::_$LT$impl$u20$rkyv..DeserializeUnsized$LT$$u5b$U$u5d$$C$D$GT$$u20$for$u20$$u5b$T$u5d$$GT$::deserialize_unsized::h474124fc1bc68d7a + 96
wasmer`rkyv::impls::core::_$LT$impl$u20$rkyv..DeserializeUnsized$LT$$u5b$U$u5d$$C$D$GT$$u20$for$u20$$u5b$T$u5d$$GT$::deserialize_unsized::h474124fc1bc68d7a:
->  0x100e32878 <+96>:  ldrb   w10, [x10, #0xc]
    0x100e3287c <+100>: orr    x12, x12, x13
    0x100e32880 <+104>: add    x13, x19, x8
    0x100e32884 <+108>: ldr    w14, [x13, #0x8]
Target 0: (wasmer) stopped.

(lldb) bt
* thread #1, name = 'main', queue = 'com.apple.main-thread', stop reason = signal SIGSEGV
  * frame #0: 0x0000000100e32878 wasmer`rkyv::impls::core::_$LT$impl$u20$rkyv..DeserializeUnsized$LT$$u5b$U$u5d$$C$D$GT$$u20$for$u20$$u5b$T$u5d$$GT$::deserialize_unsized::h474124fc1bc68d7a + 96
    frame #1: 0x0000000100e2e644 wasmer`rkyv::impls::core::_$LT$impl$u20$rkyv..DeserializeUnsized$LT$$u5b$U$u5d$$C$D$GT$$u20$for$u20$$u5b$T$u5d$$GT$::deserialize_unsized::h66ac45f55ec792ac + 108
    frame #2: 0x0000000100e10558 wasmer`wasmer_types::serialize::SerializableModule::deserialize_from_archive::h65dfe7c976758f9f + 332
    frame #3: 0x0000000100db325c wasmer`wasmer_compiler::engine::artifact::Artifact::deserialize::h7886269540b120d0 + 536
    frame #4: 0x0000000100da36bc wasmer`wasmer_compiler::engine::inner::Engine::deserialize::h6d6f8b251751f175 + 36
    frame #5: 0x0000000100da38a8 wasmer`wasmer_compiler::engine::inner::Engine::deserialize_from_file::h6d1366cb65581c5a + 272
    frame #6: 0x000000010012f5d8 wasmer`wasmer::sys::module::Module::deserialize_from_file::he67c60e9ae91344c + 64
    frame #7: 0x000000010017378c wasmer`_$LT$wasmer_cache..filesystem..FileSystemCache$u20$as$u20$wasmer_cache..cache..Cache$GT$::load::ha8734d851f5e8d07 + 340
    frame #8: 0x0000000100287154 wasmer`wasmer_cli::commands::run::RunWithPathBuf::inner_execute::h6569e91bda1656bd + 6144
    frame #9: 0x000000010028523c wasmer`wasmer_cli::commands::run::RunWithPathBuf::execute::h6d93e30f17429e76 + 1036
    frame #10: 0x00000001002884ec wasmer`wasmer_cli::commands::run::Run::execute::h2854e8bb0213a2e7 + 220
    frame #11: 0x0000000100378268 wasmer`wasmer_cli::cli::wasmer_main_inner::h53ab1b04fe516ece + 1972
    frame #12: 0x0000000100377ab0 wasmer`wasmer_cli::cli::wasmer_main::h594e7b940421a3e9 + 12
    frame #13: 0x0000000100003f48 wasmer`std::sys_common::backtrace::__rust_begin_short_backtrace::h371acbaf6ba8c1e3 + 12
    frame #14: 0x0000000100003f10 wasmer`std::rt::lang_start::_$u7b$$u7b$closure$u7d$$u7d$::h939c4ceee5e99086 + 16
    frame #15: 0x00000001012c2644 wasmer`std::rt::lang_start_internal::hef2161f9571a51d7 [inlined] core::ops::function::impls::_$LT$impl$u20$core..ops..function..FnOnce$LT$A$GT$$u20$for$u20$$RF$F$GT$::call_once::h7cd8ae72620b0d1f at function.rs:280:13 [opt]
    frame #16: 0x00000001012c2638 wasmer`std::rt::lang_start_internal::hef2161f9571a51d7 [inlined] std::panicking::try::do_call::h0bbb0f423dd9d86c at panicking.rs:492:40 [opt]
    frame #17: 0x00000001012c2638 wasmer`std::rt::lang_start_internal::hef2161f9571a51d7 [inlined] std::panicking::try::hdaabe7e5908702af at panicking.rs:456:19 [opt]
    frame #18: 0x00000001012c2638 wasmer`std::rt::lang_start_internal::hef2161f9571a51d7 [inlined] std::panic::catch_unwind::h7ee653eae81d0a43 at panic.rs:137:14 [opt]
    frame #19: 0x00000001012c2638 wasmer`std::rt::lang_start_internal::hef2161f9571a51d7 [inlined] std::rt::lang_start_internal::_$u7b$$u7b$closure$u7d$$u7d$::he727754da11a45c1 at rt.rs:128:48 [opt]
    frame #20: 0x00000001012c2638 wasmer`std::rt::lang_start_internal::hef2161f9571a51d7 [inlined] std::panicking::try::do_call::h0171064c04d908b7 at panicking.rs:492:40 [opt]
    frame #21: 0x00000001012c2638 wasmer`std::rt::lang_start_internal::hef2161f9571a51d7 [inlined] std::panicking::try::h9c521838fe914345 at panicking.rs:456:19 [opt]
    frame #22: 0x00000001012c2638 wasmer`std::rt::lang_start_internal::hef2161f9571a51d7 [inlined] std::panic::catch_unwind::hac847f960377a9be at panic.rs:137:14 [opt]
    frame #23: 0x00000001012c2638 wasmer`std::rt::lang_start_internal::hef2161f9571a51d7 at rt.rs:128:20 [opt]
    frame #24: 0x0000000100003ef4 wasmer`main + 48
    frame #25: 0x00000001b0f7fe50 dyld`start + 2544

Expected behavior

I would expect the compilation to succeed and then fail to instantiate. Import errors are expected because the binary uses one version of sock_accept(), while WASIX provides an implementation with a slightly different signature.

$ ~/.cargo/bin/wasmer run --singlepass rename-import-namespace/modified.wasm
error: failed to run `rename-import-namespace/modified.wasm`
│   1: failed to instantiate WASI module
╰─▶ 2: Error while importing "wasix_32v1"."sock_accept": incompatible import type. Expected Function(FunctionType { params: [I32, I32, I32], results: [I32] }) but received Function(FunctionType { params: [I32, I32, I32, I32], results: [I32] })

[Michael-F-Bryan]

I was unable to reproduce this on my Linux desktop and @ptitSeb couldn't reproduce this on his MacOS laptop, but I'm still able to reliably reproduce the segfault on my MacOS laptop.

@ptitSeb, I've uploaded my executable to Slack because the 34MB compressed binary is too big for GitHub. If that executable segfaults, it might give you something you can point a debugger at, otherwise we can close this ticket as "can't reproduce".

[Michael-F-Bryan]

@ptitSeb I just checked, and the same issue occurs regardless of whether I compile on nightly (rustc 1.68.0-nightly (659e169d3 2023-01-04)) or the pinned toolchain (stable - rustc 1.64.0).

[ptitSeb]

@Michael-F-Bryan Can you try to build with make build-wasmer then run it with target/release/wasmer run --cranelift modified.wasm?
Also, can you try to add --disable-cache just in case?
I still don't reproduce the issue, with cargo install or with make build-wasmer on my side.

[ptitSeb]

Closing the issue, since Wasmer 3.2.0-alpha.1 has been published, with an invalidation of old cache.