Add: WASM Emscripten example

[raphaelmenges]

Hello 👋,

I noticed that support for the WASM (WASI?) target in this crate has been officially dropped. However, I recently encountered a use case that requires me running onnxruntime on the Web. This led me to investigate whether ort could work within an Emscripten environment.

I discovered that this crate can be used as-is with the wasm32-unknown-emscripten Rust compilation target! While the setup can get a bit complex—especially when enabling multi-threading in onnxruntime—it works well. I would love to see this example merged into the official repository, but I also understand if it is considered too experimental to be officially endorsed.

Here’s a .gif for motivation, showcasing my example using the YoloV8 model to classify objects in pictures on Chromium:

---

The Less Crazy Part

Microsoft does not provide precompiled static libraries for WASM, so I created a GitHub Actions workflow to handle this. The generated libonnxruntime.a can be linked with ort as usual—even when targeting wasm32-unknown-emscripten.

To expose Rust functions to JavaScript, the Rust main file must provide a C interface, which Emscripten can export. I use rust-embed to bundle the .onnx model into the .wasm. An index.html file then incorporates the .js and .wasm outputs, which include the compiled Rust code, onnxruntime, and the model.

Additionally, the Emscripten SDK version used by the Rust compiler must match the exact version used by onnxruntime for successful linking.

The Crazy Part

Things get trickier when enabling multi-threading in onnxruntime. Since v1.19.0, Microsoft recommends enabling multi-threading using the --enable_wasm_threads build flag. This links libonnxruntime.a to pthread, meaning all linked objects—including Rust’s standard library—must also be compiled with pthread support.

However, Rust’s standard library is not compiled this way by default, so you must switch to Rust nightly and compile the Rust standard library with +atomics,+bulk-memory,+mutable-globals.

Additionally, the server must set specific CORS flags, as multi-threaded Emscripten uses SharedArrayBuffer in the Web browser, which requires these settings.

Verdict

I am already opening the pull request as a draft to get early feedback. However, at least following ToDos are pending before a pull request could happen:

• Enable debug builds of the example.
• Add CI to automatically test the target.
• Use/enhance ort's precompiled static libonnxruntime mechanism.

In the future, I would love to see execution providers for the Web to be available in ort:

• Add XNNPACK execution provider. Cannot be built at the moment.
• Add WebGL execution provider. Stable but deprecated. Might be the best option for now? Seems to exposed to the JavaScript world, only.
• Add WebGPU execution provider. Still requires Firefox Nightly, but is the most promising future option. However, might not yet be ready.
• Add WebNN execution provider. WebNN is not yet a standard.

[raphaelmenges]

• Add CI to automatically test the target.

I imagine an even more minimal example / test setup than YoloV8 that is first compiled and then run in the CI within a headless Chromium. We could check the developer console output to determine whether the model was correctly inferred. What do you think? Which model would be suited? Any good example how to use headless Chromium in GitHub workflows? Would this become its own workflow or rather a job in an existing workflow?

PS: I guess we could use puppeteer to execute JavaScript code without any .html?

[decahedron1]

YOLOv8 is already a pretty good example model - not too simple like the MNIST model, but not too large like GPT-2. I don't know off the top of my head if either supports WASM threads but we should give Node.js or ideally Deno a shot before jumping straight to headless Chromium for CI.

Btw, great work here! Until now I didn't think it was even possible to get wasm32-unknown-emscripten working 😅

[raphaelmenges]

I don't know off the top of my head if either supports WASM threads but we should give Node.js or ideally Deno a shot before jumping straight to headless Chromium for CI.

Sure, I can use Deno to test the .wasm build. I am wondering what would be the best approach to integrate such a test into this repository:

• We require Rust code that is compiled into the .wasm. Where should it live? I guess inside the tests directory, only Rust tests should live? Should I reuse the code from the example for the test? Feels also strange to use an example project for testing. Perhaps I can create a new top-level directory tests-wasm and put all relevant files there?
• In the workflow, the Rust code should be compiled into the .wasm. Should that happen in an existing workflow like the test.yml and there be a job on its own?

[raphaelmenges]

• Use/enhance ort's precompiled static libonnxruntime mechanism.

I have observed that you have added mechanisms to provide a precompiled static libonnxruntime for the Emscripten environment. However, when I try to use the download-binaries feature, cargo build panics with "downloaded binaries not available for target wasm32-unknown-emscripten". Am I doing something wrong, or do require any contribution in that regard from me or should I leave the current mechanism to get static libonnxruntime in the example code as-is?

[decahedron1]

Like I said in #349 I couldn't get binaries for 1.20.2 but they should be available for 1.21. Keep the current binary mechanism, and don't worry about CI, I'll handle those when 1.21 rolls around 👍

[raphaelmenges]

FYI, onnxruntime main branch is now using Emscripten v4.0.3: microsoft/onnxruntime@e6ae1bb

[raphaelmenges]

I have pulled the latest changes from main to my branch and tried to enable the WebGPU ep. However, I get the following error at linking:

 = note: wasm-ld: error: /var/folders/z1/_strmfhd7dlcf0ypdhm9vxvm0000gn/T/rustczlLYdj/libort_sys-77bcb0b41cbb8fd7.rlib(graph.cc.o): undefined symbol: std::__2::__fs::filesystem::path::__parent_path() const
          wasm-ld: error: /var/folders/z1/_strmfhd7dlcf0ypdhm9vxvm0000gn/T/rustczlLYdj/libort_sys-77bcb0b41cbb8fd7.rlib(graph.cc.o): undefined symbol: std::__2::__fs::filesystem::path::__root_directory() const
          wasm-ld: error: /var/folders/z1/_strmfhd7dlcf0ypdhm9vxvm0000gn/T/rustczlLYdj/libort_sys-77bcb0b41cbb8fd7.rlib(graph.cc.o): undefined symbol: std::__2::__fs::filesystem::path::__filename() const
          wasm-ld: error: /var/folders/z1/_strmfhd7dlcf0ypdhm9vxvm0000gn/T/rustczlLYdj/libort_sys-77bcb0b41cbb8fd7.rlib(graph_partitioner.cc.o): undefined symbol: std::__2::__fs::filesystem::__status(std::__2::__fs::filesystem::path const&, std::__2::error_code*)

I have removed the default-features of ort, which should make it free of std? Thus, I do not understand why it tries to link something from std at all.

Another cause for the issue could be the upgrade to Emscripten v4.0.3.

Any glue?

[decahedron1]

Those symbols are C++'s std, not Rust. It's definitely caused by the Emscripten upgrade. Was the ONNX Runtime binary compiled with Emscripten 4.0.3?

[raphaelmenges]

It should have been, according to the log files.

Update: Soon it must be Emscripten 4.0.4 :)

[xixioba]

Is there any progress? I tried to reproduce but failed. I am looking forward to the final merge.
Also, can WASI be supported, such as using wasmtime to run ort

[decahedron1]

Also, can WASI be supported, such as using wasmtime to run ort

Pure WASI is a no, unfortunately. wasmtime specifically has an ort backend for their WASI-NN implementation though 😊

[decahedron1]

@raphaelmenges Did you ever get to the bottom of that linking issue? And do you need my help on anything here? Looks like ONNX Runtime v1.21 is coming up soon out now and I'd like to get that + WASM out in rc.10 😊

[raphaelmenges]

I have not further investigated the issue, yet. I can do so next week sometime, if this is early enough for you!

[raphaelmenges]

I have upgraded my code to include your latest changes and your precompiled onnxruntime from pyke.io. However, I get the following linker error:

  = note: wasm-ld: error: unable to find library -lwebgpu_dawn

I then removed linking to webgpu_dawn from ort-sysbecause I assumed that Dawn is only required on Desktop and Mobile to provide a WebGPU environment. However, then I am confronted with the following linker errors:

  = note: wasm-ld: error: lto.tmp: undefined symbol: emwgpuDelete
          wasm-ld: error: lto.tmp: undefined symbol: emwgpuDelete
          wasm-ld: error: lto.tmp: undefined symbol: emwgpuDelete
          wasm-ld: error: lto.tmp: undefined symbol: emwgpuDelete
          wasm-ld: error: lto.tmp: undefined symbol: emwgpuDelete
          wasm-ld: error: lto.tmp: undefined symbol: emwgpuDelete
          wasm-ld: error: lto.tmp: undefined symbol: emwgpuDelete
          wasm-ld: error: lto.tmp: undefined symbol: emwgpuDelete
          wasm-ld: error: lto.tmp: undefined symbol: emwgpuDelete
          wasm-ld: error: lto.tmp: undefined symbol: emwgpuDelete
          wasm-ld: error: lto.tmp: undefined symbol: emwgpuInstanceRequestAdapter
          wasm-ld: error: lto.tmp: undefined symbol: wgpuAdapterHasFeature
          wasm-ld: error: lto.tmp: undefined symbol: wgpuAdapterHasFeature
          wasm-ld: error: lto.tmp: undefined symbol: wgpuAdapterHasFeature
          wasm-ld: error: lto.tmp: undefined symbol: wgpuAdapterHasFeature
          wasm-ld: error: lto.tmp: undefined symbol: wgpuAdapterGetLimits
          wasm-ld: error: lto.tmp: undefined symbol: emwgpuDelete
          wasm-ld: error: lto.tmp: undefined symbol: emwgpuAdapterRequestDevice
          wasm-ld: error: lto.tmp: undefined symbol: emwgpuDelete
          wasm-ld: error: lto.tmp: undefined symbol: emwgpuDelete
          wasm-ld: error: too many errors emitted, stopping now (use -error-limit=0 to see all errors)

I have then checked the logs of your precompiled onnxruntime build and indeed, Dawn seems to be used in the Emscripten build as well. I could not find a webgpu_dawn library in the outputs, though. I am left confused.

---

In the release note of v1.21 it is also mentioned that onnxruntime now supports WASM64. Should we do something about that?

[decahedron1]

Dawn is indeed required for Emscripten too. Does the -sUSE_WEBGPU emcc flag help with that linker error at all? If not we can forego WebGPU on WASM for now and stick to a vanilla build - after all, with the new alternative backends feature, there could be a wonnx backend which also gives us the benefit of WebGPU for wasm32-unknown-unknown.

Let's get WASM32 working before looking into WASM64. Browser support doesn't seem too good anyways (though better than WebGPU to be fair 😶)

[raphaelmenges]

I tried wonnx some months ago and the development appears to me rather slow or even dead, sadly. I am not sure whether integrating it into ort will bring real benefit to anyone but increase your maintenance effort.

Anyway, I am very fine if you decide to not to include the WebGPU ep in the WASM example at the moment. I am also happy if you take over this pull request now and make any desired changes you want, i.e., remove the WebGPU ep code etc!

[decahedron1]

Hmm, those emwgpu functions are defined in Dawn's own version of the script included by -sUSE_WEBGPU. I don't know how difficult it'll be to include that script but I'm starting a non-WebGPU build for wasm32-unknown-emscripten just in case. We can always revisit WebGPU on WASM at a later date, maybe when it's no longer so experimental (unless its like ort's rc situation 🫠)

[raphaelmenges]

In another context I just realized that the YoloV8 model you provide has now a different URL: https://cdn.pyke.io/0/pyke:ort-rs/[email protected]/yolov8m.onnx

Can you fix that link in the WASM example code?

[decahedron1]

I don't think I can push to your branch because it's an organization-owned fork, but I updated the model URL and did another minor tweak to fix the name of the output .js file and it's all working! Next is WebGPU =)

EDIT: yeah, no WebGPU for now. It links fine and acknowledges the EP but complains of a Rust panic during the call to SessionOptionsAppendExecutionProvider with an 'empty' message, which makes no sense. Debugging WASM is a nightmare and each cargo build takes 5 minutes so I'm throwing in the towel. At least the CPU EP works great!

EDIT 2: seems like the WebGPU build is just broken in general as removing the registration makes it panic at a different point. I still don't know where it would be panicking with an empty message. This type of weirdness only ever happens on WASM 🙃

[raphaelmenges]

EDIT 2: seems like the WebGPU build is just broken in general as removing the registration makes it panic at a different point. I still don't know where it would be panicking with an empty message. This type of weirdness only ever happens on WASM 🙃

This might be the pull request to track about the status of the WebGPU ep in WASM: microsoft/onnxruntime#23697

I am using WebGPU ep on macOS already successfully btw. :)

[decahedron1]

Opened #363 with my changes, thank you for all your hard work here!

[raphaelmenges]

EDIT 2: seems like the WebGPU build is just broken in general as removing the registration makes it panic at a different point. I still don't know where it would be panicking with an empty message. This type of weirdness only ever happens on WASM 🙃

According to @fs-eire, the WebGPU ep should work now in WASM: microsoft/onnxruntime#23072 (comment)

[raphaelmenges]

WebGPU ep via WASM now replaces the JSEP (JavaScript Execution Provider) in the onnxruntime for Web: microsoft/onnxruntime@2c041e0

I guess the WebGPU ep should work now in WASM? I have no time at the moment to play around with that, sadly.

[decahedron1]

@raphaelmenges I am working on WebGPU support with wasm32-unknown-unknown; hoping for release by late July or early August.

[raphaelmenges]

Sounds promising, cool! Do you still use Emscripten to build onnxruntime itself then?

[andrenatal]

Thanks for all the work on this, guys. I'm trying to reproduce it and build the wasm example. I needed to first revert my branch to the commit that checked it it, otherwise I'd be hit with some tls requirements, and then tried to build with the webgpu feature on ubuntu 24 and hit the same issue:

          cache:INFO:  - ok
          wasm-ld: error: lto.tmp: undefined symbol: emwgpuDelete
          wasm-ld: error: lto.tmp: undefined symbol: emwgpuDelete
          wasm-ld: error: lto.tmp: undefined symbol: emwgpuDelete
          wasm-ld: error: lto.tmp: undefined symbol: emwgpuDelete
          wasm-ld: error: lto.tmp: undefined symbol: emwgpuDelete
          wasm-ld: error: lto.tmp: undefined symbol: emwgpuDelete
          wasm-ld: error: lto.tmp: undefined symbol: emwgpuDelete
          wasm-ld: error: lto.tmp: undefined symbol: emwgpuDelete
          wasm-ld: error: lto.tmp: undefined symbol: emwgpuDelete
          wasm-ld: error: lto.tmp: undefined symbol: emwgpuDelete
          wasm-ld: error: lto.tmp: undefined symbol: emwgpuInstanceRequestAdapter
          wasm-ld: error: lto.tmp: undefined symbol: wgpuAdapterHasFeature
          wasm-ld: error: lto.tmp: undefined symbol: wgpuAdapterHasFeature
          wasm-ld: error: lto.tmp: undefined symbol: wgpuAdapterHasFeature
          wasm-ld: error: lto.tmp: undefined symbol: wgpuAdapterHasFeature
          wasm-ld: error: lto.tmp: undefined symbol: wgpuAdapterGetLimits
          wasm-ld: error: lto.tmp: undefined symbol: emwgpuDelete
          wasm-ld: error: lto.tmp: undefined symbol: emwgpuAdapterRequestDevice
          wasm-ld: error: lto.tmp: undefined symbol: emwgpuDelete
          wasm-ld: error: lto.tmp: undefined symbol: emwgpuDelete

[andrenatal]

I've also tried to change it to use the ort-tract backend, using following Cargo.toml and these are the issues I hit.

[package]
publish = false
name = "example-wasm-emscripten"
version = "0.0.0"
edition = "2021"

[[bin]]
path = "src/main.rs"
name = "wasm_example"

[dependencies]
ort-tract = "0.1.0+0.21"
#ort = { path = "../../", default-features = false, features = ["ndarray", "download-binaries"] }
ndarray = "0.16"
image = "0.25"

[build-dependencies]
glob = "0.3"
reqwest = { version = "0.12", features = ["blocking"] }

[dependencies.ort]
version = "=2.0.0-rc.10"
default-features = false # Disables the `download-binaries` feature since we don't need it
features = [
    "alternative-backend"
]

error[E0277]: the trait bound `ArrayBase<OwnedRepr<f32>, Dim<[usize; 4]>>: OwnedTensorArrayData<_>` is not satisfied
   --> src/main.rs:77:34
    |
 77 |         ort::value::Tensor::from_array(ndarray::Array4::from_shape_fn((1, 3, 640, 640), |(_, c, y, x)| image640[(x as u32, y as u32)][c] as f32 / 255_f32))
    |         ------------------------------ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ the trait `OwnedTensorArrayData<_>` is not implemented for `ArrayBase<OwnedRepr<f32>, Dim<[usize; 4]>>`
    |         |
    |         required by a bound introduced by this call
    |
    = help: the following other types implement trait `OwnedTensorArrayData<I>`:
              (D, Box<[T]>)
              (D, Vec<T>)
note: required by a bound in `value::impl_tensor::create::<impl Value<TensorValueType<T>>>::from_array`
   --> /home/andre/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/ort-2.0.0-rc.10/src/value/impl_tensor/create.rs:141:32
    |
141 |     pub fn from_array(input: impl OwnedTensorArrayData<T>) -> Result<Tensor<T>> {
    |                                   ^^^^^^^^^^^^^^^^^^^^^^^ required by this bound in `value::impl_tensor::create::<impl Value<TensorValueType<T>>>::from_array`

error[E0599]: no method named `try_extract_array` found for struct `Value<Type>` in the current scope
  --> src/main.rs:81:34
   |
81 |     let output = outputs["output0"].try_extract_array::<f32>().unwrap().t().into_owned();
   |                                     ^^^^^^^^^^^^^^^^^
   |
help: there is a method `try_extract_scalar` with a similar name
   |
81 -     let output = outputs["output0"].try_extract_array::<f32>().unwrap().t().into_owned();
81 +     let output = outputs["output0"].try_extract_scalar::<f32>().unwrap().t().into_owned();
   |

Some errors have detailed explanations: E0277, E0599.
For more information about an error, try `rustc --explain E0277`.
error: could not compile `example-wasm-emscripten` (bin "wasm_example") due to 2 previous errors

[decahedron1]

@andrenatal Regarding the last error, you just need to add the ndarray feature back.

[andrenatal]

Thanks @decahedron1 ! You mean add ndarray feature, to the ort-tract dependency?

[decahedron1]

Add it to the ort dependency like you had it before:

 [dependencies.ort]
 version = "=2.0.0-rc.10"
 default-features = false
 features = [
+    "ndarray",
     "alternative-backend"
 ]

[andrenatal]

Thank you @decahedron1 .

I'll try to reproduce and build the wasm example, using both tract and onnxruntime (with and without webgpu). I'll use osx arm64 as the builder platform and if I hit roadblocks will switch to ubuntu 24 x86.

So I first tried tract on osx and it built after I added the ndarray feature back, but I'm getting the following panic when testing on chrome:

thread '<unnamed>' panicked at /Users/anatal/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/ort-2.0.0-rc.10/src/environment.rs:104:9:
put_char @ wasm_example.js:2451
write @ wasm_example.js:2397
write @ wasm_example.js:4121
doWritev @ wasm_example.js:5445
_fd_write @ wasm_example.js:5469
$write @ wasm_example.wasm:0x45cb9fb
$std::sys::pal::unix::fd::FileDesc::write::hf5b8d9832fbafe14 @ wasm_example.wasm:0x44a2231
$_$LT$std..sys..stdio..unix..Stderr$u20$as$u20$std..io..Write$GT$::write::h3fbd68153c2d71a4 @ wasm_example.wasm:0x4478062
$std::io::Write::write_all::hab93524822083eb1 @ wasm_example.wasm:0x4478278
$std::panicking::default_hook::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::he2832d7b53aa3bbd @ wasm_example.wasm:0x44aa372
invoke_viii @ wasm_example.js:6030
$std::thread::with_current_name::_$u7b$$u7b$closure$u7d$$u7d$::h9eb57a062da35ffc @ wasm_example.wasm:0x4475900
invoke_vii @ wasm_example.js:6019
$std::thread::current::try_with_current::hd90b4eff8dae5b82 @ wasm_example.wasm:0x446ce1f
$std::thread::with_current_name::haa1d5c77a9befead @ wasm_example.wasm:0x4474fbd
invoke_vi @ wasm_example.js:6063
$std::panicking::default_hook::_$u7b$$u7b$closure$u7d$$u7d$::hf6eb6359c4ca21da @ wasm_example.wasm:0x44a9ad3
invoke_viii @ wasm_example.js:6030
$std::panicking::default_hook::h3a6d6ac078ddcc7a @ wasm_example.wasm:0x44a966a
invoke_vi @ wasm_example.js:6063
$std::panicking::rust_panic_with_hook::h9b6223228dbbfbde @ wasm_example.wasm:0x44abe0c
invoke_viiiii @ wasm_example.js:6052
$std::panicking::begin_panic_handler::_$u7b$$u7b$closure$u7d$$u7d$::h36c953dfad1b4507 @ wasm_example.wasm:0x44ab435
$std::sys::backtrace::__rust_end_short_backtrace::hd97a34d0ab63383b @ wasm_example.wasm:0x44d8330
$rust_begin_unwind @ wasm_example.wasm:0x44aa8e6
$core::panicking::panic_fmt::h06df4830ca26db72 @ wasm_example.wasm:0x4576f89
$core::result::unwrap_failed::hd151bf4a42da4819 @ wasm_example.wasm:0x45511b8
invoke_viiiii @ wasm_example.js:6052
$core::result::Result$LT$T$C$E$GT$::expect::h7eede18592993e83 @ wasm_example.wasm:0x4430728
$_$LT$ort..environment..GlobalThreadPoolOptions$u20$as$u20$core..default..Default$GT$::default::h8e05cbce53aa0ca3 @ wasm_example.wasm:0x440773a
invoke_vi @ wasm_example.js:6063
$detect_objects @ wasm_example.wasm:0xdf57d
(anonymous) @ wasm_example.js:941
(anonymous) @ (index):42Understand this error
wasm_example.js:2451 failed to create threading options: Error { code: NotImplemented, msg: "Unimplemented" }

which source is here:

impl Default for GlobalThreadPoolOptions {
	fn default() -> Self {
		let mut ptr = ptr::null_mut();
		ortsys![unsafe CreateThreadingOptions(&mut ptr).expect("failed to create threading options")];
		Self { ptr, thread_manager: None }
	}
}

My changes were this: https://github.com/andrenatal/ort/pull/1/files

[decahedron1]

tract doesn't support configuring a global thread pool, so simply deleting the line below set_api that configures it (starting with .with_global_thread_pool) is fine.

Let's move this to a new issue/discussion, I don't want to flood the other participants inboxes =)