Support using wasm-pack in Worklets (particularly AudioWorklet)

[Smona]

💡 Feature description

One of the key use cases for WebAssembly is audio processing. wasm-pack doesn't currently support AudioWorklets (purportedly the future of custom audio processing on the web) with any of its current --target options.

The web and no-modules targets get close, but error out during instantiation because the AudioWorklet context is lacking several browser APIs which the JS wrappers expect. This problem may extend to other Worker/Worklet contexts, but I've only attempted this with AudioWorklets.

💻 Basic example

my_processor.worklet.js

class MyProcessor extends AudioWorkletProcessor {
  constructor() {
    super()
    import("../pkg/audio").then(module => {
      this._wasm = module
    });
  }

  process(inputs, outputs, parameters) {
    if (!this._wasm) {
      return true
    }

    let output = outputs[0]
    this._wasm.exports.process(this._outPtr, this._size)
    for (let channel = 0; channel < output.length; ++channel) {
      output[channel].set(this._outBuf)
    }

    return true
  }
}

[alexcrichton]

Thanks for the report! Can you clarify why web and no-modules don't work in the audio worklet context? They're intended to be usable for this use case (and I think I've used no-modules there before...), but if there's assumptions baked in which prevent their usage in these locations that's bad!

[Smona]

Sure. Thanks for the quick response!

Using --target web, the following error occurs when trying to call the init function from an AudioWorklet:

audio.js:11 Uncaught ReferenceError: URL is not defined

Using --target no-modules, a very unhelpful DOMException is thrown on import, but it's reasonable to assume the problem is similar. From what I've heard, global variable can't be defined in an AudioWorklet context, and self is undefined.

There is a another problem, considering the Javascript wrapper code would be set up in the AudioWorklet context, and neither classes nor functions can be passed across the AudioWorklet's MessagePort. It would be really cool if wasm-pack included a protocol for calling its exported functions & classes across a MessagePort boundary (maybe with a new --target worklet option). But I would also understand if that's out of scope.

If I could at least use the wrapper code within the worklet, I think it would be a reasonable task for me to write some bridge code to load the generated wrappers in both the worklet and main thread, and pass arguments, object ids, and method/function names as well as return values through the MessagePort, which supports sending number, string, array, and object types.

I've successfully implemented such an interface using raw FFI, and it seems to work quite well. However, not being able to import the generated javascript within the worklet thread means missing out on all of wasm-bindgen's type conversion goodness and being limited to numeric arguments & return types. It also means I have to maintain my own Typescript api interface rather than being able to use auto-generated types. This introduces a big point of failure if the exported Rust functions and wrapper typings get out of sync.

[alexcrichton]

Oh the problem with URL is probably one that we should fix and just check for its existence before trying to use it. Is that the only blocker for --target web?

I agree that we could perhaps explore more extensive integration with audio worklets, but if --target web works that's at least a good start :)

[BenoitDel]

The --target web cannot be used with audioworklet without important modification of the generated file .

• import.meta is not handle well by webpack (See issue 6719)
• instantiateStreaming call throws Mimetype error:
  WebAssembly.instantiateStreaming failed because your server does not serve wasm with application/wasm MIME type`
• When i call postMessage to pass wasm instance to the worklet i ve got another error:
  Failed to execute 'postMessage' on 'MessagePort': #<Memory> could not be cloned

Workarounds:

• pass the exact path of the audioWorkletProcessor to init function from generated_by_wasm_pack.js generated with --target web
• Use instantiate instead instantiateStreaming which doesn't throw (i don't know why but it works)
• instantiate wasm module inside the AudioWorkletGlobalScope.

Ideally, i would like to be able to implement all the patterns from this article (audioworklet and wasm)

• Wasm module instantiation inside AudioWorkletGlobalScope (pattern A)
• Wasm module instantiation inside main.js (pattern B)
• shared memory: not tested yet (pattern C)

I've decided to continue on this issue. But if preferred, i can open another issue. thx

ps: I think dynamic import are not allowed inside AudioWorkletGlobalScope (See issue 506)

[w-ensink]

@BenoitDel Could you elaborate a little bit on the workarounds?
Or do you maybe have an example of how they work?

I'm having a hard time with audio worklets and wasm-pack too. Currently I can only directly instantiate a WebAssembly.Instance by passing the raw bytes of the wasm file (fetched on the main thread) via the postMessage function on the AudioWorkletProcessor subclass, like in this example. This is very limiting, since strings don't work and calling JS functions from Rust is also impossible.

I would love to learn some more about your workarounds!
Thanks in advance.

[BenoitDel]

I do it the same way as you do. ( I am not working on this part of my project for now, but i will in 2-3 weeks. If i find how to implement the other pattern i will tell you about it).
Wasmpack --target web instantiates the wasm module inside the main thread by default (if i remember correctly). I just modify the glue code to return ArrayBuffer and communicate via channel message with the processor part.
I am not sure that calling JS functions from the audioworklet is a good idea, this would certainly involve the garbage collection and so deteriorates performance.
I hope this help

[w-ensink]

Interesting, thanks for the quick reply! The way I did it so far is by ignoring all JS boiler plate and just straight up fetching the data from the .wasm file directly. It kinda works, but only for functions that use numeric types only.
My main hope is to get string passing to work, since it would enable easier parameter control.
I'm still a little bit fuzzy on how exactly you modify the code.
If you could show me an example of how you did it (when you get to it of course), I would be very interested!

[thomaskvnze]

What a coincidence. I started to work on my first project in the audio domain, including rust and wasm and I currently have the same problem. I have not enough experience with rust, warm, and wasm-pack to help with this issue, but I'm happy to contribute when I learned more about the topic and architecture. @w-ensink here is a sample project https://github.com/the-drunk-coder/ruffbox. He uses enums to map parameters. This might be an option to replace the strings while still maintaining readability

[w-ensink]

@cconnection Hey man! I had indeed seen that example, it was one of my goto's to get things working with Rust and AudioWorklets. Enums are a good option for small processors, but I'm trying to build a bigger, more dynamic project and then they're not a real option unfortunately.
Today I have made some real progress and I'm now able to pass strings. Unfortunately I had to say goodbye to Rust for that and fall back on C++ and Emscripten. I love Rust and I really tried for 3 days, but at this point it seems that C++ will be the only tool that can get the job done for me.
If you manage to make any progress with Rust and AudioWorklets, please let me know! :-)

[thomaskvnze]

I have started to experiment with the wasm-pack build --target web output.

The main problem is that we aren't allowed to fetch wasm code asynchronously inside the audio worklet, so we have to fetch the wasm file outside of the worklet and send it to the worklet via postMessage().

When you look on the code generated by wasm-pack build --target web, you see that the init function does everything for us to use it in a regular thread: first fetching with input = fetch(input); and then initiating via WebAssembly.initiate with const { instance, module } = await load(await input, imports); Because of the fetch, we can't use the init function in our audio worklet.

async function init(input) {
    // if (typeof input === 'undefined') {
    //     input = import.meta.url.replace(/\.js$/, '_bg.wasm');
    // }
    const imports = {};
    imports.wbg = {};
    imports.wbg.__wbg_alert_8e68ff37e2340cd2 = function(arg0, arg1) {
        alert(getStringFromWasm0(arg0, arg1));
    };

    if (typeof input === 'string' || (typeof Request === 'function' && input instanceof Request) || (typeof URL === 'function' && input instanceof URL)) {
        input = fetch(input);
    }

    const { instance, module } = await load(await input, imports);

    wasm = instance.exports;
    init.__wbindgen_wasm_module = module;

    return wasm;
}

async function load(module, imports) {
    if (typeof Response === 'function' && module instanceof Response) {

        if (typeof WebAssembly.instantiateStreaming === 'function') {
            try {
                return await WebAssembly.instantiateStreaming(module, imports);

            } catch (e) {
                if (module.headers.get('Content-Type') != 'application/wasm') {
                    console.warn("`WebAssembly.instantiateStreaming` failed because your server does not serve wasm with `application/wasm` MIME type. Falling back to `WebAssembly.instantiate` which is slower. Original error:\n", e);

                } else {
                    throw e;
                }
            }
        }

        const bytes = await module.arrayBuffer();
        return await WebAssembly.instantiate(bytes, imports);

    } else {

        const instance = await WebAssembly.instantiate(module, imports);

        if (instance instanceof WebAssembly.Instance) {
            return { instance, module };

        } else {
            return instance;
        }
    }
}

Currently, I try to split out the fetch function from the init, so that I can fetch the wasm file outside the worker and do the rest of the init process in the worklet with all wasm-pack / wasm-bindgen features. A first prototype looks like this

wasm-pack.js

let wasm;

let cachedTextDecoder = new TextDecoder('utf-8', { ignoreBOM: true, fatal: true });

cachedTextDecoder.decode();

let cachegetUint8Memory0 = null;
function getUint8Memory0() {
    if (cachegetUint8Memory0 === null || cachegetUint8Memory0.buffer !== wasm.memory.buffer) {
        cachegetUint8Memory0 = new Uint8Array(wasm.memory.buffer);
    }
    return cachegetUint8Memory0;
}

function getStringFromWasm0(ptr, len) {
    return cachedTextDecoder.decode(getUint8Memory0().subarray(ptr, ptr + len));
}
/**
*/
export function greet() {
    wasm.greet();
}

async function load(module, imports) {
    if (typeof Response === 'function' && module instanceof Response) {

        if (typeof WebAssembly.instantiateStreaming === 'function') {
            try {
                return await WebAssembly.instantiateStreaming(module, imports);

            } catch (e) {
                if (module.headers.get('Content-Type') != 'application/wasm') {
                    console.warn("`WebAssembly.instantiateStreaming` failed because your server does not serve wasm with `application/wasm` MIME type. Falling back to `WebAssembly.instantiate` which is slower. Original error:\n", e);

                } else {
                    throw e;
                }
            }
        }

        const bytes = await module.arrayBuffer();
        return await WebAssembly.instantiate(bytes, imports);

    } else {
        const instance = await WebAssembly.instantiate(module, imports);
        if (instance instanceof WebAssembly.Instance) {
            return { instance, module };

        } else {
            return instance;
        }
    }
}

export function getImports() {
    const imports = {};
    imports.wbg = {};
    imports.wbg.__wbg_alert_8e68ff37e2340cd2 = function(arg0, arg1) {
        alert(getStringFromWasm0(arg0, arg1));
    };

    return imports
}

export async function fetchWasm(input) {
    if (typeof input === 'undefined') {
        input = import.meta.url.replace(/\.js$/, '_bg.wasm');
    }
    console.log(input);
    if (typeof input === 'string' || (typeof Request === 'function' && input instanceof Request) || (typeof URL === 'function' && input instanceof URL)) {

        input = fetch(input);
    }

    return input;
}

async function init(imports, bytes) {
    console.log(imports, bytes);
    const { instance, module } = await load(bytes, imports);

    wasm = instance.exports;
    init.__wbindgen_wasm_module = module;

    return wasm;
}

export default init;

Now I can use it like this in the index.html and audio.js

////////////// index.html
<script type="module">
    import { fetchWasm } from './wasm-pack.js';

    async function run() {
        const binary = await fetchWasm('debug/wasm-pack.wasm?)
        const arrbuf = await binary.arrayBuffer();
        await context.audioWorklet.addModule('audio.js');
        const audioNode = new AudioWorkletNode(context, 'audioNode');
        audioNode.port.postMessage({type: 'loadWasm', data: arrbuf});
        audioNode.connect(context.destination);
    }

    run();
</script>

/////////////// audio.js
import init, {getImports} from './wasm-pack.js';

class Processor extends AudioWorkletProcessor {
    static get parameterDescriptors() {
        return []
    }

    constructor(options) {
        super(options);
        this.port.onmessage = message => {
            switch (message.data.type) {
                case 'loadWasm': {
                    const instantiate = async () => {
                        try {
                            const imports = getImports();
                            this._wasm = await init(imports, message.data.data)

                        } catch(e) {
                            console.log("Error in loading backend: ", e)
                        }
                    }

                    instantiate();
                }
            }
        }
    }

    process(inputs, outputs, parameters) {
      // processing
    }
}

registerProcessor('audioNode', Processor);

Now I run into one problem. I get the following message:

Uncaught (in promise) TypeError: WebAssembly.instantiate(): Import #0 module="__wbindgen_placeholder__" error: module is not an object or function

First I thought that it is related to audio worklet, but actually I get the same error when using the normal output from wasm-pack --target web directly in my index.html. Is this is a bug?

/////// index.html
<html>
<head>
    <meta content="text/html;charset=utf-8" http-equiv="Content-Type"/>
</head>
<body>
<script type="module">
    import init from './wasm-pack.js';

    async function run() {
        const wasm = await init('debug/kunzmod_backend.wasm');
    }

    run();
</script>
</body>
</html>

[BenoitDel]

I think there is no notion of import in worklet.
You have to use channel messaging to pass data between main and worklet process or use shared memory.

import init, {getImports} from './wasm-pack.js'; // this is false

[thomaskvnze]

The import works, I can call getImports and see console.logs. same for the init.
I'm still experimenting. Next think I found out: there is no textencoder/textdecoder in the AudioWorklet. So it's necessary to have a utf8 decoder to parse the uint8 array to strings.

[w-ensink]

What also might be interesting to note, is that the import works with the Emscripten version too, as can be seen in this example from Google.
@cconnection have you managed to get full string functionality with your method?

[BenoitDel]

Yes, you re right it is just dynamic import (i hope this time i am right) that is not supported. Static import is in the specs and works in chromium (at least). Sorry for the mistake.
But it should be avoided in a real time thread as most as possible. WebAudio/web-audio-api#2194

[thomaskvnze]

I just managed to have a working prototyp, including strings. At the end, wasm-pack generates glue code which decodes and encodes the uint8arrays to a javascript string in UTF-8 and back. Currently, my prototyp is a complete manual editing of the generated code. So I can't build a new version with wasm-pack because the code will be overwritten.

The next steps:
Make it possible to build the glue code which can be used for audio workers with wasm-pack directly. This also includes to introduce a text encoder and decoder library because the standard API from the browser is not available in the audio worker. I searched for a library which conforms to the standard API, so that the wasm-pack code should run without changing it. Here is one: https://github.com/anonyco/FastestSmallestTextEncoderDecoder

I don't know enough about wasm-bindgen and wasm-pack, so I'm not sure if its a bindgen problem or wasm-pack problem or both. Maybe @alexcrichton can help us to go to the right direction? I can provide a sample project to show how the current code looks like.

Update:
I found the code which generates the javascript.
Wasm-bindgen cli-support crate js/mod.rs
It's in wasm-bindgen. So we need a feature request in order to have proper generated glue code which can be used in audio worklets. Also, be aware that the encoding and decoding of strings have a cost.

[alexcrichton]

Ah if worklets have such a restrictive environment then yes, we'd need to add a new target to wasm-bindgen which included the above shim or somehow referenced it.

[thomaskvnze]

@alexcrichton how do we continue here? Should we open an issue in the wasm-bindgen repo? Is it possible to help integrating it into wasm-bindgen?

[alexcrichton]

Yeah I think it's best to have either an issue or a PR to wasm-bindgen to figure out the best way to move forward.

[stekyne]

Well, is there any real workaround for this?

[Smona]

Unfortunately, I was not able to use wasm-pack due to this issue. What I ended up doing is:

• Run cargo build && wasm-bindgen --target web --omit-default-module-path as a manual build step.
• Polyfill TextEncoder and TextDecoder in my worklet.ts processor definition, and import the wasm-bindgen typescript output there.
• Configure my bundler to build a separate JS file for the worklet.ts entrypoint, and return a URL where it can be fetched.
• In the main thread:
  • add the AudioWorklet module using that URL, and then initialize a node.
  • Fetch the .wasm output of wasm-bindgen, and send it in a load message across the node's MessagePort, as an ArrayBuffer.
• In the worklet thread, handle this load message by passing the wasm data to wasm-bindgen's init() default export.
• Create message handlers in the worklet thread to provide an interface between the main thread and wasm-bindgen generated typescript. I'm using Comlink for this, which greatly simplifies communication between the threads, and lets me define the message handlers as simple (typed) methods on the worklet processor class.

While this was pretty painful to figure out, it does give you all the benefits of wasm-bindgen in an AudioWorklet context, and the DX is pretty good once everything's set up. Hopefully this can provide some inspiration for a new wasm-pack mode for AudioWorklets, but there's some more work needed to arrive at an implementation that's generic enough to provide in a library, and supports all the various bundlers people might be using.

[Marcel-G]

I also got things working similarly to how @Smona describes. I tried to package it up to make this process easier for myself in the future https://github.com/Marcel-G/waw-rs