Hi, Evan from Useful Sensors/Moonshine here. Just letting you know this is on our radar. We're working on Transformers support right now, and we (internally) have our current ONNX models running in the browser with onnxruntime-web. Shouldn't be too difficult to get support added to transformers.js from there.

Hi, Evan from Useful Sensors/Moonshine here. Just letting you know this is on our radar. We're working on Transformers support right now, and we (internally) have our current ONNX models running in the browser with onnxruntime-web. Shouldn't be too difficult to get support added to transformers.js from there.

@evmaki Not related to this issue, but I am also eagerly waiting for the ability to finetune for supporting a new language.

Hi, Evan from Useful Sensors/Moonshine here. Just letting you know this is on our radar. We're working on huggingface/transformers#34474 right now, and we (internally) have our current ONNX models running in the browser with onnxruntime-web. Shouldn't be too difficult to get support added to transformers.js from there.

@evmaki Great to hear! I have been following the ONNX support and it looks like a great start! One issue is that you currently export two versions of the decoder (w/ and w/o PKVs), leading to weight duplication (more of a problem when running in the browser since we load the decoder twice).

We were able to solve this in Optimum by adding an If node to the graph and then choosing which path to take based on whether the past key values are provided. See here for an example. And here is the code used to merge the two decoders.

I was experimenting with your codebase to pass zero-sized tensors as input, but I get gibberish output.

Either way, once we have transformers support, I expect this to be much easier to convert (since the input/output signatures should be similar to whisper).

I have some exciting news: we've got a working version of moonshine-tiny (See PR: #1099) which offers numerous benefits over the original/upstream ONNX implementation:

1. Huge reduction in model size (285MB → 109MB), with zero quality loss:
   Original = 285MB (7+30+120+128) at fp32
   
   Ours = 109MB (31 + 78) at fp32

   This is due to the following optimizations:

   • Deduplicated tied weights (the original model also suffers from this problem, which is why it displays as 46.5M params instead of the correct 27M params)
   • Merging the two decoders (w/ and w/o PKV inputs) into a single graph

2. Multiple quantization support: int8/uint8, q4, fp16, q4f16. We are facing an issue with fp16/q4f16 decoder (cc @guschmue), but all other quantizations work. At q4 quantization, which still works pretty well, the model size is further reduced to 55MB (44.7+10.4)

cc @evmaki - I think these benefits can also be upstreamed, and I'd be happy to make a PR if you'd like! 🤗

@xenova That's fantastic! Please do open a PR – excited to take a look!