Skip to content

Add JavaScript (WebAssembly) API for PocketTTS #3163

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
csukuangfj merged 4 commits into from
Feb 10, 2026
Merged

Add JavaScript (WebAssembly) API for PocketTTS #3163

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
01d57fb
Add JavaScript (WebAssembly) API for PocketTTS
csukuangfj Feb 9, 2026
d243350
Fixes after review
csukuangfj Feb 9, 2026
6d5f357
Fixes after review
csukuangfj Feb 10, 2026
921cbc4
More fixes
csukuangfj Feb 10, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 36 additions & 0 deletions nodejs-examples/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,42 @@ node ./test-offline-speaker-diarization.js

In the following, we demonstrate how to run text-to-speech.

## ./test-offline-tts-pocket-en.js

[./test-offline-tts-pocket-en.js](./test-offline-tts-pocket-en.js)
shows how to use PocketTTS for Zero-shot TTS.

You can use the following command to run it:
```
curl -SL -O https://github.com/k2-fsa/sherpa-onnx/releases/download/tts-models/sherpa-onnx-pocket-tts-int8-2026-01-26.tar.bz2
tar xf sherpa-onnx-pocket-tts-int8-2026-01-26.tar.bz2
rm sherpa-onnx-pocket-tts-int8-2026-01-26.tar.bz2

node ./test-offline-tts-pocket-en.js
```

Note: Since our current version of onnxruntime is 1.17.1, you need to run

```python3
# Switch to ./sherpa-onnx-pocket-tts-int8-2026-01-26 and run

import onnx

for filename in [
"./lm_flow.int8.onnx",
"./lm_main.int8.onnx",
"./encoder.onnx",
"./decoder.int8.onnx",
"./text_conditioner.onnx",
]:
model = onnx.load(filename)
print("old", model.ir_version)
model.ir_version = 9
print("new", model.ir_version)
onnx.save(model, filename)
```
to modify the models.

## ./test-offline-tts-kitten-en.js

[./test-offline-tts-kitten-en.js](./test-offline-tts-kitten-en.js) shows how to use
Expand Down
1 change: 0 additions & 1 deletion nodejs-examples/test-offline-medasr-ctc.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,6 @@
//
const fs = require('fs');
const {Readable} = require('stream');
const wav = require('wav');

const sherpa_onnx = require('sherpa-onnx');

Expand Down
51 changes: 51 additions & 0 deletions nodejs-examples/test-offline-tts-pocket-en.js
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
// Copyright (c) 2026 Xiaomi Corporation (authors: Fangjun Kuang)

const sherpa_onnx = require('sherpa-onnx');

function createOfflineTts() {
let pocket = {
lmFlow: './sherpa-onnx-pocket-tts-int8-2026-01-26/lm_flow.int8.onnx',
lmMain: './sherpa-onnx-pocket-tts-int8-2026-01-26/lm_main.int8.onnx',
encoder: './sherpa-onnx-pocket-tts-int8-2026-01-26/encoder.onnx',
decoder: './sherpa-onnx-pocket-tts-int8-2026-01-26/decoder.int8.onnx',
textConditioner:
'./sherpa-onnx-pocket-tts-int8-2026-01-26/text_conditioner.onnx',
vocabJson: './sherpa-onnx-pocket-tts-int8-2026-01-26/vocab.json',
tokenScoresJson:
'./sherpa-onnx-pocket-tts-int8-2026-01-26/token_scores.json',
};
let offlineTtsModelConfig = {
offlineTtsPocketModelConfig: pocket,
numThreads: 1,
debug: 1, // set it to 1 to see verbose logs; 0 to disable logs
provider: 'cpu',
};

let offlineTtsConfig = {
offlineTtsModelConfig: offlineTtsModelConfig,
maxNumSentences: 1,
};

return sherpa_onnx.createOfflineTts(offlineTtsConfig);
}

const referenceWaveFilename =
'./sherpa-onnx-pocket-tts-int8-2026-01-26/test_wavs/bria.wav';
const wave = sherpa_onnx.readWave(referenceWaveFilename);

const generationConfig = {
silenceScale: 0.2,
referenceAudio: wave.samples,
referenceSampleRate: wave.sampleRate,
numSteps: 5,
extra: {max_reference_audio_len: 12}
};
Comment thread
coderabbitai[bot] marked this conversation as resolved.

const tts = createOfflineTts();
const text =
'Today as always, men fall into two groups: slaves and free men. Whoever does not have two-thirds of his day for himself, is a slave, whatever he may be: a statesman, a businessman, an official, or a scholar.';

const audio = tts.generateWithConfig(text, generationConfig);
tts.save('./test-pocket-en.wav', audio);
console.log('Saved to test-pocket-en.wav successfully.');
tts.free();
2 changes: 1 addition & 1 deletion wasm/asr/CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ set(MY_FLAGS " -s FORCE_FILESYSTEM=1 -s INITIAL_MEMORY=512MB -s ALLOW_MEMORY_GRO
string(APPEND MY_FLAGS " -sSTACK_SIZE=10485760 ") # 10MB
string(APPEND MY_FLAGS " -sEXPORTED_FUNCTIONS=[_CopyHeap,_malloc,_free,${all_exported_functions}] ")
string(APPEND MY_FLAGS "--preload-file ${CMAKE_CURRENT_SOURCE_DIR}/assets@. ")
string(APPEND MY_FLAGS " -sEXPORTED_RUNTIME_METHODS=['ccall','stringToUTF8','setValue','getValue','lengthBytesUTF8','UTF8ToString'] ")
string(APPEND MY_FLAGS " -sEXPORTED_RUNTIME_METHODS=['ccall','stringToUTF8','setValue','getValue','lengthBytesUTF8','UTF8ToString','HEAPU8','HEAP16','HEAP32','HEAPF32','HEAPF64'] ")

message(STATUS "MY_FLAGS: ${MY_FLAGS}")

Expand Down
26 changes: 23 additions & 3 deletions wasm/asr/sherpa-onnx-asr.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -849,13 +849,15 @@ function initSherpaOnnxOfflineFunAsrNanoModelConfig(config, Module) {
1;
const userPromptLen =
Module.lengthBytesUTF8(config.userPrompt || '语音转写：') + 1;
const languageLen = Module.lengthBytesUTF8(config.language || '') + 1;
const hotwordsLen = Module.lengthBytesUTF8(config.hotwords || '') + 1;

const n = encoderAdaptorLen + llmLen + embeddingLen + tokenizerLen +
systemPromptLen + userPromptLen;
systemPromptLen + userPromptLen + languageLen + hotwordsLen;

const buffer = Module._malloc(n);

const len = 10 * 4; // 6 pointers + 2 int + 2 float
const len = 13 * 4; // 8 pointers + 3 int + 2 float
const ptr = Module._malloc(len);

let offset = 0;
Expand All @@ -881,6 +883,12 @@ function initSherpaOnnxOfflineFunAsrNanoModelConfig(config, Module) {
config.userPrompt || '语音转写：', buffer + offset, userPromptLen);
offset += userPromptLen;

Module.stringToUTF8(config.language || '', buffer + offset, languageLen);
offset += languageLen;

Module.stringToUTF8(config.hotwords || '', buffer + offset, hotwordsLen);
offset += hotwordsLen;

offset = 0;
Module.setValue(ptr + 0 * 4, buffer + offset, 'i8*');
offset += encoderAdaptorLen;
Expand All @@ -904,6 +912,11 @@ function initSherpaOnnxOfflineFunAsrNanoModelConfig(config, Module) {
Module.setValue(ptr + 7 * 4, config.temperature || 1e-6, 'float');
Module.setValue(ptr + 8 * 4, config.topP || 0.8, 'float');
Module.setValue(ptr + 9 * 4, config.seed || 42, 'i32');
Module.setValue(ptr + 10 * 4, buffer + offset, 'i8*');
offset += languageLen;
Module.setValue(ptr + 11 * 4, config.itn || 0, 'i32');
Module.setValue(ptr + 12 * 4, buffer + offset, 'i8*');
offset += hotwordsLen;

return {
buffer: buffer,
Expand All @@ -921,7 +934,7 @@ function initSherpaOnnxOfflineWhisperModelConfig(config, Module) {
const n = encoderLen + decoderLen + languageLen + taskLen;
const buffer = Module._malloc(n);

const len = 5 * 4; // 4 pointers + 1 int32
const len = 7 * 4; // 4 pointers + 3 int32
const ptr = Module._malloc(len);

let offset = 0;
Expand Down Expand Up @@ -950,6 +963,8 @@ function initSherpaOnnxOfflineWhisperModelConfig(config, Module) {
offset += taskLen;

Module.setValue(ptr + 16, config.tailPaddings || 2000, 'i32');
Module.setValue(ptr + 20, config.enableTokenTimestamps || 0, 'i32');
Module.setValue(ptr + 24, config.enableSegmentTimestamps || 0, 'i32');

return {
buffer: buffer,
Expand Down Expand Up @@ -1220,6 +1235,9 @@ function initSherpaOnnxOfflineModelConfig(config, Module) {
temperature: 1e-6,
topP: 0.8,
seed: 42,
language: '',
itn: 0,
hotwords: '',
};
}

Expand All @@ -1230,6 +1248,8 @@ function initSherpaOnnxOfflineModelConfig(config, Module) {
language: '',
task: '',
tailPaddings: -1,
enableTokenTimestamps: 0,
enableSegmentTimestamps: 0,
};
}

Expand Down
2 changes: 1 addition & 1 deletion wasm/kws/CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ set(MY_FLAGS "-s FORCE_FILESYSTEM=1 -s INITIAL_MEMORY=512MB -s ALLOW_MEMORY_GROW
string(APPEND MY_FLAGS " -sSTACK_SIZE=10485760 ")
string(APPEND MY_FLAGS " -sEXPORTED_FUNCTIONS=[_CopyHeap,_malloc,_free,${all_exported_functions}] ")
string(APPEND MY_FLAGS "--preload-file ${CMAKE_CURRENT_SOURCE_DIR}/assets@. ")
string(APPEND MY_FLAGS " -sEXPORTED_RUNTIME_METHODS=['ccall','stringToUTF8','setValue','getValue','lengthBytesUTF8','UTF8ToString'] ")
string(APPEND MY_FLAGS " -sEXPORTED_RUNTIME_METHODS=['ccall','stringToUTF8','setValue','getValue','lengthBytesUTF8','UTF8ToString','HEAPU8','HEAP16','HEAP32','HEAPF32','HEAPF64'] ")

message(STATUS "MY_FLAGS: ${MY_FLAGS}")

Expand Down
5 changes: 3 additions & 2 deletions wasm/nodejs/CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ set(exported_functions
SherpaOnnxDestroyOfflineTtsGeneratedAudio
SherpaOnnxOfflineTtsGenerate
SherpaOnnxOfflineTtsGenerateWithCallback
SherpaOnnxOfflineTtsGenerateWithConfig
SherpaOnnxOfflineTtsNumSpeakers
SherpaOnnxOfflineTtsSampleRate
SherpaOnnxWriteWave
Expand Down Expand Up @@ -112,9 +113,9 @@ include_directories(${CMAKE_SOURCE_DIR})
set(MY_FLAGS " -s FORCE_FILESYSTEM=1 -s INITIAL_MEMORY=512MB -s ALLOW_MEMORY_GROWTH=1")
string(APPEND MY_FLAGS " -sSTACK_SIZE=10485760 ") # 10MB
string(APPEND MY_FLAGS " -sEXPORTED_FUNCTIONS=[_CopyHeap,_malloc,_free,${all_exported_functions}] ")
string(APPEND MY_FLAGS " -sEXPORTED_RUNTIME_METHODS=['ccall','stringToUTF8','setValue','getValue'] ")
string(APPEND MY_FLAGS " -sNODERAWFS=1 ")
string(APPEND MY_FLAGS " -sEXPORTED_RUNTIME_METHODS=['ccall','stringToUTF8','setValue','getValue','lengthBytesUTF8','UTF8ToString'] ")
string(APPEND MY_FLAGS " -sEXPORTED_RUNTIME_METHODS=['ccall','stringToUTF8','setValue','getValue','lengthBytesUTF8','UTF8ToString','HEAPU8','HEAP16','HEAP32','HEAPF32','HEAPF64'] ")

string(APPEND MY_FLAGS " -sMODULARIZE=1 -sWASM_ASYNC_COMPILATION=0 ")

message(STATUS "MY_FLAGS: ${MY_FLAGS}")
Expand Down
4 changes: 2 additions & 2 deletions wasm/nodejs/sherpa-onnx-wasm-nodejs.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,10 +17,10 @@ static_assert(sizeof(SherpaOnnxOfflineZipformerCtcModelConfig) == 4, "");
static_assert(sizeof(SherpaOnnxOfflineWenetCtcModelConfig) == 4, "");
static_assert(sizeof(SherpaOnnxOfflineOmnilingualAsrCtcModelConfig) == 4, "");
static_assert(sizeof(SherpaOnnxOfflineMedAsrCtcModelConfig) == 4, "");
static_assert(sizeof(SherpaOnnxOfflineFunASRNanoModelConfig) == 10 * 4, "");
static_assert(sizeof(SherpaOnnxOfflineFunASRNanoModelConfig) == 13 * 4, "");
static_assert(sizeof(SherpaOnnxOfflineDolphinModelConfig) == 4, "");
static_assert(sizeof(SherpaOnnxOfflineNemoEncDecCtcModelConfig) == 4, "");
static_assert(sizeof(SherpaOnnxOfflineWhisperModelConfig) == 5 * 4, "");
static_assert(sizeof(SherpaOnnxOfflineWhisperModelConfig) == 7 * 4, "");
static_assert(sizeof(SherpaOnnxOfflineFireRedAsrModelConfig) == 2 * 4, "");
static_assert(sizeof(SherpaOnnxOfflineMoonshineModelConfig) == 4 * 4, "");
static_assert(sizeof(SherpaOnnxOfflineTdnnModelConfig) == 4, "");
Expand Down
2 changes: 1 addition & 1 deletion wasm/speaker-diarization/CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ set(MY_FLAGS " -s FORCE_FILESYSTEM=1 -s INITIAL_MEMORY=512MB -s ALLOW_MEMORY_GRO
string(APPEND MY_FLAGS " -sSTACK_SIZE=10485760 ") # 10MB
string(APPEND MY_FLAGS " -sEXPORTED_FUNCTIONS=[_CopyHeap,_malloc,_free,${all_exported_functions}] ")
string(APPEND MY_FLAGS "--preload-file ${CMAKE_CURRENT_SOURCE_DIR}/assets@. ")
string(APPEND MY_FLAGS " -sEXPORTED_RUNTIME_METHODS=['ccall','stringToUTF8','setValue','getValue','lengthBytesUTF8','UTF8ToString'] ")
string(APPEND MY_FLAGS " -sEXPORTED_RUNTIME_METHODS=['ccall','stringToUTF8','setValue','getValue','lengthBytesUTF8','UTF8ToString','HEAPU8','HEAP16','HEAP32','HEAPF32','HEAPF64'] ")

message(STATUS "MY_FLAGS: ${MY_FLAGS}")

Expand Down
2 changes: 1 addition & 1 deletion wasm/speech-enhancement/CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ set(MY_FLAGS " -s FORCE_FILESYSTEM=1 -s INITIAL_MEMORY=128MB -s ALLOW_MEMORY_GRO
string(APPEND MY_FLAGS " -sSTACK_SIZE=10485760 ") # 10MB
string(APPEND MY_FLAGS " -sEXPORTED_FUNCTIONS=[_CopyHeap,_malloc,_free,${all_exported_functions}] ")
string(APPEND MY_FLAGS "--preload-file ${CMAKE_CURRENT_SOURCE_DIR}/assets@. ")
string(APPEND MY_FLAGS " -sEXPORTED_RUNTIME_METHODS=['ccall','stringToUTF8','setValue','getValue','lengthBytesUTF8','UTF8ToString'] ")
string(APPEND MY_FLAGS " -sEXPORTED_RUNTIME_METHODS=['ccall','stringToUTF8','setValue','getValue','lengthBytesUTF8','UTF8ToString','HEAPU8','HEAP16','HEAP32','HEAPF32','HEAPF64'] ")

message(STATUS "MY_FLAGS: ${MY_FLAGS}")

Expand Down
3 changes: 2 additions & 1 deletion wasm/tts/CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ set(exported_functions
SherpaOnnxDestroyOfflineTtsGeneratedAudio
SherpaOnnxOfflineTtsGenerate
SherpaOnnxOfflineTtsGenerateWithCallback
SherpaOnnxOfflineTtsGenerateWithConfig
SherpaOnnxOfflineTtsNumSpeakers
SherpaOnnxOfflineTtsSampleRate
SherpaOnnxWriteWave
Expand All @@ -29,7 +30,7 @@ set(MY_FLAGS " -s FORCE_FILESYSTEM=1 -s INITIAL_MEMORY=512MB -s ALLOW_MEMORY_GRO
string(APPEND MY_FLAGS " -sSTACK_SIZE=10485760 ") # 10MB
string(APPEND MY_FLAGS " -sEXPORTED_FUNCTIONS=[_CopyHeap,_malloc,_free,${all_exported_functions}] ")
string(APPEND MY_FLAGS "--preload-file ${CMAKE_CURRENT_SOURCE_DIR}/assets@. ")
string(APPEND MY_FLAGS " -sEXPORTED_RUNTIME_METHODS=['ccall','stringToUTF8','setValue','getValue','lengthBytesUTF8','UTF8ToString'] ")
string(APPEND MY_FLAGS " -sEXPORTED_RUNTIME_METHODS=['ccall','stringToUTF8','setValue','getValue','lengthBytesUTF8','UTF8ToString','HEAPU8','HEAP16','HEAP32','HEAPF32','HEAPF64'] ")

message(STATUS "MY_FLAGS: ${MY_FLAGS}")

Expand Down
Loading