Add Kotlin and Java API for ZipVoice models - #3344
Merged
Merged
Conversation
dosubot
Bot
added
the
size:L
4 tasks
📝 WalkthroughWalkthroughThis PR adds ZipVoice TTS model support across Java/Kotlin language bindings, including new configuration classes, native JNI parsing, example applications, and test workflows. Changes
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Possibly related PRs
Suggested labels
Poem
🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
📝 Coding Plan
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request significantly expands the capabilities of the Highlights
Ignored Files
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
There was a problem hiding this comment.
Pull request overview
Adds ZipVoice model support to the Kotlin and Java TTS APIs, wiring the new model config through JNI and providing runnable examples + CI coverage.
Changes:
- Add
OfflineTtsZipVoiceModelConfigto Kotlin API and Java API, and expose it viaOfflineTtsModelConfig. - Extend JNI config parsing (
offline-tts.cc) to read ZipVoice fields into nativeOfflineTtsConfig. - Add ZipVoice TTS example scripts for Kotlin/Java and run them in GitHub Actions workflows.
Reviewed changes
Copilot reviewed 12 out of 12 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
sherpa-onnx/kotlin-api/Tts.kt |
Adds Kotlin ZipVoice model config and exposes it on OfflineTtsModelConfig. |
sherpa-onnx/jni/offline-tts.cc |
Reads ZipVoice config fields from JVM objects into native config. |
sherpa-onnx/java-api/src/main/java/com/k2fsa/sherpa/onnx/OfflineTtsZipVoiceModelConfig.java |
Introduces Java builder-style ZipVoice config. |
sherpa-onnx/java-api/src/main/java/com/k2fsa/sherpa/onnx/OfflineTtsModelConfig.java |
Adds ZipVoice config field + getter/setter on the Java model config. |
sherpa-onnx/java-api/Makefile |
Ensures the new ZipVoice config is compiled into sherpa-onnx.jar. |
kotlin-api-examples/test_zipvoice_tts.kt |
New Kotlin ZipVoice zero-shot TTS example. |
kotlin-api-examples/run.sh |
Builds/runs the Kotlin ZipVoice example in the JNI example suite. |
java-api-examples/run-zipvoice-tts.sh |
New Java ZipVoice runnable script (downloads model + vocoder). |
java-api-examples/ZipVoiceTts.java |
New Java ZipVoice example program using callback + RTF reporting. |
java-api-examples/README.md |
Documents the new ZipVoice example script. |
.github/workflows/run-java-test.yaml |
Runs the ZipVoice Java TTS example in CI and cleans up artifacts. |
.github/workflows/jni.yaml |
Triggers JNI workflow when sherpa-onnx/kotlin-api/** changes. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+93
to
+96
| public Builder setZipvoice(OfflineTtsZipVoiceModelConfig zipvoice) { | ||
| this.zipvoice = zipvoice; | ||
| return this; | ||
| } |
Comment on lines
+207
to
+208
| "Lcom/k2fsa/sherpa/onnx/OfflineTtsZipVoiceModelConfig;"); | ||
| jobject zipvoice = env->GetObjectField(model, fid); |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (2)
kotlin-api-examples/test_zipvoice_tts.kt (1)
27-41: EnsureOfflineTtsis always released on exceptions.
release()currently runs only on the success path. Wrap generation/save intry/finallyso the native pointer is always freed.♻️ Proposed fix
- val tts = OfflineTts(config = config) - val text = "小米的价值观是真诚, 热爱. 真诚，就是不欺人也不自欺. 热爱, 就是全心投入并享受其中." - val referenceText = "那还是三十六年前, 一九八七年. 我呢考上了武汉大学的计算机系." - val genConfig = GenerationConfig( - referenceAudio = wave.samples, - referenceSampleRate = wave.sampleRate, - referenceText = referenceText, - numSteps = 4, - extra = mapOf("min_char_in_sentence" to "10"), - ) - - val audio = tts.generateWithConfigAndCallback(text = text, config = genConfig, callback = ::callback) - audio.save(filename = "test-zipvoice-zh-en.wav") - tts.release() + val tts = OfflineTts(config = config) + try { + val text = "小米的价值观是真诚, 热爱. 真诚，就是不欺人也不自欺. 热爱, 就是全心投入并享受其中." + val referenceText = "那还是三十六年前, 一九八七年. 我呢考上了武汉大学的计算机系." + val genConfig = GenerationConfig( + referenceAudio = wave.samples, + referenceSampleRate = wave.sampleRate, + referenceText = referenceText, + numSteps = 4, + extra = mapOf("min_char_in_sentence" to "10"), + ) + + val audio = tts.generateWithConfigAndCallback( + text = text, + config = genConfig, + callback = ::callback + ) + audio.save(filename = "test-zipvoice-zh-en.wav") + } finally { + tts.release() + } println("Saved to test-zipvoice-zh-en.wav")🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@kotlin-api-examples/test_zipvoice_tts.kt` around lines 27 - 41, The OfflineTts instance (tts) is only released on the success path; wrap the generation and save calls in a try/finally so tts.release() always runs on error or success. Specifically, enclose the call to tts.generateWithConfigAndCallback(...) and audio.save(...) inside a try block and call tts.release() in the finally block (still referencing OfflineTts, generateWithConfigAndCallback, audio.save, and tts.release) to ensure the native pointer is freed even if an exception is thrown..github/workflows/run-java-test.yaml (1)
354-363: Make cleanup idempotent for standalone files.Use
rm -ffor single-file cleanup so reruns or partial states don’t fail the step.♻️ Proposed fix
- rm vocos-22khz-univ.onnx - rm vocos_24khz.onnx + rm -f vocos-22khz-univ.onnx + rm -f vocos_24khz.onnx🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In @.github/workflows/run-java-test.yaml around lines 354 - 363, The cleanup step currently uses plain rm for standalone files (vocos-22khz-univ.onnx and vocos_24khz.onnx) which can fail if the files are already absent; change those two lines to use rm -f to make the cleanup idempotent (leave the existing wildcard rm -rf lines alone for directories/patterns).
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@sherpa-onnx/jni/offline-tts.cc`:
- Around line 205-240: The code calls env->GetObjectClass(zipvoice) and
subsequent SHERPA_ONNX_JNI_READ_* macros without checking whether jobject
zipvoice is null, which can cause JNI failures; update the logic around
fid/zipvoice to check if zipvoice == nullptr after env->GetObjectField(model,
fid) and only call env->GetObjectClass(zipvoice) and execute the
SHERPA_ONNX_JNI_READ_STRING / SHERPA_ONNX_JNI_READ_FLOAT reads when zipvoice is
non-null, otherwise skip those reads and either leave ans.model.zipvoice fields
at safe defaults or run the existing validation/error path so null configs are
handled cleanly.
---
Nitpick comments:
In @.github/workflows/run-java-test.yaml:
- Around line 354-363: The cleanup step currently uses plain rm for standalone
files (vocos-22khz-univ.onnx and vocos_24khz.onnx) which can fail if the files
are already absent; change those two lines to use rm -f to make the cleanup
idempotent (leave the existing wildcard rm -rf lines alone for
directories/patterns).
In `@kotlin-api-examples/test_zipvoice_tts.kt`:
- Around line 27-41: The OfflineTts instance (tts) is only released on the
success path; wrap the generation and save calls in a try/finally so
tts.release() always runs on error or success. Specifically, enclose the call to
tts.generateWithConfigAndCallback(...) and audio.save(...) inside a try block
and call tts.release() in the finally block (still referencing OfflineTts,
generateWithConfigAndCallback, audio.save, and tts.release) to ensure the native
pointer is freed even if an exception is thrown.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: f4294e07-0759-4a60-b58a-334d3add3aee
📒 Files selected for processing (12)
.github/workflows/jni.yaml.github/workflows/run-java-test.yamljava-api-examples/README.mdjava-api-examples/ZipVoiceTts.javajava-api-examples/run-zipvoice-tts.shkotlin-api-examples/run.shkotlin-api-examples/test_zipvoice_tts.ktsherpa-onnx/java-api/Makefilesherpa-onnx/java-api/src/main/java/com/k2fsa/sherpa/onnx/OfflineTtsModelConfig.javasherpa-onnx/java-api/src/main/java/com/k2fsa/sherpa/onnx/OfflineTtsZipVoiceModelConfig.javasherpa-onnx/jni/offline-tts.ccsherpa-onnx/kotlin-api/Tts.kt
Comment on lines
+205
to
+240
| fid = env->GetFieldID( | ||
| model_config_cls, "zipvoice", | ||
| "Lcom/k2fsa/sherpa/onnx/OfflineTtsZipVoiceModelConfig;"); | ||
| jobject zipvoice = env->GetObjectField(model, fid); | ||
| jclass zipvoice_cls = env->GetObjectClass(zipvoice); | ||
|
|
||
| SHERPA_ONNX_JNI_READ_STRING(ans.model.zipvoice.tokens, tokens, zipvoice_cls, | ||
| zipvoice); | ||
|
|
||
| SHERPA_ONNX_JNI_READ_STRING(ans.model.zipvoice.encoder, encoder, zipvoice_cls, | ||
| zipvoice); | ||
|
|
||
| SHERPA_ONNX_JNI_READ_STRING(ans.model.zipvoice.decoder, decoder, zipvoice_cls, | ||
| zipvoice); | ||
|
|
||
| SHERPA_ONNX_JNI_READ_STRING(ans.model.zipvoice.vocoder, vocoder, zipvoice_cls, | ||
| zipvoice); | ||
|
|
||
| SHERPA_ONNX_JNI_READ_STRING(ans.model.zipvoice.data_dir, dataDir, zipvoice_cls, | ||
| zipvoice); | ||
|
|
||
| SHERPA_ONNX_JNI_READ_STRING(ans.model.zipvoice.lexicon, lexicon, zipvoice_cls, | ||
| zipvoice); | ||
|
|
||
| SHERPA_ONNX_JNI_READ_FLOAT(ans.model.zipvoice.feat_scale, featScale, | ||
| zipvoice_cls, zipvoice); | ||
|
|
||
| SHERPA_ONNX_JNI_READ_FLOAT(ans.model.zipvoice.t_shift, tShift, zipvoice_cls, | ||
| zipvoice); | ||
|
|
||
| SHERPA_ONNX_JNI_READ_FLOAT(ans.model.zipvoice.target_rms, targetRms, | ||
| zipvoice_cls, zipvoice); | ||
|
|
||
| SHERPA_ONNX_JNI_READ_FLOAT(ans.model.zipvoice.guidance_scale, guidanceScale, | ||
| zipvoice_cls, zipvoice); | ||
|
|
There was a problem hiding this comment.
Guard zipvoice object before calling GetObjectClass.
At Line 209, zipvoice is used without a null check. A null zipvoice config can trigger JNI failure before config validation, instead of cleanly falling back to defaults/error handling.
🛠️ Proposed fix
fid = env->GetFieldID(
model_config_cls, "zipvoice",
"Lcom/k2fsa/sherpa/onnx/OfflineTtsZipVoiceModelConfig;");
jobject zipvoice = env->GetObjectField(model, fid);
- jclass zipvoice_cls = env->GetObjectClass(zipvoice);
-
- SHERPA_ONNX_JNI_READ_STRING(ans.model.zipvoice.tokens, tokens, zipvoice_cls,
- zipvoice);
-
- SHERPA_ONNX_JNI_READ_STRING(ans.model.zipvoice.encoder, encoder, zipvoice_cls,
- zipvoice);
-
- SHERPA_ONNX_JNI_READ_STRING(ans.model.zipvoice.decoder, decoder, zipvoice_cls,
- zipvoice);
-
- SHERPA_ONNX_JNI_READ_STRING(ans.model.zipvoice.vocoder, vocoder, zipvoice_cls,
- zipvoice);
-
- SHERPA_ONNX_JNI_READ_STRING(ans.model.zipvoice.data_dir, dataDir, zipvoice_cls,
- zipvoice);
-
- SHERPA_ONNX_JNI_READ_STRING(ans.model.zipvoice.lexicon, lexicon, zipvoice_cls,
- zipvoice);
-
- SHERPA_ONNX_JNI_READ_FLOAT(ans.model.zipvoice.feat_scale, featScale,
- zipvoice_cls, zipvoice);
-
- SHERPA_ONNX_JNI_READ_FLOAT(ans.model.zipvoice.t_shift, tShift, zipvoice_cls,
- zipvoice);
-
- SHERPA_ONNX_JNI_READ_FLOAT(ans.model.zipvoice.target_rms, targetRms,
- zipvoice_cls, zipvoice);
-
- SHERPA_ONNX_JNI_READ_FLOAT(ans.model.zipvoice.guidance_scale, guidanceScale,
- zipvoice_cls, zipvoice);
+ jclass zipvoice_cls = nullptr;
+ if (zipvoice != nullptr) {
+ zipvoice_cls = env->GetObjectClass(zipvoice);
+
+ SHERPA_ONNX_JNI_READ_STRING(ans.model.zipvoice.tokens, tokens, zipvoice_cls,
+ zipvoice);
+ SHERPA_ONNX_JNI_READ_STRING(ans.model.zipvoice.encoder, encoder,
+ zipvoice_cls, zipvoice);
+ SHERPA_ONNX_JNI_READ_STRING(ans.model.zipvoice.decoder, decoder,
+ zipvoice_cls, zipvoice);
+ SHERPA_ONNX_JNI_READ_STRING(ans.model.zipvoice.vocoder, vocoder,
+ zipvoice_cls, zipvoice);
+ SHERPA_ONNX_JNI_READ_STRING(ans.model.zipvoice.data_dir, dataDir,
+ zipvoice_cls, zipvoice);
+ SHERPA_ONNX_JNI_READ_STRING(ans.model.zipvoice.lexicon, lexicon,
+ zipvoice_cls, zipvoice);
+ SHERPA_ONNX_JNI_READ_FLOAT(ans.model.zipvoice.feat_scale, featScale,
+ zipvoice_cls, zipvoice);
+ SHERPA_ONNX_JNI_READ_FLOAT(ans.model.zipvoice.t_shift, tShift, zipvoice_cls,
+ zipvoice);
+ SHERPA_ONNX_JNI_READ_FLOAT(ans.model.zipvoice.target_rms, targetRms,
+ zipvoice_cls, zipvoice);
+ SHERPA_ONNX_JNI_READ_FLOAT(ans.model.zipvoice.guidance_scale, guidanceScale,
+ zipvoice_cls, zipvoice);
+ }📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| fid = env->GetFieldID( | |
| model_config_cls, "zipvoice", | |
| "Lcom/k2fsa/sherpa/onnx/OfflineTtsZipVoiceModelConfig;"); | |
| jobject zipvoice = env->GetObjectField(model, fid); | |
| jclass zipvoice_cls = env->GetObjectClass(zipvoice); | |
| SHERPA_ONNX_JNI_READ_STRING(ans.model.zipvoice.tokens, tokens, zipvoice_cls, | |
| zipvoice); | |
| SHERPA_ONNX_JNI_READ_STRING(ans.model.zipvoice.encoder, encoder, zipvoice_cls, | |
| zipvoice); | |
| SHERPA_ONNX_JNI_READ_STRING(ans.model.zipvoice.decoder, decoder, zipvoice_cls, | |
| zipvoice); | |
| SHERPA_ONNX_JNI_READ_STRING(ans.model.zipvoice.vocoder, vocoder, zipvoice_cls, | |
| zipvoice); | |
| SHERPA_ONNX_JNI_READ_STRING(ans.model.zipvoice.data_dir, dataDir, zipvoice_cls, | |
| zipvoice); | |
| SHERPA_ONNX_JNI_READ_STRING(ans.model.zipvoice.lexicon, lexicon, zipvoice_cls, | |
| zipvoice); | |
| SHERPA_ONNX_JNI_READ_FLOAT(ans.model.zipvoice.feat_scale, featScale, | |
| zipvoice_cls, zipvoice); | |
| SHERPA_ONNX_JNI_READ_FLOAT(ans.model.zipvoice.t_shift, tShift, zipvoice_cls, | |
| zipvoice); | |
| SHERPA_ONNX_JNI_READ_FLOAT(ans.model.zipvoice.target_rms, targetRms, | |
| zipvoice_cls, zipvoice); | |
| SHERPA_ONNX_JNI_READ_FLOAT(ans.model.zipvoice.guidance_scale, guidanceScale, | |
| zipvoice_cls, zipvoice); | |
| fid = env->GetFieldID( | |
| model_config_cls, "zipvoice", | |
| "Lcom/k2fsa/sherpa/onnx/OfflineTtsZipVoiceModelConfig;"); | |
| jobject zipvoice = env->GetObjectField(model, fid); | |
| jclass zipvoice_cls = nullptr; | |
| if (zipvoice != nullptr) { | |
| zipvoice_cls = env->GetObjectClass(zipvoice); | |
| SHERPA_ONNX_JNI_READ_STRING(ans.model.zipvoice.tokens, tokens, zipvoice_cls, | |
| zipvoice); | |
| SHERPA_ONNX_JNI_READ_STRING(ans.model.zipvoice.encoder, encoder, | |
| zipvoice_cls, zipvoice); | |
| SHERPA_ONNX_JNI_READ_STRING(ans.model.zipvoice.decoder, decoder, | |
| zipvoice_cls, zipvoice); | |
| SHERPA_ONNX_JNI_READ_STRING(ans.model.zipvoice.vocoder, vocoder, | |
| zipvoice_cls, zipvoice); | |
| SHERPA_ONNX_JNI_READ_STRING(ans.model.zipvoice.data_dir, dataDir, | |
| zipvoice_cls, zipvoice); | |
| SHERPA_ONNX_JNI_READ_STRING(ans.model.zipvoice.lexicon, lexicon, | |
| zipvoice_cls, zipvoice); | |
| SHERPA_ONNX_JNI_READ_FLOAT(ans.model.zipvoice.feat_scale, featScale, | |
| zipvoice_cls, zipvoice); | |
| SHERPA_ONNX_JNI_READ_FLOAT(ans.model.zipvoice.t_shift, tShift, zipvoice_cls, | |
| zipvoice); | |
| SHERPA_ONNX_JNI_READ_FLOAT(ans.model.zipvoice.target_rms, targetRms, | |
| zipvoice_cls, zipvoice); | |
| SHERPA_ONNX_JNI_READ_FLOAT(ans.model.zipvoice.guidance_scale, guidanceScale, | |
| zipvoice_cls, zipvoice); | |
| } |
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@sherpa-onnx/jni/offline-tts.cc` around lines 205 - 240, The code calls
env->GetObjectClass(zipvoice) and subsequent SHERPA_ONNX_JNI_READ_* macros
without checking whether jobject zipvoice is null, which can cause JNI failures;
update the logic around fid/zipvoice to check if zipvoice == nullptr after
env->GetObjectField(model, fid) and only call env->GetObjectClass(zipvoice) and
execute the SHERPA_ONNX_JNI_READ_STRING / SHERPA_ONNX_JNI_READ_FLOAT reads when
zipvoice is non-null, otherwise skip those reads and either leave
ans.model.zipvoice fields at safe defaults or run the existing validation/error
path so null configs are handled cleanly.
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces support for ZipVoice TTS models in both the Java and Kotlin APIs. It includes new example applications, corresponding run scripts, and updates to the configuration classes and JNI bindings. The changes are well-integrated and follow the existing project structure. I have one suggestion to improve consistency across the new shell scripts.
| # to download more models | ||
| if [ ! -f ./sherpa-onnx-zipvoice-distill-int8-zh-en-emilia/encoder.int8.onnx ]; then | ||
| curl -SL -O https://github.com/k2-fsa/sherpa-onnx/releases/download/tts-models/sherpa-onnx-zipvoice-distill-int8-zh-en-emilia.tar.bz2 | ||
| tar xf sherpa-onnx-zipvoice-distill-int8-zh-en-emilia.tar.bz2 |
There was a problem hiding this comment.
For consistency with other scripts in the repository, such as kotlin-api-examples/run.sh, it is recommended to use tar xvf instead of tar xf. The v flag provides verbose output, which can be helpful for debugging during file extraction.
Suggested change
| tar xf sherpa-onnx-zipvoice-distill-int8-zh-en-emilia.tar.bz2 | |
| tar xvf sherpa-onnx-zipvoice-distill-int8-zh-en-emilia.tar.bz2 |
This was referenced Mar 31, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary by CodeRabbit
Release Notes
New Features
Chores