Repository navigation
Add APIs for Online NeMo CTC models - #2454
Conversation
|
Note Other AI code review bot(s) detectedCodeRabbit has detected other AI code review bot(s) in this pull request and will avoid duplicating their findings in the review comments. This may lead to a less comprehensive review. WalkthroughThis change introduces support for the Nemo CTC model configuration in the online recognizer C API and its associated bindings. New structs and fields for Nemo CTC are added across C, C++, C#, Swift, JavaScript (WASM), and Harmony OS layers, enabling initialization, configuration, and memory management of this model type without altering existing logic. Changes
Sequence Diagram(s)sequenceDiagram
participant User
participant API_Binding (C#/Swift/JS/ETS)
participant C_API
participant Internal_Recognizer
User->>API_Binding: Provide Nemo CTC model config
API_Binding->>C_API: Initialize OnlineModelConfig with nemo_ctc
C_API->>Internal_Recognizer: Pass nemo_ctc.model in config
Internal_Recognizer-->>C_API: Recognizer created with Nemo CTC config
C_API-->>API_Binding: Return recognizer handle
API_Binding-->>User: Recognizer ready (Nemo CTC supported)
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~15 minutes Assessment against linked issues
Assessment against linked issues: Out-of-scope changesNo out-of-scope changes detected. Poem
Note ⚡️ Unit Test Generation is now available in beta!Learn more here, or try it out under "Finishing Touches" below. 📜 Recent review detailsConfiguration used: CodeRabbit UI 📒 Files selected for processing (2)
✅ Files skipped from review due to trivial changes (1)
🔇 Additional comments (2)
✨ Finishing Touches
🧪 Generate unit tests
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
|
One more thing
|
|
@csukuangfj Also currently im using the xcframework file in releases for it. Any chance I can get that, otherwise Ill have to change my code a lot for testing. Or we can write some sample swift program to test it independently |
|
You need to build it from source or wait for our next release. If you can release a streaming wenet ctc model.with.reasonable accuracy, we can support it. Otherwise, you need to modify the code by yourself. |
|
That makes sense. Ill try to build for the source then. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (1)
sherpa-onnx/pascal-api/sherpa_onnx.pas (1)
1342-1349: Fix casing inconsistency in string formatting.Line 1343 uses lowercase 'nemoCtc' in the format string while line 1348 uses uppercase 'NemoCtc' for the field reference. For consistency and readability, use the same casing.
Apply this diff to fix the casing:
- 'nemoCtc := %s'), + 'NemoCtc := %s'),
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (11)
dotnet-examples/offline-decode-files/run-zipformer-ctc.sh(1 hunks)flutter/sherpa_onnx/lib/src/keyword_spotter.dart(2 hunks)flutter/sherpa_onnx/lib/src/online_recognizer.dart(6 hunks)flutter/sherpa_onnx/lib/src/sherpa_onnx_bindings.dart(2 hunks)scripts/go/sherpa_onnx.go(4 hunks)sherpa-onnx/c-api/c-api.cc(7 hunks)sherpa-onnx/c-api/cxx-api.cc(2 hunks)sherpa-onnx/jni/common.h(1 hunks)sherpa-onnx/jni/keyword-spotter.cc(2 hunks)sherpa-onnx/jni/online-recognizer.cc(2 hunks)sherpa-onnx/pascal-api/sherpa_onnx.pas(7 hunks)
✅ Files skipped from review due to trivial changes (4)
- sherpa-onnx/jni/common.h
- dotnet-examples/offline-decode-files/run-zipformer-ctc.sh
- flutter/sherpa_onnx/lib/src/sherpa_onnx_bindings.dart
- flutter/sherpa_onnx/lib/src/keyword_spotter.dart
🚧 Files skipped from review as they are similar to previous changes (2)
- sherpa-onnx/c-api/c-api.cc
- sherpa-onnx/c-api/cxx-api.cc
🧰 Additional context used
🧠 Learnings (2)
📚 Learning: the sherpa-onnx jni library files are stored in hugging face repository at https://huggingface.co/cs...
Learnt from: litongjava
PR: k2-fsa/sherpa-onnx#2440
File: sherpa-onnx/java-api/src/main/java/com/k2fsa/sherpa/onnx/core/Core.java:4-6
Timestamp: 2025-08-06T04:23:50.237Z
Learning: The sherpa-onnx JNI library files are stored in Hugging Face repository at https://huggingface.co/csukuangfj/sherpa-onnx-libs under versioned directories like jni/1.12.7/, and the actual Windows JNI library filename is "sherpa-onnx-jni.dll" as defined in Core.java constants.
Applied to files:
sherpa-onnx/jni/keyword-spotter.ccsherpa-onnx/jni/online-recognizer.cc
📚 Learning: in sherpa-onnx java api, the native library names in core.java (win_native_library_name = "sherpa-on...
Learnt from: litongjava
PR: k2-fsa/sherpa-onnx#2440
File: sherpa-onnx/java-api/src/main/java/com/k2fsa/sherpa/onnx/core/Core.java:4-6
Timestamp: 2025-08-06T04:18:47.981Z
Learning: In sherpa-onnx Java API, the native library names in Core.java (WIN_NATIVE_LIBRARY_NAME = "sherpa-onnx-jni.dll", UNIX_NATIVE_LIBRARY_NAME = "libsherpa-onnx-jni.so", MACOS_NATIVE_LIBRARY_NAME = "libsherpa-onnx-jni.dylib") are copied directly from the compiled binary filenames and should not be changed to match other libraries' naming conventions.
Applied to files:
sherpa-onnx/jni/keyword-spotter.ccsherpa-onnx/jni/online-recognizer.cc
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (80)
- GitHub Check: ubuntu-latest
- GitHub Check: ubuntu-22.04 node v21
- GitHub Check: ubuntu-22.04 node v18
- GitHub Check: ubuntu-22.04 node v22
- GitHub Check: ubuntu-22.04 node v19
- GitHub Check: ubuntu-24.04-arm
- GitHub Check: windows-latest
- GitHub Check: ubuntu-22.04
- GitHub Check: ubuntu-latest
- GitHub Check: ubuntu-22.04 node v21
- GitHub Check: ubuntu-22.04 node v18
- GitHub Check: ubuntu-22.04 node v22
- GitHub Check: ubuntu-22.04 node v19
- GitHub Check: ubuntu-24.04-arm
- GitHub Check: windows-latest
- GitHub Check: ubuntu-22.04
- GitHub Check: ubuntu-latest
- GitHub Check: ubuntu-22.04 node v21
- GitHub Check: ubuntu-22.04 node v18
- GitHub Check: ubuntu-22.04 node v22
- GitHub Check: ubuntu-22.04 node v19
- GitHub Check: ubuntu-24.04-arm
- GitHub Check: windows-latest
- GitHub Check: ubuntu-22.04
- GitHub Check: ubuntu-latest
- GitHub Check: ubuntu-22.04 node v21
- GitHub Check: ubuntu-22.04 node v18
- GitHub Check: ubuntu-22.04 node v22
- GitHub Check: ubuntu-22.04 node v19
- GitHub Check: ubuntu-24.04-arm
- GitHub Check: windows-latest
- GitHub Check: ubuntu-22.04
- GitHub Check: ubuntu-latest
- GitHub Check: ubuntu-22.04 node v21
- GitHub Check: ubuntu-22.04 node v18
- GitHub Check: ubuntu-22.04 node v22
- GitHub Check: ubuntu-22.04 node v19
- GitHub Check: ubuntu-24.04-arm
- GitHub Check: windows-latest
- GitHub Check: ubuntu-22.04
- GitHub Check: ubuntu-latest
- GitHub Check: ubuntu-22.04 node v21
- GitHub Check: ubuntu-22.04 node v18
- GitHub Check: ubuntu-22.04 node v22
- GitHub Check: ubuntu-22.04 node v19
- GitHub Check: ubuntu-24.04-arm
- GitHub Check: windows-latest
- GitHub Check: ubuntu-22.04
- GitHub Check: ubuntu-latest
- GitHub Check: ubuntu-22.04 node v21
- GitHub Check: ubuntu-22.04 node v18
- GitHub Check: ubuntu-22.04 node v22
- GitHub Check: ubuntu-22.04 node v19
- GitHub Check: ubuntu-24.04-arm
- GitHub Check: windows-latest
- GitHub Check: ubuntu-22.04
- GitHub Check: ubuntu-latest
- GitHub Check: ubuntu-22.04 node v21
- GitHub Check: ubuntu-22.04 node v18
- GitHub Check: ubuntu-22.04 node v22
- GitHub Check: ubuntu-22.04 node v19
- GitHub Check: ubuntu-24.04-arm
- GitHub Check: windows-latest
- GitHub Check: ubuntu-22.04
- GitHub Check: ubuntu-latest
- GitHub Check: ubuntu-22.04 node v21
- GitHub Check: ubuntu-22.04 node v18
- GitHub Check: ubuntu-22.04 node v22
- GitHub Check: ubuntu-22.04 node v19
- GitHub Check: ubuntu-24.04-arm
- GitHub Check: windows-latest
- GitHub Check: ubuntu-22.04
- GitHub Check: ubuntu-latest
- GitHub Check: ubuntu-22.04 node v21
- GitHub Check: ubuntu-22.04 node v18
- GitHub Check: ubuntu-22.04 node v22
- GitHub Check: ubuntu-22.04 node v19
- GitHub Check: ubuntu-24.04-arm
- GitHub Check: windows-latest
- GitHub Check: ubuntu-22.04
🔇 Additional comments (20)
scripts/go/sherpa_onnx.go (4)
80-82: LGTM!The new
OnlineNemoCtcModelConfigstruct follows the same pattern as existing model configs likeOnlineZipformer2CtcModelConfig, maintaining consistency in the codebase.
94-94: LGTM!The
NemoCtcfield is correctly added to theOnlineModelConfigstruct, maintaining consistency with other model configuration fields.
205-206: LGTM!The NeMo CTC model configuration handling follows the established CGO pattern with proper memory management using
C.CStringanddefer C.free.
1825-1826: LGTM!The NeMo CTC model configuration handling in
NewKeywordSpottercorrectly follows the same pattern as inNewOnlineRecognizer, ensuring consistent API behavior.sherpa-onnx/jni/keyword-spotter.cc (2)
12-13: LGTM!The forward declaration for
GetOnlineModelConfigproperly enables code reuse by referencing the function implemented inonline-recognizer.cc, following good C++ practices.
63-63: LGTM!The refactoring successfully eliminates code duplication by replacing inline JNI model configuration extraction with a call to the centralized
GetOnlineModelConfigfunction, improving maintainability while preserving functionality.sherpa-onnx/jni/online-recognizer.cc (6)
13-40: LGTM!The
GetOnlineModelConfigfunction is well-implemented with proper JNI patterns. The transducer configuration extraction correctly usesGetStringUTFCharsandReleaseStringUTFCharsfor memory management.
41-57: LGTM!The paraformer configuration extraction maintains consistency with the established JNI pattern, properly handling string conversion and memory cleanup.
59-71: LGTM!The zipformer2 CTC configuration extraction correctly follows the established pattern with proper JNI field access and memory management.
73-83: LGTM!The NeMo CTC model configuration extraction is properly implemented, following the same reliable JNI pattern as other model configurations. The field naming follows Java conventions correctly.
85-122: LGTM!The remaining field extractions properly handle different JNI field types (strings, integers, booleans) with consistent memory management patterns. The provider field is correctly assigned to the nested
provider_config.providerstructure.
236-236: LGTM!The refactoring successfully centralizes model configuration extraction into the dedicated
GetOnlineModelConfigfunction, eliminating code duplication and improving maintainability while preserving all functionality.sherpa-onnx/pascal-api/sherpa_onnx.pas (4)
168-171: LGTM!The new
TSherpaOnnxOnlineNemoCtcModelConfigrecord definition follows the established pattern of other model configurations in the codebase, with consistent naming and structure.
186-186: LGTM!The
NemoCtcfield integration follows the same pattern as other model configuration fields in the record.
1325-1329: LGTM!The
ToStringmethod implementation forTSherpaOnnxOnlineNemoCtcModelConfigis consistent with other similar model configurations.
1447-1447: LGTM!The NemoCtc model initialization in the constructor correctly converts the Pascal string to PAnsiChar for the C API interface.
flutter/sherpa_onnx/lib/src/online_recognizer.dart (4)
89-108: LGTM!The
OnlineNemoCtcModelConfigclass follows the established pattern of other model configuration classes with proper JSON serialization/deserialization and string representation methods.
115-115: LGTM!The integration of the
nemoCtcfield intoOnlineModelConfigis comprehensive and follows the established patterns for other model configurations, including proper handling in constructor, JSON serialization/deserialization, string representation, and field declaration.Also applies to: 133-134, 147-147, 154-154, 167-167
363-364: LGTM!The native memory allocation for the
nemoCtc.modelfield correctly usestoNativeUtf8()and follows the established pattern for other model configurations.
409-409: LGTM!The memory cleanup for
nemoCtc.modelis properly implemented and placed in the correct location to prevent memory leaks.
There was a problem hiding this comment.
Actionable comments posted: 3
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (9)
dotnet-examples/offline-decode-files/Program.cs(1 hunks)dotnet-examples/offline-decode-files/run-paraformer-itn.sh(1 hunks)dotnet-examples/offline-decode-files/run-paraformer.sh(1 hunks)go-api-examples/non-streaming-canary-decode-files/go.mod(0 hunks)mfc-examples/NonStreamingSpeechRecognition/NonStreamingSpeechRecognitionDlg.cpp(1 hunks)sherpa-onnx/csrc/rknn/silero-vad-model-rknn.cc(2 hunks)sherpa-onnx/csrc/rknn/silero-vad-model-rknn.h(1 hunks)sherpa-onnx/pascal-api/sherpa_onnx.pas(7 hunks)sherpa-onnx/python/tests/test_offline_recognizer.py(1 hunks)
💤 Files with no reviewable changes (1)
- go-api-examples/non-streaming-canary-decode-files/go.mod
✅ Files skipped from review due to trivial changes (4)
- dotnet-examples/offline-decode-files/Program.cs
- dotnet-examples/offline-decode-files/run-paraformer.sh
- dotnet-examples/offline-decode-files/run-paraformer-itn.sh
- mfc-examples/NonStreamingSpeechRecognition/NonStreamingSpeechRecognitionDlg.cpp
🚧 Files skipped from review as they are similar to previous changes (1)
- sherpa-onnx/pascal-api/sherpa_onnx.pas
🧰 Additional context used
🧬 Code Graph Analysis (1)
sherpa-onnx/csrc/rknn/silero-vad-model-rknn.h (1)
sherpa-onnx/csrc/rknn/silero-vad-model-rknn.cc (6)
Compute(398-400)Compute(398-398)samples(88-158)samples(88-88)samples(178-247)samples(178-178)
🔇 Additional comments (3)
sherpa-onnx/csrc/rknn/silero-vad-model-rknn.h (1)
35-35: Inconsistent with PR objectives.This change adds a
Computemethod to the Silero VAD model, but the PR is supposed to be about adding APIs for Online NeMo CTC models. This appears to be an unrelated change that should either be in a separate PR or the PR description should be updated to reflect the broader scope.Likely an incorrect or invalid review comment.
sherpa-onnx/csrc/rknn/silero-vad-model-rknn.cc (2)
178-247: LGTM - Method relocation without logic changes.The
Runmethod has been relocated within theImplclass but the logic remains unchanged. The implementation correctly handles RKNN tensor setup, model execution, and result extraction.
398-400: LGTM - Clean delegation to internal implementation.The
Computemethod provides a clean public interface by delegating to the internalRunmethod. The implementation is straightforward and consistent with the class design pattern.
There was a problem hiding this comment.
Pull Request Overview
This PR adds support for Online NeMo CTC models across multiple platforms and APIs within the sherpa-onnx speech recognition framework. The changes introduce configuration structures and supporting code to handle NeMo CTC models alongside existing transducer, paraformer, and zipformer2 CTC model types.
Key changes include:
- Addition of
OnlineNemoCtcModelConfigstructures across all language bindings and platforms - Integration of NeMo CTC configuration into existing online model configuration workflows
- Updates to memory management and serialization routines to handle the new model type
Reviewed Changes
Copilot reviewed 29 out of 29 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| wasm/asr/sherpa-onnx-wasm-main-asr.cc | Adds static assertion and printing support for NeMo CTC configuration |
| wasm/asr/sherpa-onnx-asr.js | Implements JavaScript bindings and memory management for NeMo CTC models |
| swift-api-examples/SherpaOnnx.swift | Adds Swift API support for NeMo CTC model configuration |
| sherpa-onnx/jni/online-recognizer.cc | Refactors JNI code and adds NeMo CTC field extraction |
| sherpa-onnx/pascal-api/sherpa_onnx.pas | Implements Pascal API structures and methods for NeMo CTC |
| sherpa-onnx/c-api/c-api.h | Defines C API structures for NeMo CTC configuration |
| scripts/go/sherpa_onnx.go | Adds Go language bindings for NeMo CTC models |
| scripts/dotnet/OnlineNemoCtcModelConfig.cs | Creates .NET configuration class for NeMo CTC models |
| flutter/sherpa_onnx/lib/src/online_recognizer.dart | Implements Flutter/Dart support for NeMo CTC configuration |
| harmony-os/SherpaOnnxHar/sherpa_onnx/src/main/ets/components/StreamingAsr.ets | Adds HarmonyOS TypeScript bindings for NeMo CTC |
|
|
||
| if ('ctc' in config) { | ||
| freeConfig(config.ctc, Module) | ||
| if ('zipformer2Ctc' in config) { |
There was a problem hiding this comment.
The removed 'ctc' property check suggests this was renamed to 'zipformer2Ctc'. Ensure all references to the old 'ctc' property have been updated throughout the codebase to maintain consistency.
| Module.setValue(ptr + offset, config.tokensBufSize || 0, 'i32'); | ||
| offset += 4; | ||
|
|
||
| Module._CopyHeap(nemoCtc.ptr, nemoCtc.len, ptr + offset); | ||
| offset += nemoCtc.len; | ||
|
|
There was a problem hiding this comment.
[nitpick] The nemoCtc configuration is copied after all other fields including tokensBufSize. This placement differs from the pattern where model configs are grouped together earlier in the structure. Consider moving this copy operation to be with other model configurations for better code organization.
| Module.setValue(ptr + offset, config.tokensBufSize || 0, 'i32'); | |
| offset += 4; | |
| Module._CopyHeap(nemoCtc.ptr, nemoCtc.len, ptr + offset); | |
| offset += nemoCtc.len; | |
| Module._CopyHeap(nemoCtc.ptr, nemoCtc.len, ptr + offset); | |
| offset += nemoCtc.len; | |
| Module.setValue(ptr + offset, config.tokensBufSize || 0, 'i32'); | |
| offset += 4; |
Fixes #2452
CC @yagarwal1307
Summary by CodeRabbit
New Features
Bug Fixes
Documentation