Repository navigation
Add Android demo for Paraformer ASR with Qualcomm NPU. - #2932
Conversation
dosubot
Bot
added
the
size:L
Summary of ChangesHello @csukuangfj, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request significantly advances the Android demo by integrating Paraformer ASR models with Qualcomm NPU acceleration. The core changes involve extending the model configuration to support QNN-specific parameters for Paraformer, introducing robust asset management utilities for handling multi-file QNN models, and updating the Android application's initialization logic to prepare these models for NPU inference. Additionally, the build scripts have been adjusted to reflect the new model additions and naming conventions, ensuring a comprehensive update for NPU-powered ASR on Android. Highlights
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
📝 WalkthroughWalkthroughThis PR adds Qualcomm Neural Network (QNN) backend support for Paraformer ASR models across Android, Java, Kotlin, and C++ layers, including asset validation utilities, model configuration extensions, JNI config parsing updates, and new int8 model variants for APK generation scripts. Changes
Sequence DiagramsequenceDiagram
participant App as Android App
participant Asset as Asset Manager
participant Storage as Internal Storage
participant JNI as JNI Layer
participant CPP as C++ Runtime
rect rgb(230, 245, 230)
Note over App,Storage: Asset Validation & Setup Phase
App->>Asset: assetListExists(paths)
Asset-->>App: ✓ All assets present
App->>Asset: copyAssetListToInternalStorage(paths)
Asset->>Storage: Copy tokens, encoder, predictor
Asset->>Storage: Copy context_binary files
Storage-->>App: ✓ Assets copied
end
rect rgb(230, 240, 255)
Note over App,JNI: Configuration Parsing Phase
App->>JNI: getOfflineConfig() with QnnConfig
JNI->>CPP: Parse paraformer.qnnConfig
CPP->>CPP: Read backend_lib, context_binary, system_lib
CPP-->>JNI: ✓ Config parsed
JNI-->>App: ✓ Ready for inference
end
rect rgb(255, 245, 230)
Note over CPP: QNN Model Execution
App->>CPP: Create recognizer with QNN paraformer
CPP->>CPP: Initialize QNN backend with loaded configs
CPP-->>App: ✓ Ready for ASR
end
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Possibly related PRs
Poem
Pre-merge checks and finishing touches❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✨ Finishing touches
🧪 Generate unit tests (beta)
📜 Recent review detailsConfiguration used: defaults Review profile: CHILL Plan: Pro 📒 Files selected for processing (9)
🧰 Additional context used🧠 Learnings (2)📚 Learning: 2025-08-06T04:23:50.237ZApplied to files:
📚 Learning: 2025-08-06T04:18:47.981ZApplied to files:
🔇 Additional comments (19)
Warning Review ran into problems🔥 ProblemsGit: Failed to clone repository. Please run the Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request adds support for Paraformer ASR models with Qualcomm NPU acceleration to the Android demo. The changes include updating Kotlin and Java data classes to include QNN configuration for Paraformer models, adding JNI bindings to read the new configuration, and implementing logic in the Android demo to handle copying of multiple model files required by Paraformer QNN models. The code is well-structured, and I have one suggestion to refactor some duplicated code in the Android demo to improve maintainability.
| fun copyAssetListToInternalStorage( | ||
| paths: String, | ||
| context: Context | ||
| ): String { | ||
| if (paths.isBlank()) return paths | ||
|
|
||
| val pathList = paths.split(",") | ||
| .map { it.trim() } | ||
| .filter { it.isNotEmpty() } | ||
|
|
||
| val copiedPaths = pathList.map { path -> | ||
| copyAssetToInternalStorage(path, context) | ||
| } | ||
|
|
||
| return copiedPaths.joinToString(",") | ||
| } |
There was a problem hiding this comment.
The logic for splitting and cleaning the comma-separated path string is duplicated in both assetListExists and copyAssetListToInternalStorage. To improve maintainability and reduce redundancy, you could extract this logic into a private helper function that can be called from both places.
For example:
private fun splitPaths(paths: String): List<String> {
return paths.split(",")
.map { it.trim() }
.filter { it.isNotEmpty() }
}
There was a problem hiding this comment.
Pull request overview
This PR adds Android demo support for Paraformer ASR models running on Qualcomm NPU (Neural Processing Unit). The changes enable Paraformer models to utilize Qualcomm's QNN (Qualcomm Neural Network) SDK for hardware acceleration on compatible Android devices.
Key changes include:
- Added QNN configuration support to OfflineParaformerModelConfig across Kotlin, Java, and JNI layers
- Implemented three new Paraformer model configurations (indices 9023, 9024, 9025) with QNN backend
- Extended asset handling to support comma-separated model paths required by Paraformer's multi-component architecture (encoder, predictor, decoder)
Reviewed changes
Copilot reviewed 9 out of 9 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| sherpa-onnx/kotlin-api/OfflineRecognizer.kt | Added QnnConfig field to OfflineParaformerModelConfig; implemented three new Paraformer+QNN model configurations; updated model name for SenseVoice to int8 variant; reformatted long lines for readability |
| sherpa-onnx/jni/offline-recognizer.cc | Added JNI code to read QNN configuration for Paraformer models; reused qnn_config variables following existing pattern for other models |
| sherpa-onnx/java-api/src/main/java/com/k2fsa/sherpa/onnx/OfflineParaformerModelConfig.java | Added QnnConfig field with getter/setter to support QNN configuration in Java API |
| sherpa-onnx/csrc/qnn/offline-paraformer-model-qnn.cc | Updated error message to provide clearer instructions for AssetManager usage |
| scripts/apk/generate-vad-asr-apk-script.py | Updated SenseVoice model name to int8 variant; removed deletion of non-existent model.onnx file |
| scripts/apk/generate-qnn-vad-asr-apk-script.py | Added two new Paraformer model configurations for QNN-enabled APK generation |
| scripts/apk/generate-asr-2pass-apk-script.py | Updated SenseVoice model name references to int8 variant for consistency |
| android/SherpaOnnxSimulateStreamingAsr/.../Home.kt | Fixed comment to correctly reflect 0.4s offset (was incorrectly labeled as 0.25s) |
| android/SherpaOnnxSimulateStreamingAsr/.../SimulateStreamingAsr.kt | Added helper functions to handle comma-separated asset paths; extended QNN setup logic to support Paraformer models with multi-component structure |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Summary by CodeRabbit
New Features
Bug Fixes
Documentation
✏️ Tip: You can customize this high-level summary in your review settings.