Skip to content

Add Android demo for streaming zipformer transducer ASR with QNN #3654

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
csukuangfj merged 2 commits into from
Jun 3, 2026
Merged

Add Android demo for streaming zipformer transducer ASR with QNN #3654

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
829b224
Add Android demo for streaming zipformer transducer with QNN
csukuangfj Jun 2, 2026
9afb126
Fix errors
csukuangfj Jun 2, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
168 changes: 168 additions & 0 deletions .github/workflows/apk-qnn-asr.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,168 @@
name: apk-qnn-asr

on:
push:
branches:
- apk
- android-qnn-asr

workflow_dispatch:

concurrency:
group: apk-qnn-asr-${{ github.ref }}
cancel-in-progress: true

permissions:
contents: write
Comment on lines +15 to +16

@coderabbitai coderabbitai Bot Jun 2, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

FILE=".github/workflows/apk-qnn-asr.yaml"
echo "== File = $FILE =="
ls -l "$FILE"

echo
echo "== Permissions block =="
sed -n '1,120p' "$FILE" | nl -ba | sed -n '1,80p'

echo
echo "== Checkout config lines (approx 20-80) =="
rg -n "actions/checkout|persist-credentials|permissions:" "$FILE" || true

echo
echo "== GITHUB_TOKEN / GitHub write operations =="
rg -n "GITHUB_TOKEN|github\\.com|repos/|create-release|actions:|releases|tags|git push|octokit|gh " "$FILE" || true

echo
echo "== Git commands (git clone/push) =="
rg -n "git (clone|push|remote add|config)" "$FILE" || true

echo
echo "== Context around git push =="
rg -n "git push" "$FILE" || true
# show around each match
for m in $(rg -n "git push" "$FILE" | cut -d: -f1); do
  echo "---- around git push at line $m ----"
  start=$((m-15)); end=$((m+25))
  sed -n "${start},${end}p" "$FILE" | nl -ba | sed -n "$((m-start+1-2)),$((m-start+1+25+2))p" || true
done

Repository: k2-fsa/sherpa-onnx

Length of output: 281

🏁 Script executed:

#!/bin/bash
set -euo pipefail

FILE=".github/workflows/apk-qnn-asr.yaml"
echo "== File = $FILE =="
ls -l "$FILE"

echo
echo "== Permissions block =="
sed -n '1,120p' "$FILE" | awk '{printf "%6d %s\n", NR, $0}'

echo
echo "== Checkout config lines (approx 1-200) =="
rg -n "actions/checkout|persist-credentials|permissions:" "$FILE" || true

echo
echo "== GITHUB_TOKEN / GitHub write operations (1-250) =="
rg -n "GITHUB_TOKEN|github\.com|create-release|releases|tags|git push|gh |octokit|actions:" "$FILE" || true

echo
echo "== Git commands (clone/push/remote/config) (1-250) =="
rg -n "git (clone|push|remote add|config)" "$FILE" || true

echo
echo "== Context around each 'git push' match =="
push_lines=$(rg -n "git push" "$FILE" | cut -d: -f1 || true)
if [ -z "${push_lines}" ]; then
  echo "(no 'git push' found)"
else
  while read -r m; do
    [ -z "$m" ] && continue
    start=$((m-15)); end=$((m+25))
    echo "---- around git push at line $m ----"
    sed -n "${start},${end}p" "$FILE" | awk '{printf "%6d %s\n", NR+('$start'-1), $0}'
  done <<< "$push_lines"
fi

Repository: k2-fsa/sherpa-onnx

Length of output: 5590

🏁 Script executed:

#!/bin/bash
set -euo pipefail

FILE=".github/workflows/apk-qnn-asr.yaml"

echo "== File tail (120-end) =="
wc -l "$FILE"
tail -n 120 "$FILE" | awk '{printf "%6d %s\n", NR, $0}'

echo
echo "== Remaining checkout/permissions overrides =="
rg -n "permissions:|actions/checkout@|persist-credentials" "$FILE" || true

echo
echo "== Any explicit GitHub token/release creation in whole file =="
rg -n "GITHUB_TOKEN|create-release|releases|tags|gh |octokit|actions:" "$FILE" || true

Repository: k2-fsa/sherpa-onnx

Length of output: 4841

Drop GitHub write access and disable persisted checkout credentials.

.github/workflows/apk-qnn-asr.yaml sets workflow-level permissions: contents: write, but it only publishes to Hugging Face via HF_TOKEN (no steps push back to GitHub). Also, actions/checkout@v4 doesn’t set persist-credentials: false, so the write-capable token can remain in the repo for later steps/actions.

Minimal tightening 
 permissions:
-  contents: write
+  contents: read
@@
       - uses: actions/checkout@v4
         with:
           fetch-depth: 0
+          persist-credentials: false
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/apk-qnn-asr.yaml around lines 15 - 16, The workflow
currently grants repo write access via "permissions: contents: write" and leaves
checkout credentials persisted; change the workflow-level permissions to the
least privilege needed (e.g., remove or set "contents: read" / minimal
permissions since you only publish to Hugging Face with HF_TOKEN) and update the
checkout step (actions/checkout@v4) to include "persist-credentials: false" so
the write-capable token is not left available to later steps; ensure HF_TOKEN is
used only for the Hugging Face publish step and no steps require repo write
permissions.


env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: true

jobs:
asr:
if: github.repository_owner == 'csukuangfj' || github.repository_owner == 'k2-fsa'
runs-on: ${{ matrix.os }}
name: ${{ matrix.index }}/${{ matrix.total }}
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest]
total: ["1"]
index: ["0"]

steps:
- uses: actions/checkout@v4

@coderabbitai coderabbitai Bot Jun 2, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
rg -n '^\s*-\s*uses:\s+[^@]+@([^0-9a-f]{40}|[0-9a-f]{1,39})\s*$' .github/workflows/apk-qnn-asr.yaml

Repository: k2-fsa/sherpa-onnx

Length of output: 44

🏁 Script executed:

#!/bin/bash
set -euo pipefail

FILE=".github/workflows/apk-qnn-asr.yaml"

echo "== File = $FILE =="
rg -n '^\s*-\s*uses:\s+' "$FILE" || true

echo
echo "== Non-SHA refs (anything not 40-hex) =="
rg -n '^\s*-\s*uses:\s+[^@]+@(?!(?:[0-9a-f]{40})$)[^\s]+' "$FILE" || true

echo
echo "== SHA refs =="
rg -n '^\s*-\s*uses:\s+[^@]+@[0-9a-f]{40}$' "$FILE" || true

Repository: k2-fsa/sherpa-onnx

Length of output: 631

🏁 Script executed:

#!/bin/bash
set -euo pipefail
FILE=".github/workflows/apk-qnn-asr.yaml"

echo "== Total uses occurrences =="
rg -n 'uses:' "$FILE" || true

echo
echo "== Pinned to 40-hex SHAs? =="
rg -n '^\s*-\s*uses:\s+[^@]+@[0-9a-f]{40}\s*$' "$FILE" || true

echo
echo "== Non-SHA action refs (not 40-hex at end) =="
rg -n '^\s*-\s*uses:\s+[^@]+@\S+\s*$' "$FILE" || true

Repository: k2-fsa/sherpa-onnx

Length of output: 595

Pin all GitHub Actions uses: to full commit SHAs.

This workflow produces signed/uploaded APK artifacts, but it uses mutable action tags (@v4, @v1, @v1.2, @v3), e.g.:

  • actions/checkout@v4 (34)
  • actions/setup-java@v4 (44)
  • hendrikmuhs/ccache-action@v1.2 (50)
  • actions/upload-artifact@v4 (85)
  • r0adkll/sign-android-release@v1 (106)
  • nick-fields/retry@v3 (141)

Pin each uses: entry to an immutable 40-character commit SHA.

🧰 Tools
🪛 zizmor (1.25.2)

[warning] 34-36: credential persistence through GitHub Actions artifacts (artipacked): does not set persist-credentials: false

(artipacked)

[error] 34-34: unpinned action reference (unpinned-uses): action is not pinned to a hash (required by blanket policy)

(unpinned-uses)

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/apk-qnn-asr.yaml at line 34, The workflow uses mutable
action tags (e.g., actions/checkout@v4, actions/setup-java@v4,
hendrikmuhs/ccache-action@v1.2, actions/upload-artifact@v4,
r0adkll/sign-android-release@v1, nick-fields/retry@v3); update each `uses:`
entry to point to the corresponding full 40-character commit SHA instead of the
tag so the workflow is pinned to immutable commits (replace occurrences of
actions/checkout@v4, actions/setup-java@v4, hendrikmuhs/ccache-action@v1.2,
actions/upload-artifact@v4, r0adkll/sign-android-release@v1,
nick-fields/retry@v3 with their respective full commit SHAs).

with:
fetch-depth: 0

- name: Update version
shell: bash
run: |
./new-release.sh
git diff .

- uses: actions/setup-java@v4
with:
distribution: 'temurin'
java-version: '21'

- name: ccache
uses: hendrikmuhs/ccache-action@v1.2
with:
key: ${{ matrix.os }}-android-qnn-asr

- name: Display NDK HOME
shell: bash
run: |
echo "ANDROID_NDK_LATEST_HOME: ${ANDROID_NDK_LATEST_HOME}"
ls -lh ${ANDROID_NDK_LATEST_HOME}

- name: Install Python dependencies
shell: bash
run: |
python3 -m pip install --upgrade pip jinja2

- name: Setup build tool version variable
shell: bash
run: |
BUILD_TOOL_VERSION=$(ls /usr/local/lib/android/sdk/build-tools/ | tail -n 1)
echo "BUILD_TOOL_VERSION=$BUILD_TOOL_VERSION" >> $GITHUB_ENV
echo "Last build tool version is: $BUILD_TOOL_VERSION"

- name: Generate build script
shell: bash
run: |
cd scripts/apk

total=${{ matrix.total }}
index=${{ matrix.index }}

python3 ./generate-qnn-asr-apk-script.py --total $total --index $index

chmod +x build-apk-qnn-asr.sh
mv -v ./build-apk-qnn-asr.sh ../..

- uses: actions/upload-artifact@v4
with:
name: build-script-${{ matrix.total }}-${{ matrix.index }}
path: ./build-apk-qnn-asr.sh

- name: build APK
shell: bash
run: |
export CMAKE_CXX_COMPILER_LAUNCHER=ccache
export PATH="/usr/lib/ccache:/usr/local/opt/ccache/libexec:$PATH"
cmake --version

export ANDROID_NDK=$ANDROID_NDK_LATEST_HOME
./build-apk-qnn-asr.sh

- name: Display APK
shell: bash
run: |
ls -lh ./apks/
du -h -d1 .

- uses: r0adkll/sign-android-release@v1
name: Sign app APK
with:
releaseDirectory: ./apks
signingKeyBase64: ${{ secrets.ANDROID_SIGNING_KEY }}
alias: ${{ secrets.ANDROID_SIGNING_KEY_ALIAS }}
keyStorePassword: ${{ secrets.ANDROID_SIGNING_KEY_STORE_PASSWORD }}
env:
BUILD_TOOLS_VERSION: ${{ env.BUILD_TOOL_VERSION }}

- name: Display APK after signing
shell: bash
run: |
ls -lh ./apks/
du -h -d1 .

- name: Rename APK after signing
shell: bash
run: |
cd apks
rm -fv signingKey.jks
rm -fv *.apk.idsig
rm -fv *-aligned.apk

all_apks=$(ls -1 *-signed.apk)
for apk in ${all_apks[@]}; do
n=$(echo $apk | sed -e s/-signed//)
mv -v $apk $n
done

cd ..

- name: Publish to huggingface
env:
HF_TOKEN: ${{ secrets.HF_TOKEN }}
uses: nick-fields/retry@v3
with:
max_attempts: 20
timeout_seconds: 200
shell: bash
command: |
git config --global user.email "csukuangfj@gmail.com"
git config --global user.name "Fangjun Kuang"

rm -rf huggingface
export GIT_LFS_SKIP_SMUDGE=1
export GIT_CLONE_PROTECTION_ACTIVE=false

SHERPA_ONNX_VERSION=$(grep "SHERPA_ONNX_VERSION" ./CMakeLists.txt | cut -d " " -f 2 | cut -d '"' -f 2)

git clone https://csukuangfj2:$HF_TOKEN@huggingface.co/csukuangfj2/sherpa-onnx-apk huggingface
cd huggingface
git fetch
git pull
git merge -m "merge remote" --ff origin main

d=qnn-asr/$SHERPA_ONNX_VERSION
mkdir -p $d
cp -v ../apks/*.apk $d/
git lfs track "*.apk"
git add .
git commit -m "add more qnn asr apks"
git push https://csukuangfj2:$HF_TOKEN@huggingface.co/csukuangfj2/sherpa-onnx-apk main
11 changes: 11 additions & 0 deletions android/SherpaOnnx/app/src/main/AndroidManifest.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,17 @@
android:supportsRtl="true"
android:theme="@style/Theme.SherpaOnnx"
tools:targetApi="31">
<!--
Required by the QNN HTP backend on Qualcomm devices. The backend talks to
the DSP through the system library libcdsprpc.so; declaring it here lets
Android expose that vendor library to the app. Without this entry,
qnn_interface.deviceCreate() can fail with error 14001 even when the QNN
user-space libraries are packaged in jniLibs.
-->
<uses-native-library
android:name="libcdsprpc.so"
android:required="false" />

<activity
android:name=".MainActivity"
android:label="ASR: Next-gen Kaldi"
Expand Down
101 changes: 99 additions & 2 deletions android/SherpaOnnx/app/src/main/java/com/k2fsa/sherpa/onnx/MainActivity.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
package com.k2fsa.sherpa.onnx

import android.Manifest
import android.content.Context
import android.content.res.AssetManager
import android.content.pm.PackageManager
import android.media.AudioFormat
import android.media.AudioRecord
Expand All @@ -14,7 +16,8 @@ import androidx.appcompat.app.AppCompatActivity
import androidx.core.app.ActivityCompat
import java.io.File
import java.io.FileOutputStream
import java.io.IOException
import java.io.InputStream
import java.io.OutputStream
import kotlin.concurrent.thread

private const val TAG = "sherpa-onnx"
Expand All @@ -24,6 +27,54 @@ private const val REQUEST_RECORD_AUDIO_PERMISSION = 200
//
// adb emu avd hostmicon

private fun assetExists(assetManager: AssetManager, path: String): Boolean {
val dir = path.substringBeforeLast('/', "")
val fileName = path.substringAfterLast('/')

val files = assetManager.list(dir) ?: return false
return files.contains(fileName)
}
Comment on lines +30 to +36

@gemini-code-assist gemini-code-assist Bot Jun 2, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Using AssetManager.list() to check if a single asset exists is highly inefficient because it lists all files in the directory and performs a linear search. This can cause significant startup delays if there are many assets. A much faster and cleaner approach is to try opening the asset directly using AssetManager.open().

private fun assetExists(assetManager: AssetManager, path: String): Boolean {
    if (path.isEmpty()) return false
    return try {
        assetManager.open(path).use { }
        true
    } catch (e: Exception) {
        false
    }
}


private fun copyAssetToInternalStorage(path: String, context: Context): String {
val targetRoot = context.filesDir
val outFile = File(targetRoot, path)

if (!assetExists(context.assets, path = path)) {
outFile.parentFile?.mkdirs()
Log.i(TAG, "$path does not exist, return ${outFile.absolutePath}")
return outFile.absolutePath
}
Comment on lines +38 to +46

@gemini-code-assist gemini-code-assist Bot Jun 2, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

If path is empty, copyAssetToInternalStorage will attempt to resolve it against context.filesDir, which points to the files directory itself. This results in calling parentFile?.mkdirs() on the app's internal storage root and returning an incorrect path. Adding a guard clause to return early when path is empty prevents this issue.

Suggested change
private fun copyAssetToInternalStorage(path: String, context: Context): String {
val targetRoot = context.filesDir
val outFile = File(targetRoot, path)
if (!assetExists(context.assets, path = path)) {
outFile.parentFile?.mkdirs()
Log.i(TAG, "$path does not exist, return ${outFile.absolutePath}")
return outFile.absolutePath
}
private fun copyAssetToInternalStorage(path: String, context: Context): String {
if (path.isEmpty()) return path
val targetRoot = context.filesDir
val outFile = File(targetRoot, path)
if (!assetExists(context.assets, path = path)) {
outFile.parentFile?.mkdirs()
Log.i(TAG, "$path does not exist, return ${outFile.absolutePath}")
return outFile.absolutePath
}

Comment on lines +42 to +46

@coderabbitai coderabbitai Bot Jun 2, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Don't treat missing required assets as success.

Returning outFile.absolutePath when the asset is absent is only valid for first-run generated QNN context binaries. Reusing the same helper for required inputs like tokens.txt, libencoder.so, libdecoder.so, and libjoiner.so hides packaging mistakes and defers the failure to native init with a much less actionable error. Split required vs. optional copies, or add a mustExist flag and use the optional path only for contextBinary.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@android/SherpaOnnx/app/src/main/java/com/k2fsa/sherpa/onnx/MainActivity.kt`
around lines 42 - 46, The current helper silently returns outFile.absolutePath
when assetExists(...) is false, which masks missing required assets; modify the
helper (the function that checks assetExists(context.assets, path) and writes to
outFile) to accept a mustExist (or isOptional) boolean parameter and, for
mustExist==true, throw or return an error (or null) instead of returning
outFile.absolutePath so callers can fail fast; update callers that copy required
inputs (tokens.txt, libencoder.so, libdecoder.so, libjoiner.so) to call the
helper with mustExist=true and only allow mustExist=false for the generated
contextBinary path so missing optional assets still follow the old behavior.


if (outFile.exists()) {
val assetSize = context.assets.open(path).use { it.available() }
if (outFile.length() == assetSize.toLong()) {
Log.i(TAG, "$targetRoot/$path already exists, skip copying, return $targetRoot/$path")
return outFile.absolutePath
}
}

outFile.parentFile?.mkdirs()

context.assets.open(path).use { input: InputStream ->
FileOutputStream(outFile).use { output: OutputStream ->
input.copyTo(output)
}
}
Log.i(TAG, "Copied $path to $targetRoot/$path")

return outFile.absolutePath
}

private fun copyAssetListToInternalStorage(paths: String, context: Context): String {
if (paths.isBlank()) return paths

return paths.split(",")
.map { it.trim() }
.filter { it.isNotEmpty() }
.map { copyAssetToInternalStorage(it, context) }
.joinToString(",")
}

class MainActivity : AppCompatActivity() {
private val permissions: Array<String> = arrayOf(Manifest.permission.RECORD_AUDIO)

Expand Down Expand Up @@ -232,8 +283,54 @@ class MainActivity : AppCompatActivity() {
config.hr = hr
}

var assetManager: AssetManager? = application.assets
if (config.modelConfig.provider == "qnn") {
Log.i(TAG, "nativelibdir: ${applicationInfo.nativeLibraryDir}")
OnlineRecognizer.prependAdspLibraryPath(applicationInfo.nativeLibraryDir)

val transducer = config.modelConfig.transducer
val qnnConfig = transducer.qnnConfig

if (qnnConfig.backendLib.isEmpty()) {
throw IllegalArgumentException("You should provide libQnnHtp.so for qnn")
}

config.modelConfig.tokens =
copyAssetToInternalStorage(config.modelConfig.tokens, this)

if (transducer.encoder.isNotEmpty()) {
transducer.encoder =
copyAssetToInternalStorage(transducer.encoder, this)
}

if (transducer.decoder.isNotEmpty()) {
transducer.decoder =
copyAssetToInternalStorage(transducer.decoder, this)
}

if (transducer.joiner.isNotEmpty()) {
transducer.joiner =
copyAssetToInternalStorage(transducer.joiner, this)
}

if (qnnConfig.contextBinary.isNotEmpty()) {
qnnConfig.contextBinary =
copyAssetListToInternalStorage(qnnConfig.contextBinary, this)
}

if (config.hr.lexicon.isNotEmpty()) {
config.hr.lexicon = copyAssetToInternalStorage(config.hr.lexicon, this)
}

if (config.hr.ruleFsts.isNotEmpty()) {
config.hr.ruleFsts = copyAssetToInternalStorage(config.hr.ruleFsts, this)
}

assetManager = null
}

recognizer = OnlineRecognizer(
assetManager = application.assets,
assetManager = assetManager,
config = config,
)
}
Expand Down
1 change: 1 addition & 0 deletions android/SherpaOnnx/app/src/main/java/com/k2fsa/sherpa/onnx/QnnConfig.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
../../../../../../../../../../sherpa-onnx/kotlin-api/QnnConfig.kt
Loading