Skip to content

feat(wake-word): support Chinese wake phrases via sherpa wenetspeech pinyin model - #85396

Open
ruhipiano wants to merge 1 commit into
NousResearch:mainfrom
ruhipiano:feat/wake-word-chinese
Open

ruhipiano wants to merge 1 commit into
NousResearch:mainfrom
ruhipiano:feat/wake-word-chinese

Conversation

@ruhipiano

Copy link
Copy Markdown

Summary

The sherpa wake-word engine hardcoded BPE tokenization (text2token + bpe.model), which only covers Latin-script phrases on the default English GigaSpeech model. The wenetspeech KWS model — sherpa-onnx's Chinese counterpart — is pinyin-modeled (声母/韵母 phonemes) and ships no bpe.model, so pointing wake_word.sherpa.model_dir at it crashed at listener init.

Changes

  • Model-family detection: _SherpaKwsEngine now checks for bpe.model — BPE models keep the existing text2token path; pinyin models (no bpe.model) romanize phrases to phonemes.
  • Pinyin tokenizer (_phrase_to_pinyin_phonemes): splits pypinyin's toned syllables (Style.TONE) after the initial — nǐ → n ǐ, wén → w én, huì → h uì — which also fixes pypinyin's mis-timed bare finals (úen, ueì), with a ü→u fallback for finals the vocab spells without umlaut (üé → ué).
  • Graceful degradation: phrases a model cannot represent (e.g. Latin on the pinyin model) are skipped with a warning instead of failing the whole listener — so mixed-language profile routing degrades safely rather than crashing.

Verification

  • Unit tests: 31 pass (5 new pinyin-tokenization tests, offline-safe via tmp_path + pytest.importorskip).
  • E2E against the real wenetspeech model + its bundled test wavs: all 8 official keywords recognized (文森特卡索/周望军/朱丽楠/蒋友伯/女儿/法国/见面会/落实) with correct phoneme output matching the official keywords.txt byte-for-byte.
  • TTS-rendered 你好妮妮 fires the listener with correct profile routing (last_match == ('你好妮妮', 'nina')).

Docs

New "Chinese wake phrases (pinyin model)" section under the sherpa engine docs with model download link and config example.

Notes

  • pypinyin was already a wake.sherpa lazy-dep (sherpa_onnx's text2token imports it unconditionally, so no new dependency.
  • The wenetspeech model (~13 MB) is auto-downloadable from sherpa-onnx releases; users point at the unpacked directory.
    EOF
    )

…pinyin model

The sherpa engine hardcoded BPE tokenization (text2token + bpe.model),
which only covers Latin-script phrases on the English GigaSpeech model.
The wenetspeech KWS model is pinyin-modeled (声母/韵母 phonemes) and ships
no bpe.model, so pointing model_dir at it crashed at init.

Detect the model family by the presence of bpe.model:
- BPE models keep the existing text2token path (Latin phrases).
- Pinyin models romanize each phrase with pypinyin: toned syllables
  (Style.TONE) are split after their initial (nǐ → n ǐ, wén → w én,
  huì → h uì), with a ü→u fallback for finals the vocab spells without
  the umlaut (üé → ué). Phrases a model cannot represent (e.g. Latin on
  the pinyin model) are skipped with a warning instead of failing the
  listener, so mixed-language profile routing degrades gracefully.

Verified end-to-end against the real wenetspeech model: all 8 official
test wavs recognize (文森特卡索/周望军/朱丽楠/蒋友伯/女儿/法国/见面会/落实)
and a TTS-rendered 你好妮妮 fires with correct profile routing.

Docs: new 'Chinese wake phrases (pinyin model)' section.
@alt-glitch alt-glitch added type/feature New feature or request tool/tts Text-to-speech and transcription area/i18n Localization, locales, translations P3 Low — cosmetic, nice to have labels Aug 13, 2026
@Enough1122

Copy link
Copy Markdown
Contributor

AI code review — automated review for reference, author can ignore or act on any point.

PR: feat(wake-word): support Chinese wake phrases via sherpa wenetspeech pinyin model

  1. zip(initials, syllables) in _phrase_to_pinyin_phonemes (wake_word.py:119) silently truncates if pypinyin returns different-length lists for a mixed phrase (Chinese + Latin/digits), which would misalign phoneme output for the remainder of the phrase. Consider guarding len(initials) == len(syllables) and skipping the phrase (or the unaligned tail) instead of zipping blindly.
  2. _sherpa_token_vocab re-reads and re-parses tokens.txt on every phrase in __init__ (per enrolled profile + config phrase), i.e. O(phrases × vocab file) I/O on each listener start. Reading the vocab once in the constructor and passing it to the tokenizer would be simpler and cheaper.
  3. if not phrase.isascii(): return [] (wake_word.py:155) drops accented-Latin phrases ("héllo", "café") even though the BPE vocab may contain them. Consider allowing Latin-1/Unicode letters (or checking against the vocab) instead of ASCII-only, since the docstring says BPE covers "Latin-script phrases".
  4. The ü→u fallback only fires when rest not in vocab, so it can't corrupt a vocab that spells üè correctly — good. But a vocab spelling neither üé nor ué leaves the final silently dropped with no warning; a debug log for skipped finals would help users diagnose unexpected phrase degradation.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/i18n Localization, locales, translations P3 Low — cosmetic, nice to have tool/tts Text-to-speech and transcription type/feature New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants