fix(local-inference): real Kokoro StyleTTS-2/iSTFTNet decoder — Kokoro TTS actually speaks (#9588)

[lalalune]

Summary

Kokoro TTS never actually produced speech. This makes it work — end-to-end, verified intelligible on the already-shipped bundles/2b Q4_K_M GGUF (no model republish needed). Native llama.cpp submodule bump (82925c0e9 → ba598f562, branch elizaOS/llama.cpp#feat/9588-kokoro-real-decoder, tag eliza-pin-9588-kokoro-decoder).

Two root causes

1. The fused kokoro_synthesize was a placeholder. It ran a hand-rolled synth_spectrogram (envelope × 1/f timbre × random phase → iSTFT) and discarded the model — the real text-encoder → predictor → decoder graph was stubbed as "J2 follow-up."
2. Dtype bug. The scalar predictor/decoder read every tensor as raw float*, but the GGUF ships F16 + Q5_0/Q4_K/Q6_K. Reading quantized block bytes as F32 produced garbage → BERT collapsed every position to one vector → a constant-pitch buzz. (This is why the earlier "regenerate the GGUF" instinct was wrong: F16 == Q4 == identical beep; the bug was in the runtime, not the artifact.)

What landed (in the submodule)

• Dequant-at-load — kokoro_load_model builds a parallel all-F32 context (ggml to_float traits handle F16 + every quant type); predictor/decoder read F32 only. This alone un-breaks the already-correct prosody predictor (BERT now matches the PyTorch reference to 3 decimals).
• Real iSTFTNet decoder — kokoro-decoder-front.h (F0/N conv, encode, asr_res, 4 AdainResBlk1d decode blocks) + kokoro-generator.{h,cpp} (SineGen/SourceModuleHnNSF harmonic source from F0, center=True STFT, 2 ConvTranspose upsamples, 6 Snake AdaINResBlock1, noise convs/res, conv_post, center=True iSTFT) + kokoro-decoder.{h,cpp} (binds the F32 tensors, runs front→generator). Wired into kokoro_synthesize (predictor → transpose asr → decoder), variable-length.
• Real G2P — kokoro-phonemes drives libespeak-ng (en-us IPA → Kokoro vocab ids, reproducing the reference token sequence); ASCII fallback only when espeak is absent. CMake auto-detects libespeak-ng.

Verification (Apple M-series)

• End-to-end kokoro-tts (text → speech) on the published Q4_K_M GGUF, whisper-transcribed across phrases:
  • "Hello, this is a native Kokoro voice test." → "Hello, this is a native Kakoro voice test."
  • "The quick brown fox jumps over the lazy dog." → exact
  • "Open the pod bay doors please." → "Open the pod bay doors, please."
• Same on the all-F32 regenerated GGUF.
• Per-component standalone tests vs the PyTorch reference (16-agent validation workflow): decoder-front corr 1.0 every stage; Snake resblock corr 1.0 (all 6); harmonic source + STFT corr 0.9999; center=True iSTFT corr 1.0; full generator corr 0.993 + whisper.
• New unit test test-kokoro-g2p-espeak (exact reference-id match); kokoro_lib builds clean; existing kokoro tests pass.

Notes

• The shipped bundle GGUF is unchanged. The runtime dequant + real decoder make the existing published artifact speak — closes Kokoro TTS fails to load on desktop — fused-lib loader vs GGUF tensor-name mismatch #9588 without any HF republish.
• The pin is off-main (consistent with the existing eliza pin lineage); ba598f562 is reachable via the fork branch + the eliza-pin-9588-kokoro-decoder tag. The decoder also lands on the fork's main separately so future pin bumps don't regress it.

🤖 Generated with Claude Code

[greptile-apps]

Your free trial has ended. If you'd like to continue receiving code reviews, you can add a payment method here.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 58ffcae0-f916-4700-8b1c-0bee2e436683

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/9588-kokoro-real-decoder

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[claude]

Claude encountered an error —— View job

---

I'll analyze this and get back to you.