Add tts-cpp/ subtree (Chatterbox Turbo + Multilingual + Supertonic TTS) + integration fixes

[GustavoA1604]

Summary

Adds the tts-cpp/ in-tree subtree alongside the existing parakeet-cpp/ subtree, completing the QVAC speech stack inside this whisper.cpp fork. tts-cpp/ is a port of gianni-cor/chatterbox.cpp — Resemble AI's Chatterbox TTS (Turbo, English-only; Multilingual, 18 tier-1 languages) plus Supertonic TTS — running on ggml with CPU / Metal / Vulkan / CUDA backends, no Python/PyTorch dependency.

The PR also wires the subtree into the QVAC speech-stack consumption pattern (ggml-speech vcpkg port from qvac-ext-ggml/speech), folds in the round-3 review fixes that landed in the standalone repo after the initial port, and adds one parakeet-cpp streaming-API change required by downstream consumers.

• 16 commits
• 103 files changed, +57 184 lines (subtree drop is the bulk; the integration / review commits are small)
• Branch: tts-cpp → master

What's in the PR

1. tts-cpp/ subtree drop (commit ef840d5)

Initial squashed import from the standalone chatterbox.cpp source-of-truth. Layout follows parakeet-cpp/:

• tts-cpp/include/tts-cpp/ — public C++ API (chatterbox/engine.h, supertonic/engine.h, backend.h, log.h, tts-cpp.h, export.h).
• tts-cpp/src/ — implementation (Chatterbox Turbo + MTL T3, Supertonic flow + vocoder, S3Gen, S3Tokenizer, CAMPPlus voice encoder, mel2wav HiFT, MTL tokenizer, GPT-2 BPE, log).
• tts-cpp/test/ — gtest suite mirroring each engine stage (T3, S3Gen, vocoder, CAMPPlus, MTL tokenizer, streaming, etc.).
• tts-cpp/scripts/ — Python conversion scripts (HF safetensors → GGUF for T3-Turbo / T3-MTL / S3Gen / Supertonic2; reference-trace dumps for diffing against PyTorch; voice extraction).
• tts-cpp/CMakeLists.txt + tts-cpp/cmake/tts-cppConfig.cmake.in — install/export, OpenMP detection, find_package(tts-cpp CONFIG) consumer surface.
• tts-cpp/README.md + tts-cpp/PROGRESS.md + tts-cpp/PROGRESS_SUPERTONIC.md — developer docs.

Performance summary (full table in tts-cpp/README.md):

• Turbo, Vulkan RTX 5090, Q4_0: 463 ms wall, 13.8× faster than ONNX Runtime CPU Q4.
• Multilingual, Metal M3 Ultra, Q4_0 + --cfm-steps 7: 1.05 s wall, 48.4× faster than ONNX Runtime CPU Q4.

2. Top-level README.md pointer (commit a2f2dd6)

Adds a small "QVAC speech-stack ports" section between the upstream whisper.cpp intro media and the Quick start, pointing at both tts-cpp/ and parakeet-cpp/ as in-tree subtrees with their own READMEs / build flows / public C++ APIs. Upstream whisper.cpp build below is unaffected.

3. Integration deltas vs the standalone repo (commits fa0d490, ae34c58, 8ba10a6, e673182)

The standalone chatterbox.cpp ships with scripts/setup-ggml.sh + patches/ for bundled-ggml dev builds. Per reviewer guidance, the in-tree subtree is converged on a single ggml source-of-truth — the QVAC speech-stack ggml-speech vcpkg port (which ships the patches pre-applied) — instead of carrying a parallel patches tree:

• fa0d490 (review #5+#6): default TTS_CPP_USE_SYSTEM_GGML=ON in this subtree; bundled-ggml branch hard-errors with a pointer at the canonical consumption path; deletes tts-cpp/scripts/setup-ggml.sh (it referenced patches/ that no longer exist; it would have errored out anyway).
• ae34c58 (review #26): rewrite tts-cpp/README.md §1 from "clone + setup-ggml.sh + patches" to "build via the QVAC speech stack"; flip the TTS_CPP_USE_SYSTEM_GGML row default in the cmake-options table; rewrite Consumer integration from the standalone perspective to the wrapper-port perspective; relabel benchmark and code-tree references from chatterbox.cpp/ to tts-cpp/ where they refer to this directory (kept where they refer to the upstream project itself).
• 8ba10a6 (review N8): drop the now-unreachable TTS_CPP_GGML_LIB_PREFIX block (the find_package(ggml) path resolves libspeech-ggml-* natively from the vcpkg port, the local-rename helper is dead code in this subtree). README row updated with a strikethrough + "n/a in this subtree" note pointing at the standalone repo for the OFF path.
• e673182 (review N10): scrub two stale patches/ references from tts-cpp/README.md (§3.24 Metal-optimisation explanation and the repository-layout tree); rewrite to attribute the patch to the ggml-speech vcpkg port. Five other patches/ mentions stay on purpose — they're cross-references to the standalone repo or describe the rejection rationale.

4. Round-3 review fixes mirrored from standalone (commits 4b5d2d7, 28ef67d, e8f6065, 04b87ea, 64abb81, 1963f9f, 942686d)

Re-syncs the subtree with the standalone source-of-truth for fixes that landed there after the initial port. Each commit is a 1:1 git apply --directory=tts-cpp/ of the corresponding upstream diff:

• 4b5d2d7 (review N1–N7): Supertonic alive-registry guards thread_local cache teardown vs freed backend (N1); Engine BPE try/catch + dead-state cleanup (N3, N6, N7); log docstring softening (N2); s3gen cancel checkpoint between STFT and HiFT + Engine::cancel() doc (N4); s3gen_preload/unload refcount semantics on the public header (N5).
• 28ef67d: drop internal ggml-quants.h include from supertonic_gguf.cpp; use ggml_get_type_traits()->to_float instead. Required because ggml-speech doesn't install internal ggml headers. Bit-equivalent runtime.
• e8f6065: tts-cppConfig.cmake.in re-imports OpenMP via find_dependency before including the targets file. Without this, consumers of the static archive failed at target-property time with OpenMP::OpenMP_CXX … target was not found.
• 04b87ea: scope the find_dependency(OpenMP) to COMPONENTS CXX. Bare find_dependency(OpenMP) was probing OpenMP_C in consumers, which fails on bare-make's clang-cl-style toolchain even when CXX-side OpenMP is fine.
• 64abb81: TTS_CPP_OPENMP option, defaults OFF on Windows non-MinGW (where vcpkg's MSVC toolchain links OpenMP transitively but consumers can't re-probe it). Override via -DTTS_CPP_OPENMP_USER_OVERRIDE=ON -DTTS_CPP_OPENMP=ON. Trade-off: 9 #pragma omp parallel for loops in campplus.cpp run serially in this build mode; CAMPPlus is a small fraction of total synth time.
• 1963f9f: Engine::backend_device() public API + BackendDevice enum on chatterbox::Engine and supertonic::Engine, mirroring parakeet::Engine::backend_device(). Routes through ggml_backend_get_device + ggml_backend_dev_type; works in both GGML_BACKEND_DL modes. Required by qvac3's tts-ggml addon to mirror ParakeetModel's load-time backend resolution.
• 942686d: synthesize_batch gates apply_trim_fade on the actual presence of a voice override. The previous unconditional fade was clipping the leading consonant of the first word ("Hello" → "lo", "El" → "l", "A" → nothing) for the chatterbox-mtl built-in-voice path. Reference-audio / voice_dir paths keep the fade.

5. MTL Chatterbox correctness fixes (commits db87f42, 0b44674)

Two issues that surfaced once chatterbox-mtl ran end-to-end on real workloads:

• db87f42: Engine::run_t3 for MTL wraps text tokens with start_text_token (255) / stop_text_token, matching chatterbox_cli.cpp's tokenisation path and the Python ChatterboxMultilingualTTS.generate reference (mtl_tts.py:288-291). Without it, the autoregressive decode dropped the first speech tokens, audible as a missing leading syllable. Turbo path is unaffected.
• 0b44674: port the CLI's existing 3-identical-token early-stop guard into Engine::run_t3 (gated on is_mtl). MTL T3 occasionally emits an end-of-speech silence cadence mid-utterance and then hallucinates ~40 s of trailing low-energy junk until n_predict=1000. The CLI was already guarded; the addon path (which doesn't go through the CLI) was hitting the regression on certain language/seed combinations (most reliably reproduced on German with seed=42).

6. parakeet-cpp/ SentencePiece word-start signal (commit 761eca0)

Adds bool starts_word on parakeet::StreamingSegment, set true when the segment's first token's piece carries the SentencePiece ▁ (U+2581) word-boundary marker. Streaming consumers can use it to decide whether to insert a space between successive segments without re-parsing whitespace from seg.text (the inner detokenizer strips leading whitespace at the session level). Also exposes bool token_is_word_start(BpeVocab, int32_t) from sentencepiece_bpe.h so other engines that build their own segments (EOU per-utterance, attributed) can stamp the flag the same way. Defaults starts_word = true so existing callers are byte-equivalent.

Bundled into this PR rather than its own because the parakeet-cpp/ consumer in qvac3/packages/parakeet-ggml and the tts-cpp/ consumer in qvac3/packages/tts-ggml ship via the same ggml-speech vcpkg port version bump; splitting them would force two coordinated registry flips for a single addon release.

Design notes (preempting review questions)

These call out the deliberate choices that look unusual at first glance — flagging here so re-review doesn't re-litigate them.

Why is TTS_CPP_USE_SYSTEM_GGML=ON the default in this subtree?

Reviewer guidance (#5, #6): converge the speech stack on a single ggml source-of-truth. The ggml-speech vcpkg port (qvac-ext-ggml/speech) ships the chatterbox-specific Metal / OpenCL patches pre-applied — carrying a parallel patches/ tree inside this subtree would mean two sources of truth for the same patches and a coordination tax on every ggml bump. The bundled-ggml dev flow is preserved in the standalone chatterbox.cpp repo, which keeps setup-ggml.sh + patches/ + TTS_CPP_USE_SYSTEM_GGML=OFF default; this subtree is the integrated artefact, not a parallel dev environment.

-DTTS_CPP_USE_SYSTEM_GGML=OFF here intentionally hard-errors at configure time with a pointer at both the canonical consumption path (ggml-speech vcpkg port) and the standalone repo for users who need bundled ggml.

Why mirror commits from chatterbox.cpp instead of squashing into the initial port?

Two reasons:

1. The initial port (ef840d5) was force-pushed five times during the integration cycle (review-iteration round 1 → round 3); each round-3 fix landed on the standalone repo after this PR's force-push window closed. Mirroring them as separate commits keeps the diff against the standalone source-of-truth reviewable: chatterbox.cpp commit hash → tts-cpp commit, one-to-one.
2. Each mirror commit captures the upstream commit hash in its message, so a future bisect or audit can trace back to the standalone PR / discussion.

If a squashed history is preferred at merge time, GitHub's "Squash and merge" handles it; the per-commit messages are written to survive that.

Why is the parakeet-cpp/ change in this PR?

parakeet-cpp/ and tts-cpp/ ship through the same ggml-speech vcpkg port version bump (single port-version flip in qvac-registry-vcpkg). Consumers (qvac3/packages/parakeet-ggml, qvac3/packages/tts-ggml) bump together. Splitting parakeet-cpp/'s starts_word change into its own PR would force two coordinated registry flips for one release. The change is small (+15 lines of API surface, no behaviour change for existing callers) and gated by an additive bool field that defaults to true.

Why is OpenMP defaulted OFF on Windows non-MinGW?

vcpkg's MSVC toolchain port build links OpenMP::OpenMP_CXX into the static archive's transitive interface; consumers — including bare-make's clang-cl CMake — then re-probe OpenMP_CXX (or OpenMP_C) at find_package(tts-cpp) time, and that probe fails on toolchains where CXX-OpenMP isn't auto-detected. Defaulting OFF on the affected toolchain combination keeps the consumer surface portable; the perf cost is bounded to 9 #pragma omp parallel for loops in campplus.cpp (CAMPPlus runs once per voice-encode at session init, small fraction of total synth time). Override available for users on toolchains with working CXX OpenMP.

Why ~57 K added lines?

The bulk is the standalone chatterbox.cpp source dropped under tts-cpp/ (ef840d5). Major contributors:

• tts-cpp/src/mtl_unicode_tables.inc — autogenerated NFKD lookup tables (one-time, regenerable via tts-cpp/scripts/gen-nfkd-table.py).
• tts-cpp/src/dr_wav.h, tts-cpp/src/npy.h — vendored single-header libs (verbatim upstream copies; their licences are in tts-cpp/NOTICE).
• tts-cpp/test/ — gtest suite (~22 test files, one per engine stage).
• tts-cpp/PROGRESS.md, tts-cpp/PROGRESS_SUPERTONIC.md — developer notebooks.

Implementation source under tts-cpp/src/ (excluding the autogenerated table and vendored headers) is roughly 12 K lines.

Why doesn't this PR bump qvac-registry-vcpkg/ports/tts-cpp?

Per the standalone repo's pre-merge convention: while the PR is open, the port-version 0 entry is force-amended in place to point at the latest tip of this branch. The actual port bump (port-version 0 → port-version 1, or new commit hash for version 0) lands in qvac-registry-vcpkg after this PR merges to master, in a follow-up PR there.

Why are some chatterbox.cpp references kept verbatim in tts-cpp/README.md?

Five intentional ones survived the §1 rewrite (commit ae34c58):

1. The title-card upstream URL.
2. The §1 "use the standalone repo for bundled-ggml dev builds" pointer.
3. The cmake-options note about the OFF default (lives upstream).
4. The "How TTS_CPP_USE_SYSTEM_GGML=ON resolves ggml" prose (cross-references the standalone build flow).
5. The repo-layout caveat naming the source-of-truth.

Each one points at github.com/gianni-cor/chatterbox.cpp by URL or repo name, not at this directory. They stay because the standalone repo is the development source-of-truth for the engine code and we want a single grep to find it.

Why is BackendDevice shaped exactly like parakeet::Engine::backend_device()?

Intentional API parallelism so the qvac3 addons can share their backend-resolution code path between tts-ggml and parakeet-ggml. Both addons read backend_device() + backend_name() at session init, map through a shared backendIdFromName(), and expose the same RuntimeStats shape on the JS API. Diverging the C++ API would mean two parallel addon-side wrappers for the same data.

Test plan

• tts-cpp/ builds cleanly via vcpkg with -DTTS_CPP_USE_SYSTEM_GGML=ON against the ggml-speech port (qvac-ext-ggml/speech branch).
• -DTTS_CPP_USE_SYSTEM_GGML=OFF configure attempt errors out at the patches/-absent guard with the documented message pointing at the standalone repo and the vcpkg port.
• cmake -S tts-cpp -B build (no flags) errors at find_package(ggml CONFIG REQUIRED) with the reviewer-asked message pointing at ggml-speech.
• tts-cpp/test/ gtest suite passes on Linux (CPU + Vulkan), macOS (CPU + Metal), Windows (CPU; Metal/Vulkan N/A).
• Turbo end-to-end synth on a short English sentence with reference-voice cloning matches the tts-cpp/README.md performance table (Vulkan RTX 5090, Metal M3 Ultra, CPU Ryzen 9 9950X, CPU M3 Ultra NEON).
• Multilingual end-to-end synth on the documented Spanish prompt + built-in voice, seed 42, matches the --cfm-steps 7 and default-cfm-steps rows on Metal M3 Ultra and M4.
• German seed-42 reproduction (the worst case for the 3-identical-token bug from 0b44674): no trailing-silence-then-hallucination tail; output ends cleanly at sentence end.
• MTL leading-syllable regression (db87f42): "Hello from the multilingual" no longer renders as "lo from the multilingual" on the addon path.
• Built-in voice (apply_trim_fade gate from 942686d): "El", "A", "Hello" first words preserved on the chatterbox-mtl built-in path; reference-audio path unchanged.
• find_package(tts-cpp CONFIG REQUIRED) from a downstream CMake project resolves the static archive plus the OpenMP transitive dep with the COMPONENTS CXX scope (04b87ea), on Linux, macOS, and Windows MSVC + clang-cl.
• Engine::backend_device() on Vulkan/CUDA/Metal returns BackendDevice::GPU; on CPU returns BackendDevice::CPU. Matches parakeet::Engine::backend_device() shape.
• parakeet-cpp/ starts_word signal: streaming consumer rebuild of "see if" stays as "see if"; chunk-boundary split "pun" + "ctuation" rejoins as "punctuation"; default-true callers unchanged.
• Upstream whisper.cpp build below the new top-level README section is unaffected (existing make / cmake builds and tests pass identically to master).

Related

• tts-cpp/ source-of-truth: github.com/gianni-cor/chatterbox.cpp.
• parakeet-cpp/ source-of-truth: standalone parakeet.cpp repo.
• ggml consumption: qvac-ext-ggml/speech branch via the ggml-speech vcpkg port.
• Downstream consumers: qvac3/packages/tts-ggml, qvac3/packages/parakeet-ggml.
• Pre-merge port-version policy: amend qvac-registry-vcpkg/ports/tts-cpp port-version 0 in place; the actual port bump lands in qvac-registry-vcpkg in a follow-up after this PR merges to master.