QVAC-17872 [TTS GGML] Optimize cpp backend multilingual model for Vulkan

[Zbig9000]

multilingual on Vulkan, RTX 5090, warm-state seg 2..6

Metric	upstream/main baseline	this PR (3 commits)	Δ
S3GEN_INFER	169.9 ms	140.8 ms	−29.1 ms (−17.1 %)
cfm_total	132.5 ms	118.7 ms	−13.8 ms (−10.4 %)
cfm_step0	24.1 ms	13.2 ms	−10.9 ms (−45.2 %)
Cold-start	~2.7 s	~250 ms	−2.4 s (−91 %)

Bit-exact preserving on multilingual: locked MD5 invariants (single-shot c65d98f1…, 6-segment multi-synth 0b374c74…, Turbo single-shot 6219f433…) match byte-for-byte across 4 successive iterations of the test-first regression harness.

The biggest remaining single piece of S3GEN_INFER (~120 ms cfm_total) is the actual GPU CFM compute — not host-cacheable. Closing that requires shader-side work (e.g. tensor-core engagement via cooperative_matrix2); listed as a deferred follow-up below.

---

What this PR does

Round 1 — f6893b2 (squashed source port from the closed PR #1)

Component	What it is	Multilingual benefit
patches/ggml-vulkan-pipeline-cache.patch (199 lines, NEW)	Persistent VkPipelineCache keyed by <vendorID>-<deviceID>-<driverVersion>, opt-in via GGML_VK_PIPELINE_CACHE_DIR.	Cold-start ~2.7 s → ~250 ms. Mesa / Adreno / Mali biggest target (no driver-side binary cache).
patches/ggml-vulkan-eager-cache-save.patch (104 lines, NEW)	Crash-safe pipeline-cache flush. Skips disk write when cache size unchanged.	Stacks on the first patch; perf-neutral on warm runs.
g_cfm_estimator_cache (process-wide global)	cfm_estimator_cache was the last graph-builder still local-scope in s3gen_synthesize_to_wav — every synth call paid the full ~50 ms graph rebuild cost.	cache.b2 flag handles Turbo (batch=1) ↔ multilingual (batch=2 CFG) mode flip transparently. cfm_step0 24 → 13 ms.
g_time_mlp_results + g_time_emb_results	Two-layer cache by t-value (Turbo + multilingual) and (t, r) pair (Turbo only).	Multilingual fires more: 10 distinct t-values per inference (cosine schedule) vs Turbo's 2-3. 9–19 graph submissions / inf → 0.
g_weight_cpu_mirror (cached_cpu_weights_f32)	CPU mirror of flow/input_embedding + flow/spk_embed_affine/{w,b}.	Multilingual is bigger: ~28 MB embedding vs Turbo's ~13 MB. Saves the device→host transfer on every synth.
3 HiFT ggml_cont removals	conv_transpose_1d_f32 exit, ISTFT y_trim exit, f0_predictor xp permute.	Code-quality + future-proofing; CONT total in HiFT is only 0.13 % of HiFT runtime per the perf logger.
G2 dump-script gap closure (scripts/dump-s3gen-reference.py + 1-line test_s3gen.cpp fix)	regress-tensor-compare.sh was aborting at stage G2 with cannot open cfm_concat.npy.	Full Python ↔ C++ stage-compare pipeline now runs end-to-end through G2 / G3 / G4 / H1 / H3 / H4 / H5; max rel err 7.92e-3 on STFT (PyTorch FFT vs hand-built DFT, expected), max ≤ 4.7e-5 elsewhere; final waveform max_abs = 8.20e-08.

Verification — 5084ee4 (PROGRESS.md only)

After QVAC-18422 §3.34's converter shipped chatterbox-s3gen-mtl-q4_0.gguf (788 MB) + chatterbox-t3-mtl-q4_0.gguf (345 MB), I built a fresh upstream/multilingual_merged HEAD baseline (no Vulkan patches) and ran identical multilingual synthesis on both:

• Bit-exact — single-shot MD5 c65d98f15a59b8fe9cad98e46eb3fb30 and 6-segment multi-synth MD5 0b374c7474895a3387b9f1df10b3c1b8 match between this PR and the upstream baseline byte-for-byte. These are the first locked multilingual F32 invariants on the multilingual_merged/main Vulkan base.
• Multilingual perf (n=15 warm-state samples per build): −9.5 % S3GEN_INFER, −13.4 % cfm_total, −47.7 % cfm_step0 vs upstream baseline (round-1-only measurement).

Round 2 — d5c261c (multilingual-targeted host-side caches)

Targets the per-synth host-CPU overhead that round 1 didn't address. All seven caches sit alongside the round-1 caches; same destroy()-before-ggml_backend_free discipline.

Cache	Keyed on	Purpose / multilingual benefit
g_encoder_graph_cache	T (encoder input length)	Full run_encoder graph + gallocator. Multilingual T~350+ → bigger encoder graph rebuild was being repeated every synth.
g_hift_graph_cache (+ g_hift_inv_alpha_entries)	pack(T_mel, T_stft)	Full run_hift_decode graph + gallocator. Parallel (graph-input-name, source-tensor-ptr) metadata lets cache hits re-feed every alpha-input slot from g_inv_alpha_results without rebuilding. HiFT audio length scales with prompt length → multilingual is the biggest beneficiary.
g_f0_graph_cache	T_mel	Full run_f0_predictor graph + gallocator.
g_pos_emb_results (cached_pos_emb)	pack(T, D)	compute_pos_emb output. Pure CPU compute (~T × D × 5 trig ops); fired twice per encoder run (T and 2T). Multilingual T=350+ at D=512 was the dominant scaffolding cost.
g_inv_alpha_results (cached_inv_alpha)	ggml_tensor *	HiFT calls invert_alpha_cpu ~72× per synth (12 ResBlocks × 6 alpha tensors); each is a tensor_get + per-element reciprocal. Alpha tensors are constant for the model lifetime.
g_hann_window_cache / g_istft_kernel_cache	n_fft	Pure functions of n_fft (constant 16).
g_window_sum_cache (cached_window_sum)	pack(n_fft, hop, T_stft)	Stable across same-shape synth calls.

A new graph_cache struct (used by encoder / HiFT / F0) and a pack_hift_key helper centralise the explicit destroy()-on-teardown pattern so future per-stage caches plug in with one struct + one mutex acquisition. Destroy path is unified into s3gen_release_synth_caches() (renames the old g_cfm_estimator_cache_destroy()), called from s3gen_model_cache_release / cache-miss / s3gen_unload.

Within-process win on top of round 1 (warm-state seg 2..6, n=5 per build):

Metric	round 1 alone	+ round 2	Δ
S3GEN_INFER	159.8 ms	140.8 ms	−19.0 ms (−11.9 %)
hift_total	17.96 ms	16.30 ms	−1.7 ms (−9.4 %)

Negative result documented (bug caught and fixed during dev)

First implementation of the HiFT cache hung indefinitely on the very first synth call. Root cause: the alpha-input refresh loop held g_synth_caches_mu while calling cached_inv_alpha, which itself takes the same mutex internally — classic re-entrant deadlock. Fix: snapshot g_hift_inv_alpha_entries under the mutex into a local vector, then iterate without the lock. Inline comment in run_hift_decode documents the rule for future investigators: never hold a cache-state mutex while calling any other cached_* helper.

---

Bit-exactness — locked invariants

Locked F32 MD5s on RTX 5090 + NVIDIA 590.48 + Vulkan 1.3.275:

Test	Locked MD5	Verification
Multilingual single-shot (seed 42)	c65d98f15a59b8fe9cad98e46eb3fb30	✓ across 4 iters of regress-mtl-vk.sh verify
Multilingual 6-segment multi-synth	0b374c7474895a3387b9f1df10b3c1b8	✓ across 4 iters
Turbo single-shot (seed 42)	6219f4338b1b4fb9dc60481216153b49	✓ across 4 iters

The test-first regression harness bench-logs-vk-mtl/regress-mtl-vk.sh (in the qvac monorepo, out-of-tree) locks a snapshot before any change then verifys after every cache addition. Every round-2 cache passed verify immediately after addition.

Tensor-level Python ↔ C++ stage compare (added via the round-1 G2 fix) runs end-to-end through G2 / G3 / G4 / H1 / H3 / H4 / H5 with max rel err 7.92e-3 on STFT (expected — PyTorch torch.stft vs hand-built DFT-via-conv1d), max ≤ 4.7e-5 everywhere else, final waveform max_abs = 8.20e-08.

---

How to validate

cd <chatterbox.cpp>

# 1. Apply the Metal + OpenCL + 2 new Vulkan patches
bash scripts/setup-ggml.sh

# 2. Build (Vulkan)
cmake -S . -B build-vk -DCMAKE_BUILD_TYPE=Release -DGGML_VULKAN=ON
cmake --build build-vk -j --target tts-cli test-s3gen

# 3. Bit-exactness (locked MD5 invariants — multilingual + Turbo)
#    Snapshot lives in inputFilesForAI/qvac-17872-findings/bench-logs-vk-mtl/
#    in the qvac monorepo; the harness verifies all 3 invariants in one shot.
bash inputFilesForAI/qvac-17872-findings/bench-logs-vk-mtl/regress-mtl-vk.sh \
     build-vk verify-pr8 verify
# Expected:
#   PASS  multilingual single-shot  md5=c65d98f15a59b8fe9cad98e46eb3fb30
#   PASS  multilingual auto-split    md5=0b374c7474895a3387b9f1df10b3c1b8
#   PASS  turbo single-shot          md5=6219f4338b1b4fb9dc60481216153b49

# 4. Cold-start (round-1 VkPipelineCache patch)
rm -rf ~/.cache/ggml/vulkan
./build-vk/tts-cli ...   # first run: ~2.7 s cold
./build-vk/tts-cli ...   # second run: ~250 ms (ggml cache warm)

# 5. Multilingual perf — 6-segment auto-split (within-process warm caches)
./build-vk/tts-cli \
    --model models/chatterbox-t3-mtl-q4_0.gguf \
    --s3gen-gguf models/chatterbox-s3gen-mtl-q4_0.gguf \
    --language en \
    --text "Hello from ggml first synthesis. Second synthesis run here now. \
            Third sentence here. Fourth sentence runs too. Fifth sentence wraps." \
    --max-sentence-chars 32 --out /tmp/mtl.wav \
    --n-gpu-layers 99 --threads 4 --seed 42 --temp 0 --top-k 1 --verbose
# Expected segments 2..6 average:
#   S3GEN_INFER ~141 ms, cfm_total ~119 ms, cfm_step0 ~13 ms, hift_total ~16 ms
# Compare a fresh upstream/main baseline build via the same recipe; multilingual
# WAV must produce IDENTICAL md5sum (PR makes synthesis faster, not different).

The full reproduction recipe (including how to build the upstream baseline in a separate git worktree for comparison) is in PROGRESS.md §3.32 ► "Round 2 ► Reproduction".

---

Risk assessment

• Bit-exact preserving — verified across multilingual + Turbo invariants; verified across 4 successive iterations on the test-first harness.
• Default behaviour unchanged unless explicit env vars are set:
  • Round-1 cache reads/writes from $XDG_CACHE_HOME/ggml/vulkan / $HOME/.cache/ggml/vulkan — opt-out via empty GGML_VK_PIPELINE_CACHE_DIR="".
  • All seven round-2 caches are always-on (no env var gate). They never change output, only avoid recomputing it.
• No GGUF format change — existing chatterbox-s3gen-{turbo,mtl-q4_0}.gguf work as-is.
• No public-API change — include/tts-cpp/chatterbox/*.h untouched.
• Two ggml-vulkan patches (round-1 + round-2) shipped under patches/, applied in scripts/setup-ggml.sh (same vendoring model as the existing Metal + OpenCL patches).
• Same cmake -DGGML_VULKAN=ON invocation as before — no new dependencies.
• Memory cap: every cache is bounded by the number of distinct shape keys it sees across the process lifetime (typically 1-2 entries each). Steady-state per-process overhead: ~280 MB total (the bulk is the three 64 MB graph arenas). Streaming sessions with many distinct chunk sizes can grow these caches; a future LRU bound is documented in PROGRESS.md §3.32 as a deferred follow-up.
• Teardown ordering: s3gen_release_synth_caches() runs before ggml_backend_free (same constraint as the pre-existing thread_local time_mlp_cache); registered via atexit() on first cache insertion + called explicitly from the cache-miss / s3gen_unload paths.

---

Files

PROGRESS.md                                ~+540 / -10  (§3.32 entry: round-1 + verification + round-2)
src/chatterbox_tts.cpp                     ~+625 / -98  (round-1 + round-HIFT + round-2 graph + scaffolding caches)
patches/ggml-vulkan-pipeline-cache.patch    +199         (NEW)
patches/ggml-vulkan-eager-cache-save.patch  +104         (NEW)
scripts/dump-s3gen-reference.py             +65
scripts/setup-ggml.sh                       +20 / -8     (applies the two new Vulkan patches)
patches/README.md                           +13 / -8     (documents the new patches)
src/test_s3gen.cpp                          +6           (G2 set_output(xc) fix)
                                            -----------
Total: 8 files, ~+1500 / −115, 3 commits on upstream/main.

CHANGELOG.md deliberately not added (per PR #1 review feedback) — the investigation entry lives in PROGRESS.md §3.32 instead.

The inputFilesForAI/qvac-17872-findings/{FINDINGS,PR_DESCRIPTION,bench-logs-vk-mtl/}* companion docs stay in the qvac monorepo (out-of-tree) — same arrangement as the QVAC-18422 sister PR.

---

Deferred follow-ups (separate PRs)

Candidate	Estimated multilingual win	Why deferred
C1 — F16 CFM matmul weights (opt-in CHATTERBOX_F16_CFM)	~125 MB device-memory + bandwidth-bound mobile win	Multilingual CFM is ~6× larger than Turbo, so this is the bandwidth-bound mobile lever. multilingual_merged's load_s3gen_gguf uses ggml_dup_tensor + ggml_backend_alloc_ctx_tensors — different from the main-base path our F16 conversion was written against. ~100 lines of adaptation + new locked MD5 baselines (NVIDIA + AMD, F32 + F16).
cooperative_matrix2 (CM2) tensor-core engagement for narrow CFM matmuls	−8.6 % cfm_total measured in the prior main-base CM2 Tier-3 close-out	Requires LunarG SDK 1.4.341.1 + glslc 2026.1; politically blocked behind a cmake flag pending project-wide baseline-set sign-off. See inputFilesForAI/qvac-17872-findings/FINDINGS_ROUND_CM2.md.
Round-4 / 6 Q/K/V batched matmul fusion composition with multilingual_merged's zero-cont strided 3D Q/K/V views (849507a)	~1.3 ms RTX 5090 + larger on bandwidth-starved targets	Pick-one-approach decision deferred; needs Vulkan flash_attn_ext stride-tolerance verification.
T3 step-graph cache (multilingual fires build_step_graph_mtl 2× per token via CFG)	~12 % T3 wall reduction on synth #2+ in long-running processes	Covered by QVAC-18422 §3.35 on the CPU branch; same pattern would port.
Mobile validation (Adreno / Mali / Apple)	unknown — biggest remaining evidence gap	Hardware-bound. AMD/RADV proxy refuted the original main-base mobile-bandwidth projections on rounds 2 / 3 / 5 / 6 / C1, so real mobile runs would either confirm or force revision.
CI integration of regress-mtl-vk.sh + the existing regress-c1.sh / regress-amd.sh / regress-tensor-compare.sh	n/a — test-infra	Now unblocked since round-1 closed the G2 gap. Catches future regressions like the deadlock that surfaced (and was fixed) during round-2 dev.