Skip to content

Fix use-after-free in async TTS progress callbacks - #3781

Merged
csukuangfj merged 2 commits into
k2-fsa:masterfrom
kecoco16:fix-tts-async-callback-use-after-free
Jul 23, 2026
Merged

csukuangfj merged 2 commits into
k2-fsa:masterfrom
kecoco16:fix-tts-async-callback-use-after-free

Conversation

@kecoco16

@kecoco16 kecoco16 commented Jul 22, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #3780

Problem

OfflineTts.generateAsync() with an onProgress callback can deterministically abort the process during multi-chunk generation (full stack and minimal reproduction in #3780). Root cause: the TSFN queue holds raw pointers to TtsCallbackData objects owned by the AsyncWorker's data_list_, and nothing orders queue drain before the worker destructor frees them.

Changes

  • Give each callback chunk single-owner RAII cleanup.
  • Remove the global callback mutex and manual pending-data lists.
  • Use per-generation cancellation state.
  • Add a bounded TSFN queue with backpressure (the producer runs on a worker thread, so a full queue blocks synthesis, never the event loop).
  • Handle napi_closing without accessing the TSFN again.
  • Drain queued callbacks before settling the generation promise — signaled by a FIFO done-sentinel processed on the main thread; TSFN finalizers never run JS (unsafe during environment teardown).
  • Reject the promise when onProgress throws.
  • Preserve best-effort cancellation semantics for already-generated chunks.
  • Cover both the legacy and generationConfig async workers.
  • Cover external and copied output buffers.

The file is shared with the HarmonyOS binding via symlink, so both platforms get the fix.

Behavior change

A throwing onProgress previously invoked undefined behavior (the exception unwound through the N-API boundary). It now cancels the generation (best-effort) and the promise rejects with an error containing the thrown callback message. Exceptions never cross the N-API boundary, in both node-addon-api exception modes.

Testing

Added test_tts_async_callback_stress.js, covering:

  • Repeated async generation.
  • Both async worker paths.
  • Cancellation from onProgress (one callback total; no callbacks after cancellation).
  • Throwing progress callbacks (promise rejects with an error containing the thrown callback message; nothing reaches uncaughtException).
  • Concurrent generations with isolated cancellation.
  • enableExternalBuffer: false.
  • Steady-state RSS reporting.

The test is included in the existing Node.js addon CI script using the Piper English model already downloaded by that workflow.

Tested locally with:

node test_tts_async_callback_stress.js ./vits-piper-en_GB-cori-medium 30

Summary by CodeRabbit

Summary by CodeRabbit

  • Bug Fixes

    • Improved reliability of async offline text-to-speech completion.
    • Enhanced cancellation and error handling so progress updates and final promise resolution behave consistently.
    • Improved stability when running multiple TTS generations concurrently, without disrupting progress callbacks.
  • Tests

    • Added/expanded a stress test covering repeated, concurrent, cancelled, and throwing async TTS progress-callback scenarios, plus audio/buffering behavior checks.
    • Extended CI coverage by running the async callback stress test for an additional TTS model.

@gemini-code-assist

Copy link
Copy Markdown

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

@dosubot dosubot Bot added the size:L This PR changes 100-499 lines, ignoring generated files. label Jul 22, 2026
@coderabbitai

coderabbitai Bot commented Jul 22, 2026 •

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 074cd4fc-8328-4895-803f-61f9298002d3

📥 Commits

Reviewing files that changed from the base of the PR and between 6c65d9a and 2b9b857.

📒 Files selected for processing (2)
  • harmony-os/SherpaOnnxHar/sherpa_onnx/src/main/cpp/non-streaming-tts.cc
  • nodejs-addon-examples/test_tts_async_callback_stress.js
🚧 Files skipped from review as they are similar to previous changes (2)
  • nodejs-addon-examples/test_tts_async_callback_stress.js
  • harmony-os/SherpaOnnxHar/sherpa_onnx/src/main/cpp/non-streaming-tts.cc

📝 Walkthrough

Walkthrough

Async offline TTS progress handling now coordinates generation and callback-queue completion through shared settlement state, bounded TSFN queues, cancellation/error propagation, and done sentinels. A Node.js stress test covers sequential, cancellation, throwing, concurrent, RSS, and buffer-marshalling scenarios.

Changes

Async TTS callback lifecycle

Layer / File(s) Summary
Shared callback settlement state
harmony-os/SherpaOnnxHar/sherpa_onnx/src/main/cpp/non-streaming-tts.cc
Replaces per-chunk tracking and the global mutex with shared settlement state, heap-owned chunks, cancellation flags, callback error handling, and queue-drain tracking.
Worker and TSFN integration
harmony-os/SherpaOnnxHar/sherpa_onnx/src/main/cpp/non-streaming-tts.cc
Updates both async TTS workers and wrappers to use bounded TSFN queues, shared state, done sentinels, and explicit handling for TSFN closing or sentinel failures.
Callback stress validation
nodejs-addon-examples/test_tts_async_callback_stress.js, .github/scripts/test-nodejs-addon-npm.sh
Adds sequential, cancellation, throwing, concurrent, RSS, and buffer-marshalling tests, then runs the stress test in the npm TTS workflow.

Estimated code review effort: 4 (Complex) | ~45 minutes

Possibly related PRs

  • k2-fsa/sherpa-onnx#3133: Earlier async TTS progress and cancellation callback plumbing in the same implementation area.
  • k2-fsa/sherpa-onnx#3139: Added the generation-config JavaScript API path updated by this callback rework.
  • k2-fsa/sherpa-onnx#3419: Earlier mutex-based synchronization for async TTS chunk tracking replaced by this settlement and sentinel flow.

Sequence Diagram(s)

sequenceDiagram
  participant NodeTest
  participant OfflineTts
  participant TtsGenerateWorker
  participant TSFN
  NodeTest->>OfflineTts: generateAsync with onProgress
  OfflineTts->>TtsGenerateWorker: start generation
  TtsGenerateWorker->>TSFN: queue progress chunks
  TtsGenerateWorker->>TSFN: queue done sentinel
  TSFN->>OfflineTts: deliver callbacks and completion
  OfflineTts-->>NodeTest: resolve or reject promise
Loading
🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely summarizes the main fix in async TTS progress callbacks.
Linked Issues check ✅ Passed The changes address the reported use-after-free with safer callback ownership, drain-before-settle handling, and stress-test coverage.
Out of Scope Changes check ✅ Passed The modified code and test additions are directly related to the async TTS callback crash fix and its validation.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🧹 Nitpick comments (1)
nodejs-addon-examples/test_tts_async_callback_stress.js (1)

50-78: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Assert that progress callbacks actually fire in sequential() and copyBuffer().

Both functions track a chunk counter but never assert it's non-zero, unlike cancellation()/throwing()/concurrent() which strictly check callback counts. Since this PR's bug class is exactly "callbacks silently lost/UAF'd," a regression that drops callbacks entirely (0 chunks) would pass these two paths undetected.

  • nodejs-addon-examples/test_tts_async_callback_stress.js#L50-L78: after the loop (or per-iteration), assert chunks > 0 (and/or totalChunks >= iterations) before logging.
  • nodejs-addon-examples/test_tts_async_callback_stress.js#L198-L215: assert chunks > 0 alongside the existing r.samples.length check.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nodejs-addon-examples/test_tts_async_callback_stress.js` around lines 50 -
78, Strengthen callback coverage in sequential() by asserting each generation
produces at least one progress callback, or equivalently that totalChunks meets
the expected iteration count, before logging. Also update copyBuffer() at
nodejs-addon-examples/test_tts_async_callback_stress.js lines 198-215 to assert
chunks > 0 alongside the existing r.samples.length check.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@harmony-os/SherpaOnnxHar/sherpa_onnx/src/main/cpp/non-streaming-tts.cc`:
- Around line 1019-1032: Update both TtsGenerateWorker::OnOK
(non-streaming-tts.cc lines 1019-1032) and TtsGenerateWithConfigWorker::OnOK
(non-streaming-tts.cc lines 1237-1250) so the sentinel-failure finalization also
runs when tsfn_closing_ is true; preserve the existing error assignment and
force callbacks_drained before SettleOrFail so the promise settles without a
done sentinel.
- Around line 793-824: Update SettleIfReady to validate state->audio immediately
after generation and before either result-building branch. If it is null, reject
the promise and return without dereferencing or transferring the audio pointer;
preserve the existing external- and internal-buffer handling for valid audio.

---

Nitpick comments:
In `@nodejs-addon-examples/test_tts_async_callback_stress.js`:
- Around line 50-78: Strengthen callback coverage in sequential() by asserting
each generation produces at least one progress callback, or equivalently that
totalChunks meets the expected iteration count, before logging. Also update
copyBuffer() at nodejs-addon-examples/test_tts_async_callback_stress.js lines
198-215 to assert chunks > 0 alongside the existing r.samples.length check.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 632df1bc-96b2-4acf-af42-0468cc3a3907

📥 Commits

Reviewing files that changed from the base of the PR and between 2d8286d and 6c65d9a.

📒 Files selected for processing (3)
  • .github/scripts/test-nodejs-addon-npm.sh
  • harmony-os/SherpaOnnxHar/sherpa_onnx/src/main/cpp/non-streaming-tts.cc
  • nodejs-addon-examples/test_tts_async_callback_stress.js

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes a deterministic use-after-free crash in the Node.js addon’s OfflineTts.generateAsync() progress-callback path by reworking TSFN queue ownership and ensuring queued callbacks are drained before the generation promise is settled (shared with HarmonyOS via symlink).

Changes:

  • Refactors async TTS progress callback delivery to use single-owner heap chunks (RAII) and a bounded TSFN queue with backpressure, plus a FIFO “done” sentinel to guarantee drain-before-settle.
  • Adjusts cancellation/error behavior so callback cancellation/throws stop further callbacks and influence promise settlement deterministically.
  • Adds a new Node.js stress test for repeated, concurrent, cancelled, and throwing progress callbacks, and runs it in the existing CI script.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File Description
harmony-os/SherpaOnnxHar/sherpa_onnx/src/main/cpp/non-streaming-tts.cc Reworks async generation callback plumbing to fix lifetime ordering and settle only after callback drain.
nodejs-addon-examples/test_tts_async_callback_stress.js Adds a stress/regression test for async progress callbacks, cancellation, throwing callbacks, concurrency, and buffer-copy path.
.github/scripts/test-nodejs-addon-npm.sh Runs the new stress test in the Node.js addon CI script.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

return;
}

Napi::Object ans = Napi::Object::New(env);
Comment on lines +913 to +916
if (env.IsExceptionPending()) {
Napi::Error e = env.GetAndClearPendingException();
error_message = e.Message();
cancel_requested = true;
@@ -0,0 +1,230 @@
// Copyright (c) 2026 Kevin Castillo

@csukuangfj csukuangfj left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for your contribution!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:L This PR changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] Node.js async TTS progress callback causes a use-after-free crash

3 participants