Skip to content

fix(desktop): restore human barge-in over agent TTS in huddles - #6431

Merged
tlongwell-block merged 9 commits into
mainfrom
wren/huddle-barge-in-phase1
Aug 21, 2026
Merged

fix(desktop): restore human barge-in over agent TTS in huddles#6431
tlongwell-block merged 9 commits into
mainfrom
wren/huddle-barge-in-phase1

Conversation

@tlongwell-block

@tlongwell-block tlongwell-block commented Aug 20, 2026

Copy link
Copy Markdown
Collaborator

Problem

Humans cannot interrupt agents in huddles: agent TTS keeps playing over a talking human (reported by @tlongwell-block, 2026-08-20). The full requirement: any human talking — local or remote, any input mode, any audio rig — must interrupt any agent on the huddle.

This is a restoration, not a new feature

Design

All floor state lives under the single PlaybackCoordinator lock — onset acceptance, epoch bump, synthesis invalidation, player replacement, and output lease are one committed transition. No cancel flag can be observed out of order with the playback state it describes.

  • Human floor: local + per-peer remote ownership with epoch invalidation. Onset cancels playback by queue replacement; late synthesis for a stale epoch cannot append or restart.
  • Local onset (VAD): on an isolated output route (all CoreAudio output-stream terminals report headphones), a confirmed short onset interrupts immediately — no echo path exists. On a coupled route (speakers, unknown, virtual, mixed), the restored 20-frame / 320 ms sustained-speech debounce discriminates a real human from speaker bleed; the deleted code's comment records that 80 ms was tried and false-triggered on laptop speakers. Route classification is queried fresh at each onset (never cached — default-device re-routing mid-huddle would strand a stale verdict).
  • Mic-open gate is per-frame: barge-in observes on any frame where the mic is actually open — pure VAD mode, or PTT with the mic manually unmuted (key-held frames defer to the shortcut's own cancel).
  • Remote onset: sustained non-DTX frames from a peer enter the same persistent floor (independent of whether playback is live — a human speaking while TTS is idle blocks late-arriving synthesis from starting over them). Release on sustained DTX/absence, peer departure, and recv-loop exit, with guards so a vanished peer cannot wedge the floor.
  • Output lease: accepted appends renew an Active lease; drain/cancel/onset start a 100 ms tail hangover (conservative against measured ~12 ms/~1 ms CoreAudio tails), so speaker-tail bleed in the just-drained window cannot self-trigger the coupled path.

Known limitations (phase 2 pointers)

  • Coupled-route mid-output barge-in pays the 320 ms debounce; a playback-reference echo discriminator would shorten it.
  • A speakers-rig participant's bleed can enter their mic and hold the floor for other machines (bounded by release debounce).
  • Non-macOS routes classify as coupled (fail-safe).

Verification

  • Full buzz-desktop --lib suite at head b0459ae4a: 2707 passed, 0 failed, 18 ignored (pinned cargo 1.95.0).
  • Exact CI recipe just desktop-tauri-clippy: PASS at head; base arm at merge-base b728a2af3 confirms the two #[allow(clippy::too_many_arguments)]s cover branch-caused threshold crossings (human_floor threading), not inherited noise.
  • Regression tests pin: 20-frame threshold + reset-on-gap, short-coupled rejection, sustained-coupled acceptance, coupled-idle acceptance, remote-idle delayed-TTS rejection, output-tail hangover boundary (during = rejected, after = accepted), isolated onset, per-frame mic-open gate truth table, PTT+manual-unmute sustained coupled acquisition.
  • LocalBargeIn::observe is covered as two joined halves (gate truth table in local_barge_in.rs, floor transition in tts_playback.rs); its body is a straight-line wrapper around a live CoreAudio query, left uninjected deliberately.
  • Coverage precision (mutation-verified): manual_open_ptt_sustained_speech_acquires_coupled_floor pins the gate → 20-frame debounce → acquire → floor-blocked chain on the coupled-idle cell. The live-output override leg is carried by sustained_coupled_speech_overrides_live_output_suppression (tts_playback.rs); the joiner's sustained_coupled argument is not load-bearing there (flipping it to false leaves the test green, while shortening the debounce by one frame turns it red).
  • Live arms in progress: pre-regression build b29c8cdaa^ staged to confirm the deleted mechanism worked; two-endpoint remote-leg test pending a second human.

Commits

  1. fb681a5a9 — restore human barge-in (coordinator floor, lease, route isolation, remote floor, 320 ms coupled debounce)
  2. b4265418e — enable barge-in for manually opened mics (per-frame gate; closes the PTT-open-mic gap)
  3. 886489f2b — extract local barge-in policy module (file-size ratchet; also hoists the CoreAudio route query from per-frame to per-onset, named in the commit message)
  4. b0459ae4a — two targeted clippy allows for the widened worker signatures

Credits

Built by Wren. Regression archaeology and the PTT-open-mic gap by Dawn (who also killed her own first fix as vacuous and caught a clippy blocker before it hit CI). Review blockers (coordinator serialization, idle-onset floors, output lease) by Mari. Live rig verification by Max. Coordination and verification by Eva. Opened by Eva with Tyler's explicit direction; commits carry agent trailers.

Wren added 4 commits August 20, 2026 15:40
Restore local VAD interruption through a coordinator-owned human floor, preserve remote floor ownership across turns, and reject stale TTS work after a human takes the floor. Use route-aware onset policy with the prior 320 ms coupled-output fallback and a bounded output-tail lease.

Co-authored-by: Wren <5217c5c2f7bfb4333e46d17c98a9255a52dadee18dcd43a43536b95e6776dfa0@buzz.block.builderlab.xyz>
Signed-off-by: Wren <5217c5c2f7bfb4333e46d17c98a9255a52dadee18dcd43a43536b95e6776dfa0@buzz.block.builderlab.xyz>
Read manual mic and PTT state per frame so push-to-talk users who open the mic receive the same VAD interruption policy without double-cancelling while the shortcut is held.

Co-authored-by: Wren <5217c5c2f7bfb4333e46d17c98a9255a52dadee18dcd43a43536b95e6776dfa0@buzz.block.builderlab.xyz>
Signed-off-by: Wren <5217c5c2f7bfb4333e46d17c98a9255a52dadee18dcd43a43536b95e6776dfa0@buzz.block.builderlab.xyz>
Move local barge-in policy and coverage into its own module to satisfy the file-size ratchet. Query output-route isolation only at a qualified onset instead of enumerating CoreAudio devices on every open-mic frame.

Co-authored-by: Wren <5217c5c2f7bfb4333e46d17c98a9255a52dadee18dcd43a43536b95e6776dfa0@buzz.block.builderlab.xyz>
Signed-off-by: Wren <5217c5c2f7bfb4333e46d17c98a9255a52dadee18dcd43a43536b95e6776dfa0@buzz.block.builderlab.xyz>
Co-authored-by: Wren <5217c5c2f7bfb4333e46d17c98a9255a52dadee18dcd43a43536b95e6776dfa0@buzz.block.builderlab.xyz>
Signed-off-by: Wren <5217c5c2f7bfb4333e46d17c98a9255a52dadee18dcd43a43536b95e6776dfa0@buzz.block.builderlab.xyz>
@tlongwell-block
tlongwell-block requested a review from a team as a code owner August 20, 2026 20:19
Wren added 5 commits August 20, 2026 16:28
Co-authored-by: Wren <5217c5c2f7bfb4333e46d17c98a9255a52dadee18dcd43a43536b95e6776dfa0@buzz.block.builderlab.xyz>
Signed-off-by: Wren <5217c5c2f7bfb4333e46d17c98a9255a52dadee18dcd43a43536b95e6776dfa0@buzz.block.builderlab.xyz>
Co-authored-by: Wren <5217c5c2f7bfb4333e46d17c98a9255a52dadee18dcd43a43536b95e6776dfa0@buzz.block.builderlab.xyz>
Signed-off-by: Wren <5217c5c2f7bfb4333e46d17c98a9255a52dadee18dcd43a43536b95e6776dfa0@buzz.block.builderlab.xyz>
Co-authored-by: Wren <5217c5c2f7bfb4333e46d17c98a9255a52dadee18dcd43a43536b95e6776dfa0@buzz.block.builderlab.xyz>
Signed-off-by: Wren <5217c5c2f7bfb4333e46d17c98a9255a52dadee18dcd43a43536b95e6776dfa0@buzz.block.builderlab.xyz>
Co-authored-by: Wren <5217c5c2f7bfb4333e46d17c98a9255a52dadee18dcd43a43536b95e6776dfa0@buzz.block.builderlab.xyz>
Signed-off-by: Wren <5217c5c2f7bfb4333e46d17c98a9255a52dadee18dcd43a43536b95e6776dfa0@buzz.block.builderlab.xyz>
Co-authored-by: Wren <5217c5c2f7bfb4333e46d17c98a9255a52dadee18dcd43a43536b95e6776dfa0@buzz.block.builderlab.xyz>
Signed-off-by: Wren <5217c5c2f7bfb4333e46d17c98a9255a52dadee18dcd43a43536b95e6776dfa0@buzz.block.builderlab.xyz>
@tlongwell-block
tlongwell-block merged commit 6039fed into main Aug 21, 2026
25 checks passed
@tlongwell-block
tlongwell-block deleted the wren/huddle-barge-in-phase1 branch August 21, 2026 16:51
wpfleger96 pushed a commit that referenced this pull request Aug 21, 2026
…c-agent-commit-identity

* origin/main:
  fix(desktop): restore human barge-in over agent TTS in huddles (#6431)

Signed-off-by: Duncan <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz>
salman1993 added a commit that referenced this pull request Aug 21, 2026
…uctions-prompt

* origin/main:
  fix(benchmarks): wait for scripted event delivery (#6487)
  Polish mobile channel navigation and message sends (#6488)
  Clarify huddle message destination (#6496)
  feat(archive): add observer-frame retention schema and gated DB adapter (#5719)
  fix(desktop): restore human barge-in over agent TTS in huddles (#6431)
  chore(release): release Buzz Desktop version 0.5.18 (#6489)
  fix(desktop): simplify duplicate agent provenance (#6401)
  test(benchmarks): expand Buzz-native dataset (#6448)
  fix(desktop): sender names in notifications + macOS click-through routing (#6427)
  docs: clarify two-layer moderation ownership (#6481)
  Fix mobile thread tail and iOS channel header (#6399)
  chore(deps): pin earshot below 1.2.0 pending a VAD threshold re-pick (#6392)
  polish(desktop): finish Projects navigation and context chrome (#6429)
  fix(desktop): clarify add agents channel action (#6374)
  Repair stale large channel roster snapshots (#6251)

Signed-off-by: Salman Mohammed <smohammed@squareup.com>
baxen pushed a commit that referenced this pull request Aug 21, 2026
* origin/main: (88 commits)
  docs(nips): comprehensive NIP-FI — core + claimable profiles (EDGE/LIFECYCLE/DELEG/CONF) (#5946)
  fix(benchmarks): wait for scripted event delivery (#6487)
  Polish mobile channel navigation and message sends (#6488)
  Clarify huddle message destination (#6496)
  feat(archive): add observer-frame retention schema and gated DB adapter (#5719)
  fix(desktop): restore human barge-in over agent TTS in huddles (#6431)
  chore(release): release Buzz Desktop version 0.5.18 (#6489)
  fix(desktop): simplify duplicate agent provenance (#6401)
  test(benchmarks): expand Buzz-native dataset (#6448)
  fix(desktop): sender names in notifications + macOS click-through routing (#6427)
  docs: clarify two-layer moderation ownership (#6481)
  Fix mobile thread tail and iOS channel header (#6399)
  chore(deps): pin earshot below 1.2.0 pending a VAD threshold re-pick (#6392)
  polish(desktop): finish Projects navigation and context chrome (#6429)
  fix(desktop): clarify add agents channel action (#6374)
  Repair stale large channel roster snapshots (#6251)
  feat(desktop-messages): show compact Buzz link metadata (#6252)
  feat(workflows): reply in-thread from send_message action (#6178)
  perf(desktop): split discover_acp_providers into cheap and forced paths (#6330)
  fix(desktop): restore recent channel sorting (#6402)
  ...

Co-authored-by: Bradley Axen <baxen@squareup.com>
Signed-off-by: Bradley Axen <baxen@squareup.com>

# Conflicts:
#	crates/buzz-acp/src/base_prompt.md
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant