Skip to content

perf(tui_gateway): bound heavy SessionDB reads off the event loop - #441

Merged
Kyzcreig merged 1 commit into
mainfrom
land/session-db-heavy-gate
Jul 26, 2026
Merged

Kyzcreig merged 1 commit into
mainfrom
land/session-db-heavy-gate

Conversation

@Kyzcreig

Copy link
Copy Markdown
Collaborator

Symptom

A single Discord voice message posted its 🎙️ transcript echo twice — two identical messages 4.3s apart. The gateway log confirms the same .ogg was transcribed twice:

09:27:37,033 INFO tools.transcription_tools: Transcribed audio_b8e37bff145d.ogg via local whisper (base, lang=en, 66.3s audio)
09:27:41,333 INFO tools.transcription_tools: Transcribed audio_b8e37bff145d.ogg via local whisper (base, lang=en, 66.3s audio)

Root cause

Echo dedupe was keyed on Python object identity. _gateway_pending_stt_echo_sent is set via setattr on a MessageEvent, but one platform message reaches the echo path as two distinct event objects:

  1. The busy/interrupt path echoes the inbound object.
  2. The drain path later prepares the pending-slot object — which merge_pending_message_event may have replaced, and whose cached STT attrs _invalidate_pending_stt_cache deliberately clears.

Neither guard survives that hand-off.

There was also a second, independent emitter: _prepare_inbound_message_text sent the echo inline with its own loop, never consulting the once-flag at all — so the drain leg re-posted even when the flag was intact.

Fix

Whole bug class, not just the one site — all 6 echo call sites now funnel through the single deduped helper:

  • Dedupe on the durable identity of the message — platform message_id when present, else the concrete downloaded audio paths — held in a bounded LRU on the runner, so the guard survives across event objects.
  • Route _prepare_inbound_message_text through _echo_pending_stt_transcripts_once instead of sending inline.

Test-gate honesty

New tests/gateway/test_stt_transcript_echo_dedupe.py asserts the invariant (one platform message → exactly one echo), plus the negative cases: distinct messages still each echo, a fresh voice message still echoes, quiet-STT stays silent, and the LRU stays bounded. 4 of the 7 fail without this fix.

test_all_gateway_transcript_echo_sends_are_gated was rewritten from a fixed 12-line proximity window to an AST-resolved enclosing function check. The config gate is a property of the function; the window check broke as soon as unrelated logic landed between the gate and the send. Verified by mutation — injecting an ungated echo site still fails the test.

Verification

  • Full tests/gateway/ suite run.
  • The 3 remaining failures (test_readiness, test_systemd_notify, test_api_server::test_health_detailed_returns_ok) reproduce identically on clean main in a separate baseline worktree — pre-existing and unrelated (test_systemd_notify needs Linux abstract sockets).

Share a loop-local async SessionDB heavy-read gate between dashboard REST reads and WebSocket session-list RPCs. The gate reads dashboard.heavy_read_max_concurrency lazily, queues with a bounded wait, logs queue_wait, exposes shed stats, and returns retryable backend-busy errors on saturation.\n\nVerified:\n- scripts/run_tests.sh tests/test_web_server_sessiondb_eventloop.py -v (14 passed)\n- scripts/run_tests.sh -j 16 tests/tui_gateway tests/test_tui_gateway*.py tests/test_web_server*.py tests/hermes_cli/test_web_server*.py (1133 passed)\n- git diff --check\n- /Users/alexgierczyk/.hermes/hermes-agent/venv/bin/python -m py_compile hermes_cli/session_db_heavy_gate.py hermes_cli/web_server.py tui_gateway/server.py tui_gateway/ws.py\n\nNot pushed.
@Kyzcreig

Copy link
Copy Markdown
Collaborator Author

FleetReview

Confidence: 2/5

Findings


FleetReview provenance · models: B=gpt-5.6-sol, C=claude-code-opus-4-8, F=gpt-5.6-sol, G=grok-4.5 · cost: $9.48 · duration: 20m 55s · rounds: 1 · files examined: 6

@Kyzcreig
Kyzcreig merged commit 0986938 into main Jul 26, 2026
40 checks passed
@Kyzcreig
Kyzcreig deleted the land/session-db-heavy-gate branch July 26, 2026 13:27
@Kyzcreig
Kyzcreig restored the land/session-db-heavy-gate branch September 21, 2026 10:32
Kyzcreig added a commit that referenced this pull request Sep 25, 2026
… 3 premises re-checked (t_2a1bd9cd)

#115 DROP->KEEP (swiftui-skills skill is live), #215/#441 DROP->UNRESOLVED
(tui_gateway/ws.py still consumes the gate; dashboard live), nopr:8a8b81638c
UPSTREAM->UNRESOLVED (leak needs the fork-only auto-attach detector).
Branches: 5 revert branches built + targeted pytest green; upstream-617 and
upstream-466 hand-ported onto upstream/main with tests green.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant