fix(tui_gateway): event-driven delivery for notify_on_complete + watch_patterns

[liftaris]

What does this PR do?

Makes terminal(background=true, notify_on_complete=true) and watch_patterns deliver their notifications to TUI sessions. Currently they work everywhere except the TUI: events produced by tools.process_registry.completion_queue are drained by cli.py and gateway/run.py between agent turns, but tui_gateway/server.py has no consumer — events pile up in the queue and the agent never takes a follow-up turn on them.

The fix adds a router thread that blocks on completion_queue.get() and hands each event to a short-lived worker from a bounded ThreadPoolExecutor. Workers wait on a new per-session threading.Event (idle) before synthesizing a prompt.submit that re-enters the normal turn dispatch. idle is seeded at session construction and mirrored onto the existing running flag at every transition via a single _set_session_running() helper. If a goal-continuation turn or a real user prompt wins the race to the slot (4009 session busy), the dispatcher re-waits on the next idle edge and retries, bounded by the same overall 300s deadline. Entirely event-driven — every wait is on a threading.Event or a blocking queue.get(), no polling.

Message text matches gateway/run.py::_format_gateway_process_notification and cli.py::_format_process_notification, so the agent sees the same prompt regardless of transport.

Builds on and credits #15329 by @MestreY0d4-Uninter, whose PR established the right high-level shape but was closed for lack of maintainer signal with an explicit invitation to rebuild. This rebuild swaps the 100ms session["running"] polling for edge-triggered threading.Event, swaps the single serial worker for a bounded per-event pool (so a busy session can no longer head-of-line-block an idle sibling), and adds a 300s idle-timeout circuit breaker.

Related Issue

Fixes #15248

(Does not attempt #10760 — that's the API server / WebUI variant with a different root cause. Happy to file a follow-up once this lands.)

Type of Change

• 🐛 Bug fix (non-breaking change that fixes an issue)

Changes Made

• tui_gateway/server.py (+293 / -6):
  • Module-top: _notify_pool ThreadPoolExecutor, _notify_router_thread singleton, _NOTIFY_IDLE_TIMEOUT_SECONDS (all three env-configurable)
  • New helpers: _format_process_notification, _session_for_process_event, _dispatch_process_notification, _process_notification_router, _ensure_notification_router, _set_session_running
  • session["idle"] seeded in session.create and _init_session
  • 6 running transition sites now route through _set_session_running(session, ...)
  • Router started lazily on first session init
• tests/test_tui_gateway_server.py (+536): 11 new tests

How to Test

Manual

hermes --tui
# in the TUI composer:
> run `sleep 3 && echo done` in the background with notify_on_complete=true

Before: process completes, nothing happens, user has to manually ask.
After: ~3s later a [SYSTEM: Background process proc_... completed (exit code 0). ...] prompt appears and the agent takes a new turn on it.

Automated

./scripts/run_tests.sh tests/test_tui_gateway_server.py tests/tools/test_notify_on_complete.py tests/tools/test_watch_patterns.py
# 235 passed in 1.53s

New tests cover: event-edge contract (dispatcher provably blocked on idle.wait() until the edge fires), 4009-busy retry, session-close-during-wait, concurrent-sessions-no-HoL, orphaned events, already-consumed short-circuit, idle timeout, formatter parity, synthetic request ID, the _set_session_running invariant, and an integration test that puts a real event onto completion_queue and observes the synthesized prompt.

Platforms tested

• Linux (Manjaro, Python 3.11)

Checklist

Code

• I've read the Contributing Guide
• My commit messages follow Conventional Commits (fix(tui_gateway): …)
• I searched for existing PRs — fix(tui): deliver notify_on_complete process notifications to TUI sessions #15329 is the prior attempt, closed by its author inviting this rebuild
• My PR contains only changes related to this fix (no unrelated commits)
• I've run ./scripts/run_tests.sh and the target suites pass (235/235)
• I've added tests for my changes (11 new)
• I've tested on my platform: Linux (Manjaro, Python 3.11)

Documentation & Housekeeping

• I've updated relevant documentation — N/A (no user-facing config or schema changes; behavior matches existing CLI/gateway paths)
• I've updated cli-config.yaml.example if I added/changed config keys — N/A (new env vars are opt-in tuning knobs, sensible defaults)
• I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows — N/A
• I've considered cross-platform impact — threading primitives and queue.Queue are portable; no Unix-only syscalls introduced
• I've updated tool descriptions/schemas if I changed tool behavior — N/A (no tool surface changes; only completes an existing contract)

Screenshots / Logs

Can attach a before/after ~/.hermes/logs/agent.log snippet on request.

[MestreY0d4-Uninter]

Thanks for the credit and for picking this up, @liftaris! The switch to threading.Event and the bounded per-event pool address the exact bottlenecks I was wrestling with in #15329. The 300s idle timeout is a nice addition I hadn't considered.

If you need a second pair of eyes on the threading logic or the event-driven flow once the CI stabilizes, let me know. Happy to review.

[teknium1]

Thanks for the careful TUI delivery work. An automated hermes-sweeper review found that the requested behavior is already implemented on current main.

• d5416284f11ccbc735c8357f0ab35ce5f683ccc3 (fix(tui): autonomous background process completion notifications) added the TUI completion-queue poller and is included in v2026.5.16.
• tui_gateway/server.py:8658 consumes process_registry.completion_queue; tui_gateway/server.py:8721-8751 formats notifications, emits TUI process status, and starts the follow-up agent turn.
• tui_gateway/server.py:1443-1445, 4720-4722, and 8863-8873 start that poller for TUI session initialization paths.
• tools/process_registry.py:2080-2106 supplies shared formatting for completion, watch_match, and watch_disabled events; tests/test_tui_gateway_server.py:7646-7711 covers completion delivery into an agent turn.

Closing as implemented on main.