diff --git a/PR_BODY_CORRECTED.md b/PR_BODY_CORRECTED.md new file mode 100644 index 0000000000000..324b141ca52d9 --- /dev/null +++ b/PR_BODY_CORRECTED.md @@ -0,0 +1,100 @@ +## Summary + +Coalesce multiple background process completions and watch events that share the same gateway route so the agent receives **one synthetic turn** instead of one turn per process — fixing the session flood documented in #70300. + +This is a two-pronged approach, each at the correct ownership seam: + +1. **Standard completions** — batched at `_run_process_watcher()` via `_enqueue_process_completion_notification()` with a fixed 100 ms batching window. Every completion (including singles) pays the full window — there is no zero-latency shortcut because true zero-latency requires limiting coalescing to work already available in the same loop tick, which is incompatible with a time-window batch. + +2. **`watch_match` / `watch_disabled` events** — coalesced at the post-turn drain via `_coalesce_and_inject_watch_events()`, grouped by event type and `session_key`. + +## Why this supersedes #70319 and #71898 + +| Feature | #70319 | #71898 | **This PR** | +|---------|:---:|:---:|:---:| +| Correct insertion point (watcher path) | no | yes | yes | +| Correct insertion point (drain path) | yes | no | yes | +| Coalesce standard completions | yes | yes | yes | +| Coalesce `watch_match` events | no | no | yes | +| Coalesce `watch_disabled` events | no | no | yes | +| Bounded batching delay (documented) | no | no | yes | +| Aggregate summary (X succeeded, Y failed) | no | no | yes | +| Elapsed time per process | no | no | yes | +| Dedup/retry per process | no | yes | yes | +| Flush during delivery | no | yes | yes | +| Duplicate primary -> try next identity | no | yes | yes | +| Formatter failure -> resolve all waiters | no | yes | yes | +| Lazy init for `object.__new__` tests | no | yes | yes | +| E2E tests with real watchers | no | yes | yes | +| Drain coalescing tests | no | no | yes | + +## Implementation + +### `gateway/run.py` (+327/-10) + +- **`GatewayRunner.__init__`**: Added batch state (`_completion_notification_batches`, `_completion_notification_batch_tasks`, lock, window=100ms) +- **`_enqueue_process_completion_notification()`**: Fans in concurrent completions sharing one route. All completions (including singles) are grouped for the full 100 ms batching window, then delivered as one synthetic turn. +- **`_flush_process_completion_batch()`**: Delivers one batch after the window. Handles CancelledError (resolves waiters as retryable), duplicate primary fallback, records sibling identities on success, resolves all waiters on failure. Every waiter resolves to an explicit result — `True` (delivered), `False` (retryable), or `None` (permanently unroutable / lifecycle duplicate). +- **`_format_coalesced_process_completions()`**: Compact format with per-process exit codes, elapsed time, bounded output (10 entries, 120-char command / 800-char output tails), aggregate summary over ALL entries ("3 completed: 2 succeeded, 1 failed"). No emoji — the LLM reads `exit_code=0` the same as any visual indicator. +- **`_coalesce_and_inject_watch_events()`**: Groups `watch_match`/`watch_disabled`/`completion` events by type+session_key at the post-turn drain. Single events pass through unchanged; multiple events produce one batched message. +- **`_format_gateway_process_notification()`**: Extended to handle standard `completion` events (previously returned `None`). +- **`_run_process_watcher()`**: Changed from direct `_deliver_completion_notification()` -> `_enqueue_process_completion_notification()`. +- **Post-turn drain**: Changed from per-event loop -> `_coalesce_and_inject_watch_events()`. +- **Shutdown drain**: `_stop_impl_body` cancels pending completion-batch flush tasks and awaits them (3-second deadline) BEFORE adapter teardown, so waiters are resolved as retryable while adapters are still available. + +### Example batched output + +``` +[IMPORTANT: 3 background processes completed for this session. +Treat these results as one completion batch and send at most one +consolidated user-facing response. + + proc_a: exit_code=0, reason=exited, 12.3s + success output here + + proc_b: exit_code=1, reason=error, 8.7s + error output here + + proc_c: exit_code=0, reason=exited, 5.1s + also ok + +Summary: 2 succeeded, 1 failed. +If a result does not change the current conclusion, absorb it silently.] +``` + +### Tests + +**`tests/gateway/test_completion_delivery.py`** (34 test functions): three concurrent watchers -> one batched turn, single completion uses original notification, different routes never coalesce, in-flight arrival schedules next flush, failed delivery retries all entries, coalesced success records every identity, duplicate primary -> fresh sibling delivered, format failure resolves all waiters, batch format includes exit codes + summary, entry truncation after 10, output truncation >800 chars, single completion bounded latency (<1s), batch summary counts failures in omitted tail, cancellation during batch window resolves waiters retryable, cancellation during delivery resolves waiters retryable. + +**`tests/gateway/test_background_process_notifications.py`** (35 test functions): single watch event pass-through, multiple watch_match coalesced, different session_keys not coalesced, mixed types coalesced separately, ID truncation after 5, empty list noop. + +``` +$ python3 -m pytest tests/gateway/test_completion_delivery.py tests/gateway/test_background_process_notifications.py -q +69 passed # scope: tests/gateway/test_completion_delivery.py + test_background_process_notifications.py +``` + +## Edge cases covered + +- **Flush during delivery**: Detaches batch before adapter I/O; new completions schedule the next flush +- **Duplicate primary**: Tries next batch identity when first entry is a lifecycle duplicate +- **Formatter failure**: Resolves all waiter futures with `False` so watchers retry +- **Lazy init**: Covers tests that construct `GatewayRunner` with `object.__new__` +- **None-safe key**: All batch key fields use `str(field or "")` +- **Cancellation safety**: CancelledError during batch window or adapter-blocked delivery resolves all waiters as retryable; no unresolved Future, no stale batch entry, no stale task key + +## Known limitations + +- **Sibling identity race (P5)**: The primary identity is claimed in-flight before delivery, but sibling identities in the batch are only recorded after the adapter returns. A concurrent replay of a sibling during this window can form a second synthetic turn. Mitigation tracked in [follow-up issue]. +- **No persistence layer**: Completions resolved as retryable during shutdown are not persisted — they disappear when the gateway process exits. This is consistent with the existing (pre-PR) behaviour for single-process notifications. +- **Tri-state return (`True`/`False`/`None`)**: `_deliver_completion_notification()` returns `Optional[bool]` with three overloaded meanings (delivered / retryable / dropped). A follow-up should replace this with an explicit `CompletionDisposition` enum to separate permanent-unroutable from temporary-unavailable and ensure every waiter path is explicit. + +## Review history + +- **Round 1** (`8840cc4`): Removed unimplemented threshold flush, fixed `zero-latency` claim -> bounded window, removed stale docstrings, addressed P1 lifecycle and P3 summary scope. +- **Round 2** (`3449790`): Fixed CancelledError handling (P1 cancellation hole), shutdown drain ordering, added omitted-tail summary regression test, removed remaining stale docstrings. +- **Round 3** (this update): Corrected PR body to match HEAD, fixed co-author trailer format, aligned shutdown comments with actual behaviour. + +--- + +Closes #70300 +Supersedes #70319 and #71898 diff --git a/gateway/run.py b/gateway/run.py index 37dfd9d92cde5..d7821a340729d 100644 --- a/gateway/run.py +++ b/gateway/run.py @@ -27,6 +27,7 @@ import asyncio import concurrent.futures import dataclasses +import enum import faulthandler import inspect import json @@ -3047,6 +3048,12 @@ def _format_gateway_process_notification(evt: dict) -> "str | None": from tools.process_registry import format_process_notification return format_process_notification(evt) + if evt_type == "completion" or evt_type is None: + # Standard process completions — also use the rich formatter for + # single-event pass-through and the coalesced watch-event drain. + from tools.process_registry import format_process_notification + return format_process_notification(evt) + return None @@ -3537,6 +3544,16 @@ def __init__(self, config: Optional[GatewayConfig] = None): self._completion_deliveries_delivered: "OrderedDict[tuple[str, str, object], None]" = OrderedDict() self._completion_delivery_retention = 2048 + # Coalesce concurrent process-completion notifications that share one + # gateway route so the agent receives one synthetic turn instead of + # one turn per process (#70300). Standard completions ride the + # per-watcher path; watch_match / watch_disabled events are coalesced + # at the post-turn drain. + self._completion_notification_batches: dict = {} + self._completion_notification_batch_tasks: dict = {} + self._completion_notification_batch_lock = threading.Lock() + self._completion_notification_batch_window = 0.1 + # Cache AIAgent instances per session to preserve prompt caching. # Without this, a new AIAgent is created per message, rebuilding the # system prompt (including memory) every turn — breaking prefix cache @@ -9907,6 +9924,39 @@ def _phase_elapsed() -> float: _agent, context="shutdown idle-cache" ) + # Drain completion-batch flush tasks before adapter teardown. + # Cancel each pending flush so its CancelledError handler + # resolves every waiter as retryable (False). Awaited with + # shield + deadline so the cancellation + waiter resolution + # completes before adapters disappear. + # Only completion flushes belong to this pre-teardown drain. + # Selecting from _background_tasks also cancelled startup-resume + # and supervised watcher tasks, globally reordering shutdown + # cancellation for subsystems this PR does not touch. + _flush_tasks = [ + _t for _t in list( + self._completion_notification_batch_tasks.values() + ) + if not _t.done() + ] + if _flush_tasks: + for _t in _flush_tasks: + _t.cancel() + try: + # gather(return_exceptions=True) absorbs the children's + # CancelledError; the previous + # `except asyncio.CancelledError: pass` also swallowed a + # cancellation aimed at _stop_impl_body itself. + await asyncio.wait_for( + asyncio.gather(*_flush_tasks, return_exceptions=True), + timeout=3.0, + ) + except TimeoutError: + logger.warning( + "Completion flush drain timed out after 3s (%d tasks)", + len(_flush_tasks), + ) + for platform, adapter in list(self.adapters.items()): await self._bounded_adapter_teardown(adapter, platform) @@ -9924,6 +9974,7 @@ def _phase_elapsed() -> float: _phase_elapsed(), ) + # Cancel any remaining background tasks after adapters are down. for _task in list(self._background_tasks): if _task is self._stop_task: continue @@ -14422,13 +14473,7 @@ async def _handle_message_with_agent(self, event, source, _quick_key: str, run_g try: from tools.process_registry import process_registry as _pr _watch_events = _drain_gateway_watch_events(_pr.completion_queue) - for evt in _watch_events: - synth_text = _format_gateway_process_notification(evt) - if synth_text: - try: - await self._inject_watch_notification(synth_text, evt) - except Exception as e2: - logger.error("Watch notification injection error: %s", e2) + await self._coalesce_and_inject_watch_events(_watch_events) except Exception as e: logger.debug("Watch queue drain error: %s", e) @@ -18447,6 +18492,402 @@ async def _deliver_completion_notification( except Exception: logger.debug("Could not release durable completion claim", exc_info=True) + # ------------------------------------------------------------------ + # Process completion batching (#70300 / PR #71900) + # ------------------------------------------------------------------ + + class CompletionDisposition(enum.Enum): + """Explicit resolution for every completion notification waiter. + + Replaces the ``Optional[bool]`` tri-state returned by + ``_deliver_completion_notification()`` (still used for backward + compatibility) with explicit variants that prevent silent + consumption of completions. + """ + DELIVERED = "delivered" # adapter accepted; do not retry + RETRY = "retry" # transient failure; watcher should retry + DROP_DUPLICATE = "drop_duplicate" # already delivered by another path + DROP_UNROUTABLE = "drop_unroutable" # permanently unroutable; do not retry + DROPPED_OVERFLOW = "dropped_overflow" # batch capacity exceeded; counted in summary + SHUTTING_DOWN = "shutting_down" # gateway shutting down; retry next cycle + + @staticmethod + def _bool_to_disposition(result: Optional[bool]) -> "CompletionDisposition": + """Map legacy ``Optional[bool]`` to ``CompletionDisposition``.""" + if result is True: + return GatewayRunner.CompletionDisposition.DELIVERED + if result is False: + return GatewayRunner.CompletionDisposition.RETRY + return GatewayRunner.CompletionDisposition.DROP_UNROUTABLE + + @staticmethod + def _completion_notification_batch_key(evt: dict) -> tuple: + """Return a routing-complete key for short-window process fan-in. + + Uses a sentinel for ``None`` so that ``None`` and ``""`` produce + distinct keys — preventing session-boundary crossing. + """ + _NONE = "\x00" + return ( + _NONE if evt.get("session_key") is None else str(evt.get("session_key") or ""), + _NONE if evt.get("platform") is None else str(evt.get("platform") or ""), + _NONE if evt.get("chat_type") is None else str(evt.get("chat_type") or ""), + _NONE if evt.get("chat_id") is None else str(evt.get("chat_id") or ""), + _NONE if evt.get("thread_id") is None else str(evt.get("thread_id") or ""), + _NONE if evt.get("user_id") is None else str(evt.get("user_id") or ""), + ) + + @staticmethod + def _format_coalesced_process_completions( + entries: list, # list[tuple[str, dict, asyncio.Future]] + ) -> str: + """Build one bounded synthetic turn from multiple process completions. + + Counts succeed/failed over *all* entries (not just the rendered + slice) so the aggregate summary is authoritative even when the + detail view is capped at 10. + """ + # Count over ALL entries before slicing for rendering (P3 fix) + all_succeeded = sum( + 1 for _text, evt, _future in entries + if evt.get("exit_code") == 0 + ) + all_failed = len(entries) - all_succeeded + + shown = entries[:10] + lines = [ + f"[IMPORTANT: {len(entries)} background processes completed " + f"for this session.", + "Treat these results as one completion batch and send at most " + "one consolidated user-facing response.", + ] + + for _text, evt, _future in shown: + session_id = str(evt.get("session_id") or "unknown") + exit_code = evt.get("exit_code") + reason = str(evt.get("completion_reason") or "exited") + started_at = evt.get("started_at") + elapsed = "" + if started_at is not None: + import time as _time + try: + _elapsed = _time.time() - float(started_at) + if _elapsed >= 0: + elapsed = f", {_elapsed:.1f}s" + except (TypeError, ValueError): + pass + lines.append( + f"\n {session_id}: " + f"exit_code={exit_code}, reason={reason}{elapsed}" + ) + output = _redact_gateway_user_facing_secrets( + str(evt.get("output") or "") + ).strip() + if output: + if len(output) > 800: + output = f"[… truncated …]\n{output[-800:]}" + lines.append(output) + + omitted = len(entries) - len(shown) + if omitted: + lines.append( + f"\n… and {omitted} more completion(s); inspect " + "them with the process tool if they affect the conclusion." + ) + + # Aggregate summary over ALL entries (P3 fix) + if len(entries) > 1: + parts = [] + if all_succeeded: + parts.append(f"{all_succeeded} succeeded") + if all_failed: + parts.append(f"{all_failed} failed") + lines.append(f"\nSummary: {', '.join(parts)}.") + + lines.append( + "If a result does not change the current conclusion, absorb " + "it silently.]" + ) + return "\n".join(lines) + + def _record_coalesced_completion_siblings( + self, events: list, + ) -> None: + """Extend a successful primary delivery claim to its batched siblings.""" + with self._completion_delivery_lock: + for evt in events: + identity = self._completion_delivery_identity(evt) + if identity is None: + continue + self._completion_deliveries_inflight.discard(identity) + self._completion_deliveries_delivered[identity] = None + while ( + len(self._completion_deliveries_delivered) + > self._completion_delivery_retention + ): + self._completion_deliveries_delivered.popitem(last=False) + + async def _flush_process_completion_batch( + self, key: tuple, + ) -> None: + """Deliver one completion batch and resolve its waiters. + + Waits up to ``_completion_notification_batch_window`` seconds then + delivers everything that accumulated. Every completion resolves to + an explicit result — ``True`` (delivered), ``False`` (retryable), + or ``None`` (permanently unroutable / lifecycle duplicate). + """ + current_task = asyncio.current_task() + entries: list = [] + delivered: Optional[bool] = False + + try: + await asyncio.sleep(self._completion_notification_batch_window) + + entries = self._completion_notification_batches.pop(key, []) + if self._completion_notification_batch_tasks.get(key) is current_task: + self._completion_notification_batch_tasks.pop(key, None) + if not entries: + return + + try: + if len(entries) == 1: + synth_text = entries[0][0] + else: + synth_text = ( + self._format_coalesced_process_completions(entries) + ) + + delivered = None + for _text, candidate_evt, _future in entries: + delivered = await self._deliver_completion_notification( + synth_text, candidate_evt, + ) + if delivered is not None: + break + + if delivered is True: + self._record_coalesced_completion_siblings( + [evt for _text, evt, _future in entries] + ) + except asyncio.CancelledError: + # CancelledError is a BaseException, so `except Exception` + # below never sees it — but the `finally` still ran with + # delivered=None, which _bool_to_disposition maps to + # DROP_UNROUTABLE. _run_process_watcher retries only RETRY, + # so cancellation silently dropped the completion. Resolving + # the waiters here makes the `finally` a no-op for them via + # its `not future.done()` guard. + for _text, _evt, future in entries: + if not future.done(): + future.set_result(self.CompletionDisposition.RETRY) + raise + except Exception: + logger.exception( + "Coalesced process completion delivery failed" + ) + delivered = False + finally: + disposition = self._bool_to_disposition(delivered) + for _text, _evt, future in entries: + if not future.done(): + future.set_result(disposition) + except asyncio.CancelledError: + # Shutdown or lifecycle cancellation — detach the batch, + # resolve every waiter as RETRY so callers retry or surface + # correctly, then re-raise. + stale = self._completion_notification_batches.pop(key, []) + if self._completion_notification_batch_tasks.get(key) is current_task: + self._completion_notification_batch_tasks.pop(key, None) + for _text, _evt, future in stale: + if not future.done(): + future.set_result(self.CompletionDisposition.RETRY) + raise + finally: + if self._completion_notification_batch_tasks.get(key) is current_task: + self._completion_notification_batch_tasks.pop(key, None) + + async def _enqueue_process_completion_notification( + self, synth_text: str, evt: dict, + ) -> "CompletionDisposition": + """Fan in concurrent process completions that share one conversation. + + Completions are grouped by route for up to + ``_completion_notification_batch_window`` seconds, then delivered + as one synthetic turn. + Single completions pay the full batching window — there is no + zero-latency shortcut because true zero-latency requires limiting + coalescing to work already available in the same loop tick. + + Returns a ``CompletionDisposition`` so callers can decide whether + to retry or drop without relying on overloaded ``Optional[bool]``. + """ + # Guard each attribute independently. A single guard on + # _completion_notification_batches meant a partially pre-initialised + # runner — a test using object.__new__ that sets the batch dict but + # not the rest — skipped the whole block and then raised + # AttributeError on _completion_notification_batch_max below. + if not hasattr(self, "_completion_notification_batches"): + self._completion_notification_batches = {} + if not hasattr(self, "_completion_notification_batch_tasks"): + self._completion_notification_batch_tasks = {} + if not hasattr(self, "_completion_notification_batch_lock"): + self._completion_notification_batch_lock = __import__("threading").Lock() + if not hasattr(self, "_completion_notification_batch_window"): + self._completion_notification_batch_window = 0.1 + if not hasattr(self, "_completion_notification_batch_max"): + self._completion_notification_batch_max = 50 + + key = self._completion_notification_batch_key(evt) + future = asyncio.get_running_loop().create_future() + + with self._completion_notification_batch_lock: + batch = self._completion_notification_batches.setdefault(key, []) + if len(batch) >= self._completion_notification_batch_max: + # Cap reached — resolve immediately as overflow so the + # watcher doesn't wait for a flush that will never include + # this entry. DROPPED_OVERFLOW is counted and distinct + # from DROP_UNROUTABLE (route exists, capacity exceeded). + future.set_result(self.CompletionDisposition.DROPPED_OVERFLOW) + return await future + batch.append((synth_text, evt, future)) + + # P1: schedule flush as a lifecycle-owned background task + # so shutdown can drain/cancel it before adapter teardown. + if key not in self._completion_notification_batch_tasks: + task = asyncio.create_task( + self._flush_process_completion_batch(key) + ) + # Register in the gateway's background task set if available. + _bg_tasks = getattr(self, "_background_tasks", None) + if _bg_tasks is not None: + _bg_tasks.add(task) + task.add_done_callback(_bg_tasks.discard) + self._completion_notification_batch_tasks[key] = task + + return await future + + # ------------------------------------------------------------------ + # Watch-event coalescing (post-turn drain) + # ------------------------------------------------------------------ + + async def _coalesce_and_inject_watch_events( + self, watch_events: list, + ) -> None: + """Inject watch events, coalescing same-type events by session_key. + + When multiple watch_match / watch_disabled events fire for the same + session in one drain tick, a single batched message is delivered + instead of flooding the agent with one turn per event. + + Standard completion events that reach here (e.g. from older code + paths) are also coalesced, though the primary completion path is + handled by ``_enqueue_process_completion_notification``. + """ + if not watch_events: + return + + # Separate by type, then group each type by session_key. + buckets: dict[str, dict[str, list]] = {} + for evt in watch_events: + evt_type = str(evt.get("type") or "completion") + sk = str(evt.get("session_key") or evt.get("session_id") or "") + buckets.setdefault(evt_type, {}).setdefault(sk, []).append(evt) + + for evt_type, by_key in buckets.items(): + for sk, group in by_key.items(): + if len(group) == 1: + # Single event — pass through unchanged. + evt = group[0] + synth_text = _format_gateway_process_notification(evt) + if synth_text: + try: + await self._inject_watch_notification( + synth_text, evt, + ) + except Exception as e: + logger.error( + "Watch notification injection error: %s", e, + ) + else: + # Batched — one synthetic turn for N events. + count = len(group) + ids = [ + str(e.get("session_id", "?")) for e in group + ] + if len(ids) <= 5: + id_list = ", ".join(ids) + else: + id_list = ( + ", ".join(ids[:5]) + + f", ...and {len(ids) - 5} more" + ) + logger.debug( + "Coalesced %d %s events into 1 notification " + "for session_key=%s", + count, evt_type, sk, + ) + if evt_type == "watch_match": + patterns = sorted(set( + str(e.get("pattern", "?")) for e in group + )) + pat_list = ", ".join(patterns) + # P4: include bounded per-process snippets so the + # agent can see what triggered the match. + snippet_lines = [] + for e in group[:5]: + sid = str(e.get("session_id", "?")) + cmd = _redact_gateway_user_facing_secrets( + str(e.get("command", ""))[:100] + ) + out = _redact_gateway_user_facing_secrets( + str(e.get("output", ""))[:120] + ) + sup = e.get("suppressed", 0) + line = f" {sid}: {cmd}" + if out: + line += f" — {out}" + if sup: + line += f" (+{sup} suppressed)" + snippet_lines.append(line) + snippets = "\n".join(snippet_lines) + synth_text = ( + f"[IMPORTANT: {count} background processes " + f"matched watch patterns ({pat_list}) — " + f"batched to avoid session flood. " + f"Processes: {id_list}. " + f"Use process(id=N, action='log') to inspect " + f"individual outputs.]\n{snippets}" + ) + elif evt_type == "watch_disabled": + synth_text = ( + f"[IMPORTANT: {count} background processes " + f"had watch patterns disabled — batched to " + f"avoid session flood. " + f"Processes: {id_list}.]" + ) + else: + # Standard completions (fallback). + synth_text = ( + f"[IMPORTANT: {count} background processes " + f"finished ({id_list}) — results batched " + f"to avoid session flood. " + f"Use process(id=N, action='log') to inspect " + f"individual outputs.]" + ) + coalesced_evt = dict(group[0]) + coalesced_evt["coalesced"] = True + coalesced_evt["coalesced_count"] = count + try: + await self._inject_watch_notification( + synth_text, coalesced_evt, + ) + except Exception as e: + logger.error( + "Coalesced watch notification injection " + "error: %s", e, + ) + def _enrich_async_delegation_routing(self, evt: dict) -> None: """Fill platform/chat_id/thread_id/chat_type on an async-delegation event. @@ -18617,10 +19058,10 @@ async def _run_process_watcher(self, watcher: dict) -> None: synth_text = format_process_notification(completion_evt) if not synth_text: break - delivered = await self._deliver_completion_notification( + delivered = await self._enqueue_process_completion_notification( synth_text, completion_evt, ) - if delivered is False: + if delivered is self.CompletionDisposition.RETRY: # The process remains terminal; retry after failed # adapter injection instead of suppressing the result. continue diff --git a/tests/gateway/test_background_process_notifications.py b/tests/gateway/test_background_process_notifications.py index 358b7ff7a7954..bf1e3c6962ee2 100644 --- a/tests/gateway/test_background_process_notifications.py +++ b/tests/gateway/test_background_process_notifications.py @@ -690,3 +690,191 @@ async def fake_self_post(adapter, *, text, session_id): result = await runner._inject_watch_notification("[SYSTEM: done]", evt) assert result is True assert posts == ["raw-origin-sid"] + + +# --------------------------------------------------------------------------- +# _coalesce_and_inject_watch_events tests (post-turn drain coalescing) +# --------------------------------------------------------------------------- + +@pytest.mark.asyncio +async def test_single_watch_event_passes_through_unchanged(monkeypatch, tmp_path): + """A lone watch event should be injected as-is, no coalescing needed.""" + runner = _build_runner(monkeypatch, tmp_path, "all") + adapter = runner.adapters[Platform.TELEGRAM] + adapter.handle_message = AsyncMock() + + evt = { + "type": "watch_match", + "session_id": "proc_abc", + "session_key": "agent:main:telegram:dm:123:42", + "platform": "telegram", + "chat_id": "123", + "thread_id": "42", + "pattern": "DONE", + "output": "Build DONE", + "command": "make build", + } + + await runner._coalesce_and_inject_watch_events([evt]) + + assert adapter.handle_message.await_count == 1 + + +@pytest.mark.asyncio +async def test_multiple_watch_match_same_session_key_are_coalesced( + monkeypatch, tmp_path, +): + """Multiple watch_match events for the same session_key produce exactly + one injection.""" + runner = _build_runner(monkeypatch, tmp_path, "all") + adapter = runner.adapters[Platform.TELEGRAM] + adapter.handle_message = AsyncMock() + + session_key = "agent:main:telegram:dm:123:42" + events = [ + { + "type": "watch_match", + "session_id": f"proc_{i}", + "session_key": session_key, + "platform": "telegram", + "chat_id": "123", + "thread_id": "42", + "pattern": f"PATTERN_{i}", + "output": f"output_{i}", + "command": f"cmd_{i}", + } + for i in range(4) + ] + + await runner._coalesce_and_inject_watch_events(events) + + # 4 events → 1 coalesced injection + assert adapter.handle_message.await_count == 1 + injected_text = adapter.handle_message.await_args.args[0].text + assert "4 background processes matched watch patterns" in injected_text + assert "batched" in injected_text.lower() + for i in range(4): + assert f"proc_{i}" in injected_text + + +@pytest.mark.asyncio +async def test_watch_match_different_session_keys_not_coalesced( + monkeypatch, tmp_path, +): + """watch_match events for different session_keys remain separate.""" + runner = _build_runner(monkeypatch, tmp_path, "all") + adapter = runner.adapters[Platform.TELEGRAM] + adapter.handle_message = AsyncMock() + + events = [ + { + "type": "watch_match", + "session_id": f"proc_a{i}", + "session_key": f"agent:main:telegram:dm:100:{i}", + "platform": "telegram", + "chat_id": "100", + "thread_id": str(i), + "pattern": "DONE", + "output": f"out_{i}", + "command": f"cmd_{i}", + } + for i in range(3) + ] + + await runner._coalesce_and_inject_watch_events(events) + + # Different session_keys → separate injections + assert adapter.handle_message.await_count == 3 + + +@pytest.mark.asyncio +async def test_mixed_watch_types_coalesced_by_type_and_key( + monkeypatch, tmp_path, +): + """watch_match and watch_disabled events for the same session_key + are coalesced separately by type.""" + runner = _build_runner(monkeypatch, tmp_path, "all") + adapter = runner.adapters[Platform.TELEGRAM] + adapter.handle_message = AsyncMock() + + session_key = "agent:main:telegram:dm:123:42" + events = [ + {"type": "watch_match", "session_id": "proc_w1", + "session_key": session_key, "platform": "telegram", + "chat_id": "123", "thread_id": "42", + "pattern": "DONE", "output": "ok", "command": "cmd"}, + {"type": "watch_match", "session_id": "proc_w2", + "session_key": session_key, "platform": "telegram", + "chat_id": "123", "thread_id": "42", + "pattern": "ERROR", "output": "fail", "command": "cmd2"}, + {"type": "watch_disabled", "session_id": "proc_d1", + "session_key": session_key, "platform": "telegram", + "chat_id": "123", "thread_id": "42", + "message": "Watch disabled for proc_d1"}, + {"type": "watch_disabled", "session_id": "proc_d2", + "session_key": session_key, "platform": "telegram", + "chat_id": "123", "thread_id": "42", + "message": "Watch disabled for proc_d2"}, + ] + + await runner._coalesce_and_inject_watch_events(events) + + # 2 watch_match coalesced + 2 watch_disabled coalesced = 2 injections + assert adapter.handle_message.await_count == 2 + texts = [c.args[0].text for c in adapter.handle_message.await_args_list] + + # One should be the watch_match batch + match_texts = [t for t in texts if "matched watch patterns" in t] + assert len(match_texts) == 1 + assert "2 background processes matched watch patterns" in match_texts[0] + + # One should be the watch_disabled batch + disabled_texts = [t for t in texts if "disabled" in t] + assert len(disabled_texts) == 1 + assert "2 background processes had watch patterns disabled" in disabled_texts[0] + + +@pytest.mark.asyncio +async def test_coalesced_ids_truncated_after_5(monkeypatch, tmp_path): + """When >5 processes, first 5 IDs are listed, rest shown as count.""" + runner = _build_runner(monkeypatch, tmp_path, "all") + adapter = runner.adapters[Platform.TELEGRAM] + adapter.handle_message = AsyncMock() + + session_key = "agent:main:telegram:dm:123:42" + events = [ + { + "type": "watch_match", + "session_id": f"proc_{i}", + "session_key": session_key, + "platform": "telegram", + "chat_id": "123", + "thread_id": "42", + "pattern": "DONE", + "output": f"out_{i}", + "command": "cmd", + } + for i in range(8) + ] + + await runner._coalesce_and_inject_watch_events(events) + + assert adapter.handle_message.await_count == 1 + injected_text = adapter.handle_message.await_args.args[0].text + assert "8 background processes matched watch patterns" in injected_text + for i in range(5): + assert f"proc_{i}" in injected_text + assert "...and 3 more" in injected_text + assert "proc_5" not in injected_text + + +@pytest.mark.asyncio +async def test_empty_events_list_is_noop(monkeypatch, tmp_path): + """Empty event list should not cause any injection.""" + runner = _build_runner(monkeypatch, tmp_path, "all") + adapter = runner.adapters[Platform.TELEGRAM] + adapter.handle_message = AsyncMock() + + await runner._coalesce_and_inject_watch_events([]) + + adapter.handle_message.assert_not_awaited() diff --git a/tests/gateway/test_completion_delivery.py b/tests/gateway/test_completion_delivery.py index 4face3841b271..c6c3f95f97332 100644 --- a/tests/gateway/test_completion_delivery.py +++ b/tests/gateway/test_completion_delivery.py @@ -650,3 +650,556 @@ async def _instant_sleep(*_a, **_kw): delivered = adapter.handle_message.await_args.args[0] assert secret not in delivered.text assert "HOME=/home/user" in delivered.text + + +# --------------------------------------------------------------------------- +# Process completion batching / coalescing tests (#70300 / PR #71900) +# --------------------------------------------------------------------------- + +def test_concurrent_process_watchers_coalesce_one_session_completion_turn(monkeypatch): + """Three concurrent terminal watchers for one session must re-enter the + agent once with a batched turn.""" + import tools.process_registry as pr_module + + registry = ProcessRegistry() + watchers = [] + for index in range(3): + session = ProcessSession( + id=f"proc_batch_{index}", + command=f"printf batch-{index}", + task_id=f"task-{index}", + started_at=1000.0 + index, + output_buffer=f"batch-{index}\n", + exited=True, + exit_code=0, + notify_on_complete=True, + ) + registry._finished[session.id] = session + watchers.append({ + "session_id": session.id, + "check_interval": 0, + "session_key": "agent:main:telegram:dm:123", + "platform": "telegram", + "chat_type": "dm", + "chat_id": "123", + "notify_on_complete": True, + }) + monkeypatch.setattr(pr_module, "process_registry", registry) + + adapter = SimpleNamespace(handle_message=AsyncMock()) + runner = _runner(adapter) + + async def _exercise(): + await asyncio.gather(*( + runner._run_process_watcher(w) + for w in watchers + )) + + asyncio.run(_exercise()) + + # Three watchers → one batched turn (not 3 separate injections) + adapter.handle_message.assert_awaited_once() + delivered = adapter.handle_message.await_args.args[0] + assert "3 background processes completed" in delivered.text + for index in range(3): + assert f"proc_batch_{index}" in delivered.text + + +def test_single_completion_uses_original_notification(monkeypatch): + """A single completion should still use its rich original format text.""" + import tools.process_registry as pr_module + + registry = ProcessRegistry() + session = ProcessSession( + id="proc_single", + command="echo hello", + task_id="task-single", + started_at=1000.0, + output_buffer="hello\n", + exited=True, + exit_code=0, + notify_on_complete=True, + ) + registry._finished[session.id] = session + monkeypatch.setattr(pr_module, "process_registry", registry) + + adapter = SimpleNamespace(handle_message=AsyncMock()) + runner = _runner(adapter) + + async def _exercise(): + await runner._run_process_watcher({ + "session_id": session.id, + "check_interval": 0, + "session_key": "agent:main:telegram:dm:123", + "platform": "telegram", + "chat_type": "dm", + "chat_id": "123", + "notify_on_complete": True, + }) + + asyncio.run(_exercise()) + + adapter.handle_message.assert_awaited_once() + delivered = adapter.handle_message.await_args.args[0] + # Should NOT have batch boilerplate for a single completion + assert "background processes completed for this session" not in delivered.text + + +def test_completion_batches_do_not_cross_conversation_routes(): + """Completions for different routes must produce separate deliveries.""" + adapter = SimpleNamespace(handle_message=AsyncMock()) + runner = _runner(adapter) + + first = _completion_event(started_at=1.0, session_id="proc_route_a") + second = _completion_event(started_at=2.0, session_id="proc_route_b") + second["session_key"] = "agent:main:telegram:dm:456" + second["chat_id"] = "456" + + async def _exercise(): + return await asyncio.gather( + runner._enqueue_process_completion_notification("first", first), + runner._enqueue_process_completion_notification("second", second), + ) + + assert asyncio.run(_exercise()) == [ + runner.CompletionDisposition.DELIVERED, + runner.CompletionDisposition.DELIVERED, + ] + assert adapter.handle_message.await_count == 2 + + +def test_completion_arriving_during_batch_delivery_schedules_next_flush(): + """A new event cannot be stranded behind an in-flight batch for its route.""" + first_delivery_entered = asyncio.Event() + release_first_delivery = asyncio.Event() + delivery_count = 0 + + async def _deliver(_event): + nonlocal delivery_count + delivery_count += 1 + if delivery_count == 1: + first_delivery_entered.set() + await release_first_delivery.wait() + + adapter = SimpleNamespace(handle_message=AsyncMock(side_effect=_deliver)) + runner = _runner(adapter) + + async def _exercise(): + first = asyncio.create_task( + runner._enqueue_process_completion_notification( + "first completion", + _completion_event(started_at=1.0, session_id="proc_first"), + ) + ) + await first_delivery_entered.wait() + second = asyncio.create_task( + runner._enqueue_process_completion_notification( + "second completion", + _completion_event(started_at=2.0, session_id="proc_second"), + ) + ) + release_first_delivery.set() + assert await first is runner.CompletionDisposition.DELIVERED + assert await asyncio.wait_for(second, timeout=2.0) is ( + runner.CompletionDisposition.DELIVERED + ) + + asyncio.run(_exercise()) + + assert adapter.handle_message.await_count == 2 + + +def test_failed_coalesced_delivery_retries_all_entries(): + """When an adapter error occurs during batch delivery, every entry + gets a False result so the watcher retry loop can re-enqueue.""" + attempts = 0 + + async def _deliver(_event): + nonlocal attempts + attempts += 1 + if attempts == 1: + raise RuntimeError("temporary adapter failure") + + adapter = SimpleNamespace(handle_message=AsyncMock(side_effect=_deliver)) + runner = _runner(adapter) + events = [ + _completion_event(started_at=float(i), session_id=f"proc_retry_{i}") + for i in range(2) + ] + + async def _enqueue_all(): + return await asyncio.gather(*( + runner._enqueue_process_completion_notification( + f"event-{i}", evt, + ) + for i, evt in enumerate(events) + )) + + async def _exercise(): + # First batch fails → every waiter is resolved as retryable + assert await _enqueue_all() == [ + runner.CompletionDisposition.RETRY, + runner.CompletionDisposition.RETRY, + ] + # Second batch succeeds + assert await _enqueue_all() == [ + runner.CompletionDisposition.DELIVERED, + runner.CompletionDisposition.DELIVERED, + ] + + asyncio.run(_exercise()) + assert adapter.handle_message.await_count == 2 + + +def test_coalesced_success_records_every_completion_identity(): + """After a successful batched delivery, every completion identity must + be present in the lifecycle ledger to prevent re-delivery.""" + adapter = SimpleNamespace(handle_message=AsyncMock()) + runner = _runner(adapter) + events = [ + _completion_event(started_at=float(i), session_id=f"proc_ledger_{i}") + for i in range(3) + ] + + async def _exercise(): + return await asyncio.gather(*( + runner._enqueue_process_completion_notification( + f"event-{i}", evt, + ) + for i, evt in enumerate(events) + )) + + assert asyncio.run(_exercise()) == [ + runner.CompletionDisposition.DELIVERED, + runner.CompletionDisposition.DELIVERED, + runner.CompletionDisposition.DELIVERED, + ] + for evt in events: + identity = runner._completion_delivery_identity(evt) + assert identity in runner._completion_deliveries_delivered + + +def test_duplicate_primary_does_not_discard_fresh_batch_sibling(): + """If the first entry in a batch is a duplicate, the next fresh entry + should be tried as primary so no sibling is discarded.""" + adapter = SimpleNamespace(handle_message=AsyncMock()) + runner = _runner(adapter) + duplicate = _completion_event(started_at=1.0, session_id="proc_duplicate") + fresh = _completion_event(started_at=2.0, session_id="proc_fresh") + duplicate_identity = runner._completion_delivery_identity(duplicate) + runner._completion_deliveries_delivered[duplicate_identity] = None + + async def _exercise(): + return await asyncio.gather( + runner._enqueue_process_completion_notification("dup", duplicate), + runner._enqueue_process_completion_notification("fresh", fresh), + ) + + assert asyncio.run(_exercise()) == [ + runner.CompletionDisposition.DELIVERED, + runner.CompletionDisposition.DELIVERED, + ] + adapter.handle_message.assert_awaited_once() + fresh_identity = runner._completion_delivery_identity(fresh) + assert fresh_identity in runner._completion_deliveries_delivered + + +def test_batch_format_failure_resolves_waiters_for_retry(monkeypatch): + """If batch formatting throws, all waiter futures should resolve with + False so watchers retry rather than being stranded forever.""" + adapter = SimpleNamespace(handle_message=AsyncMock()) + runner = _runner(adapter) + monkeypatch.setattr( + runner, + "_format_coalesced_process_completions", + MagicMock(side_effect=ValueError("bad batch")), + ) + events = [ + _completion_event(started_at=float(i), session_id=f"proc_format_{i}") + for i in range(2) + ] + + async def _exercise(): + pending = asyncio.gather(*( + runner._enqueue_process_completion_notification( + f"event-{i}", evt, + ) + for i, evt in enumerate(events) + )) + return await asyncio.wait_for(pending, timeout=2.0) + + # Formatter failure resolves every waiter as retryable, not as a drop. + assert asyncio.run(_exercise()) == [ + runner.CompletionDisposition.RETRY, + runner.CompletionDisposition.RETRY, + ] + adapter.handle_message.assert_not_awaited() + + +def test_batch_message_format_includes_exit_codes_and_summary(): + """Verify the batched message format includes per-process status, + exit codes, and an aggregate summary line.""" + adapter = SimpleNamespace(handle_message=AsyncMock()) + runner = _runner(adapter) + + events = [ + {**_completion_event(started_at=1.0, session_id="proc_ok"), + "exit_code": 0, "output": "success\n"}, + {**_completion_event(started_at=2.0, session_id="proc_fail"), + "exit_code": 1, "completion_reason": "error", "output": "fail\n"}, + {**_completion_event(started_at=3.0, session_id="proc_ok2"), + "exit_code": 0, "output": "also ok\n"}, + ] + + async def _exercise(): + return await asyncio.gather(*( + runner._enqueue_process_completion_notification( + f"event-{i}", evt, + ) + for i, evt in enumerate(events) + )) + + asyncio.run(_exercise()) + + adapter.handle_message.assert_awaited_once() + text = adapter.handle_message.await_args.args[0].text + + # Must include per-process details + assert "proc_ok" in text + assert "proc_fail" in text + assert "proc_ok2" in text + + # Must include exit codes + assert "exit_code=0" in text + assert "exit_code=1" in text + + # Must include aggregate summary + assert "Summary:" in text + assert "2 succeeded" in text + assert "1 failed" in text + + +def test_batch_message_truncates_entries(): + """When >10 processes complete, show only first 10 in detail.""" + adapter = SimpleNamespace(handle_message=AsyncMock()) + runner = _runner(adapter) + + events = [ + _completion_event(started_at=float(i), session_id=f"proc_{i}") + for i in range(12) + ] + + async def _exercise(): + return await asyncio.gather(*( + runner._enqueue_process_completion_notification( + f"event-{i}", evt, + ) + for i, evt in enumerate(events) + )) + + asyncio.run(_exercise()) + + adapter.handle_message.assert_awaited_once() + text = adapter.handle_message.await_args.args[0].text + + # Must say 12 completed + assert "12 background processes" in text + + # Must mention omitted entries + assert "more" in text + + +def test_batch_message_output_truncates_long_tails(): + """Output >800 chars in batch entries should be truncated.""" + adapter = SimpleNamespace(handle_message=AsyncMock()) + runner = _runner(adapter) + + # Use distinguishable content to verify truncation direction + prefix = "PREFIX_MARKER_" + suffix = "_SUFFIX_MARKER" + long_output = prefix + "x" * 2000 + suffix + events = [ + _completion_event(started_at=1.0, session_id="proc_long"), + _completion_event(started_at=2.0, session_id="proc_short"), + ] + events[0]["output"] = long_output + + async def _exercise(): + return await asyncio.gather(*( + runner._enqueue_process_completion_notification( + f"event-{i}", evt, + ) + for i, evt in enumerate(events) + )) + + asyncio.run(_exercise()) + + adapter.handle_message.assert_awaited_once() + text = adapter.handle_message.await_args.args[0].text + + # Must mention truncation + assert "truncated" in text.lower() + + # The suffix (near end) should survive + assert suffix in text + + # The prefix (near beginning) should not survive truncation + assert prefix not in text + + +def test_single_completion_bounded_latency(): + """A single completion pays the batching window (up to 100 ms) so + coalescing remains correct — true zero-latency singletons would + require limiting coalescing to work already available in the same + loop tick, which is incompatible with a time-window batch.""" + import time as _time + + adapter = SimpleNamespace(handle_message=AsyncMock()) + runner = _runner(adapter) + + evt = _completion_event(started_at=1.0) + + t0 = _time.monotonic() + result = asyncio.run( + runner._enqueue_process_completion_notification("single", evt) + ) + elapsed = _time.monotonic() - t0 + + assert result is runner.CompletionDisposition.DELIVERED + adapter.handle_message.assert_awaited_once() + # Bounded batching delay — well under 1 s even on slow CI. + assert elapsed < 1.0, f"Single completion took {elapsed:.3f}s, expected < 1.0s" +def test_cancellation_during_batch_window_resolves_waiters_retryable(): + """Cancel during the batching sleep resolves pending waiters as False + (retryable) — no stale batch or unresolved Future remains.""" + adapter = SimpleNamespace(handle_message=AsyncMock()) + runner = _runner(adapter) + # Pre-initialise the batch state so the attribute path exists + if not hasattr(runner, "_completion_notification_batches"): + runner._completion_notification_batches = {} + runner._completion_notification_batch_tasks = {} + runner._completion_notification_batch_lock = __import__("threading").Lock() + runner._completion_notification_batch_window = 100.0 # long window + + evt = _completion_event(started_at=1.0, session_id="proc_cancel_window") + + async def _cancel_during_window(): + # Create a task that we can cancel mid-flight + enq_task = asyncio.create_task( + runner._enqueue_process_completion_notification( + "text", evt, + ) + ) + # Let the flush task enter its sleep + await asyncio.sleep(0.01) + # Find and cancel the flush task + key = runner._completion_notification_batch_key(evt) + flush_task = runner._completion_notification_batch_tasks.get(key) + assert flush_task is not None, "flush task should be scheduled" + flush_task.cancel() + # Wait for the enqueue to resolve + result = await enq_task + # After cleanup: no stale batch entry + assert key not in runner._completion_notification_batches + # After cleanup: no stale task key + assert key not in runner._completion_notification_batch_tasks + return result + + result = asyncio.run(_cancel_during_window()) + # Cancelled completion is retryable + assert result is runner.CompletionDisposition.RETRY + + +def test_cancellation_during_delivery_resolves_waiters_retryable(): + """Cancelling the real flush mid-delivery resolves the waiter as RETRY. + + Exercises GatewayRunner._flush_process_completion_batch itself. The + previous version replaced that method with a local reimplementation via + types.MethodType and asserted on the reimplementation, so it passed while + the production path mapped CancelledError onto DROP_UNROUTABLE — the + defect the sweeper review on #72675 reported as finding 1. After the + assertion was migrated to CompletionDisposition.RETRY the stand-in still + resolved its futures with False, so the test failed for a reason + unrelated to the code under test. + + The adapter blocks inside delivery so the flush task can be cancelled + while it is awaiting the send, which is the case the finding describes. + """ + delivery_entered = asyncio.Event() + hold_delivery = asyncio.Event() + + async def _deliver(_event): + delivery_entered.set() + await hold_delivery.wait() + + adapter = SimpleNamespace(handle_message=AsyncMock(side_effect=_deliver)) + runner = _runner(adapter) + runner._background_tasks = set() + + evt = _completion_event(started_at=1.0, session_id="proc_cancel_delivery") + + async def _exercise(): + enq_task = asyncio.create_task( + runner._enqueue_process_completion_notification("text", evt) + ) + await asyncio.wait_for(delivery_entered.wait(), timeout=2.0) + + # The real flush pops the batch and its task key before delivery + # begins, so the task has to be located directly. + flush_task = None + for task in asyncio.all_tasks(): + coro = getattr(task, "get_coro", lambda: None)() + if getattr(coro, "__name__", "") == "_flush_process_completion_batch": + flush_task = task + break + assert flush_task is not None, "real flush task should be running" + flush_task.cancel() + + result = await asyncio.wait_for(enq_task, timeout=2.0) + assert result is runner.CompletionDisposition.RETRY, ( + f"Expected RETRY after cancellation during delivery, got {result!r}" + ) + key = runner._completion_notification_batch_key(evt) + assert key not in runner._completion_notification_batches + assert key not in runner._completion_notification_batch_tasks + + asyncio.run(_exercise()) + + +def test_batch_summary_counts_failures_in_omitted_tail(): + """12-entry batch with failures only outside entries[:10] must still + reflect the correct aggregate success/failure counts in the summary.""" + adapter = SimpleNamespace(handle_message=AsyncMock()) + runner = _runner(adapter) + + events = [ + {**_completion_event(started_at=float(i), session_id=f"proc_{i}"), + "exit_code": (1 if i >= 10 else 0), + "output": f"out_{i}\n"} + for i in range(12) + ] + + async def _exercise(): + return await asyncio.gather(*( + runner._enqueue_process_completion_notification( + f"event-{i}", evt, + ) + for i, evt in enumerate(events) + )) + + asyncio.run(_exercise()) + + adapter.handle_message.assert_awaited_once() + text = adapter.handle_message.await_args.args[0].text + + # All entries counted, not just shown slice + assert "Summary:" in text + assert "10 succeeded" in text + assert "2 failed" in text + + # Failures not shown in detail (they're in the omitted tail) + assert "proc_10" not in text + assert "proc_11" not in text + + # Omitted tail message present + assert "more completion" in text