Skip to content

fix(gateway): close 3 watch-event gaps left by the completion-notification hardening - #86261

Open
pierrenode wants to merge 1 commit into
NousResearch:mainfrom
pierrenode:fix/watch-notification-parity
Open

fix(gateway): close 3 watch-event gaps left by the completion-notification hardening#86261
pierrenode wants to merge 1 commit into
NousResearch:mainfrom
pierrenode:fix/watch-notification-parity

Conversation

@pierrenode

@pierrenode pierrenode commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?

This week's completion-notification hardening series (b9e7bea, c0d2048, 84b4fb9, a96cd10, 7619564, 8dc9401) forced secret redaction, a spawning-session-boundary pre-flight, and title/compression filtering onto completion/async_delegation notifications. watch_match/watch_disabled/watch_overflow_* events were left on the old, weaker behavior in three places. This closes all three gaps for the watch-event family — plus a fourth, related gap found while implementing the above: watch-event notifications weren't attributed to the subagent that owned the watched process.

1. Redaction

_format_gateway_process_notification (gateway/run.py) and the shared format_process_notification (tools/process_registry.py, also used by the TUI gateway) rendered watch_match's output/command from the producer-side, non-forced _redact_process_result pass only. A user with security.redact_secrets: false would get a raw secret sent straight to the chat platform if a watch pattern matched a line containing one.

Both formatters now apply the same forced, unconditional redaction floor the completion path already has (_redact_gateway_user_facing_secrets / redact_terminal_output(..., force=True)).

2. Session-boundary gate

_drain_watch_notifications called _inject_watch_notification directly, bypassing _classify_completion_target entirely. A watch_match/watch_disabled event from a process spawned in session A could still land in session B's chat after /new closed A — the exact bug #70300's fix closed for completion events, left open for watch events.

ProcessSession.parent_session_id (already stamped at spawn time regardless of notification mode) is now also carried on watch_match/watch_disabled events and checked before injection. Unstamped/global events (the cross-session overflow summaries, which have no single owning session) keep delivering unconditionally, matching completion's own legacy fallback. Watch events have no watcher to re-poll them later, so a "retry" (transient DB uncertainty) verdict fails open and delivers rather than losing the match outright — narrower than completion's retry handling, by design.

3. Title/compression filters

title_generator._is_real_user_turn and context_compressor._is_synthetic_compression_user_turn didn't recognize any of the "[IMPORTANT: ...]"/"[ASYNC DELEGATION ...]" notification shapes — only the unrelated compaction/continuation/model-switch markers. A session whose opening turn was a background notification would title itself after it, and a mid-conversation notification could get selected as a compaction anchor.

Both now check message.get("display_kind") == "internal_notification" (the structural marker gateway/run.py already stamps at persist time, #82888) as the primary, text-independent signal, plus explicit text-prefix entries for the raw-string call paths that don't have a message dict to check (is_titleable_user_message's direct callers; context_compressor's existing SessionDB-projected-row test pattern, which calls the recognizer with a bare {"role": "user", "content": ...} dict).

4. Subagent attribution (added)

completion and async_delegation events carry a task_id, which _delegation_attribution_line() resolves (via the _active_subagents registry in tools/delegate_tool.py) into a "Started by subagent ... of delegation ... Task: ..." provenance line — so a user watching a delegated worker's background process knows which subagent it belongs to. watch_match already carried this attribution; watch_disabled (the "I gave up watching, here's the tail" notification fired when the strike limit trips) never did, in both formatters (tools/process_registry.py::format_process_notification and gateway/run.py::_format_gateway_process_notification).

Fixed by stamping task_id on the watch_disabled event dict (mirroring watch_match's existing field) in _check_watch_patterns, and adding the same attribution-line lookup to format_process_notification's watch_disabled branch. gateway/run.py's separate formatter gets the same fix via a lazy import of _delegation_attribution_line for both its watch_match and watch_disabled branches, deliberately kept additive (not delegated to the shared formatter) to avoid silently swapping this function's _redact_gateway_user_facing_secrets guarantee for redact_terminal_output.

Testing

  • 22 new/extended test cases across tests/gateway/test_background_process_notifications.py, tests/gateway/test_completion_session_boundary.py, tests/agent/test_title_generator.py, tests/agent/test_context_compressor_zero_user_provenance.py, tests/tools/test_watch_patterns.py.
  • Mutation-verified: all fail against the pre-fix code (surgically reverted the relevant hunks, confirmed the new tests fail, restored) and pass with it restored.
  • Full neighbor suites green: tests/gateway/, tests/agent/test_context_compressor*.py, tests/tools/test_process_registry.py, tests/tools/test_watch_patterns.py, tests/tools/test_async_delegation.py, tests/tools/test_delegate_control_actions.py, tests/agent/test_title_generator.py — 270 passed, 5 pre-existing/unrelated skips, 0 failures.
  • ruff check clean on all changed files.

Adjacent open PRs (checked, no semantic overlap)

Type of Change

  • Bug fix (non-breaking change that fixes an issue)
  • Security-relevant (redaction floor)

@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint comp/gateway Gateway runner, session dispatch, delivery tool/terminal Terminal execution and process management area/compression Context compression and continuation sessions sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state labels Aug 14, 2026
@alt-glitch

Copy link
Copy Markdown
Collaborator

This was generated by AI during triage.

Related: #85923 and #59858. This addresses remaining watch-event redaction, session-boundary, and notification-provenance gaps after the merged notification salvage.

@Enough1122

Copy link
Copy Markdown
Contributor

AI code review — automated review for reference, author can ignore or act on any point.

fix(gateway): close 3 watch-event gaps left by the completion-notification hardening

  1. Prefix-list drift: the five new notification prefixes are duplicated verbatim in agent/context_compressor.py (_WATCH_DISABLED_NOTIFICATION_PREFIX, _WATCH_OVERFLOW_*, _ASYNC_DELEGATION_*) and agent/title_generator.py (_MACHINE_PREFIXES). The comments in both files acknowledge the producer formats live in tools/process_registry.py / gateway/run.py. A future producer-side wording change will silently break both recognizers unless all three files change together. Consider importing the prefix constants from one shared definition (or a small helper) so the list has a single source of truth.
  2. gateway/run.py_drain_watch_notifications boundary gate: unstamped (legacy/global) events keep delivering unconditionally, while watch_disabled events are now stamped with parent_session_id and gated. Since process_registry.py stamps both watch_disabled and watch_match, the only unstamped family left is the overflow pair — that asymmetry is intentional per the comment, but worth noting the watch_disabled event emitted from _check_watch_patterns (the rate-limit strike path) is stamped, whereas the overflow events are not, so an overflow notification can land in a /new-closed chat. If that is acceptable, a one-line test asserting the overflow event is not gated would lock the intended behavior.
  3. Redaction divergence: gateway/run.py uses _redact_gateway_user_facing_secrets() for both output and command, while tools/process_registry.py::format_process_notification uses redact_terminal_output(..., force=True) for the same two fields. If these two redactors have different coverage (prefix matching vs. terminal-output heuristics), the same watch_match event renders differently depending on which formatter produced it (gateway adapter vs TUI surface). A cross-formatter test asserting both produce the same redacted text for a representative secret would prevent future divergence.
  4. agent/title_generator.py_is_real_user_turn now returns False for any display_kind == "internal_notification" user turn. This is checked before the content parsing, so a genuinely user-authored message that somehow carries that stamp (e.g. a persisted row mis-stamped by an older gateway version) would be permanently excluded from titling. The structural marker is authoritative per the fix, but a fallback that also requires the content to match a known notification shape would be more defensive.

@pierrenode

Copy link
Copy Markdown
Contributor Author

CI caught a real regression in the last commit, now fixed in daa6c88506 (both jobs were the same root cause — All required checks pass is just the aggregator gate for the tests job).

What broke: item 3 of this PR (title/compression filters) made context_compressor._is_synthetic_compression_user_turn recognize "[ASYNC DELEGATION COMPLETE"/"[ASYNC DELEGATION BATCH COMPLETE" text and the generic display_kind="internal_notification" marker as synthetic — lumping async-delegation completions into the same "bookkeeping notification, exclude from compaction anchors" bucket as watch/background-process events. That was wrong: unlike those, _format_async_delegation's own docstring says the block carries "the complete result summary" — genuine actionable content, not status noise. This directly conflicted with the pre-existing test_completion_survives_compaction_verbatim_after_blank_echo (bc48241, "preserve latest actionable user turn"), which constructs exactly this wire format with real content and asserts it survives compaction — my two new async_delegation/async_delegation_batch test cases asserted the opposite for the same shape.

Fix: async-delegation completions are now excluded from _is_synthetic_compression_user_turn's notion of synthetic (both the removed text-prefix checks and a guard on the display_kind fast path, since that marker is stamped generically across all internal-notification types and can't distinguish on its own). title_generator.py's exclusion is untouched — a session title should never literally be the delegation-complete boilerplate regardless of the payload, which is a separate concern from compaction (which must preserve the payload). Removed my two wrong test cases, added test_async_delegation_completion_is_not_synthetic covering both paths, and mutation-verified it against the original wrong code plus each half of the fix independently — all three reproduce the exact CI failure.

Full neighbor sweep (273 tests across the touched files + agent/'s compress/compaction suite) green, ruff clean.

…ation hardening

This week's completion-notification hardening series (b9e7bea, c0d2048,
84b4fb9, a96cd10, 7619564, 8dc9401) forced secret redaction, a
spawning-session-boundary pre-flight, and title/compression filtering onto
completion/async_delegation notifications. watch_match/watch_disabled/
watch_overflow_* events were left on the old, weaker behavior in three
places — plus a fourth, related gap found while implementing the above:
watch-event notifications weren't attributed to the subagent that owned the
watched process.

1. Redaction: _format_gateway_process_notification (gateway/run.py) and the
   shared format_process_notification (tools/process_registry.py, also used
   by the TUI gateway) rendered watch_match's output/command from the
   producer-side, non-forced _redact_process_result pass only. A user with
   security.redact_secrets: false would get a raw secret sent straight to
   the chat platform if a watch pattern matched a line containing one. Both
   formatters now apply the same forced, unconditional redaction floor the
   completion path already has.

2. Session-boundary gate: _drain_watch_notifications called
   _inject_watch_notification directly, bypassing _classify_completion_target
   entirely. A watch_match/watch_disabled event from a process spawned in
   session A could still land in session B's chat after /new closed A.
   ProcessSession.parent_session_id (already stamped at spawn time) is now
   also carried on watch_match/watch_disabled events and checked before
   injection. Unstamped/global events (the cross-session overflow summaries)
   keep delivering unconditionally, matching completion's own legacy
   fallback. Watch events have no watcher to re-poll them later, so a
   "retry" (transient DB uncertainty) verdict fails open and delivers rather
   than losing the match outright.

3. Title/compression filters: title_generator._is_real_user_turn and
   context_compressor._is_synthetic_compression_user_turn didn't recognize
   any of the "[IMPORTANT: ...]"/"[ASYNC DELEGATION ...]" notification
   shapes — only the unrelated compaction/continuation/model-switch markers.
   Both now check message.get("display_kind") == "internal_notification"
   (the structural marker gateway/run.py stamps at persist time) as the
   primary signal, plus explicit text-prefix entries for the raw-string call
   paths that don't have a message dict to check. Async-delegation
   completions are excluded from the compression side of this: unlike
   watch/background-process bookkeeping, _format_async_delegation's own
   docstring says the block carries "the complete result summary" — genuine
   actionable content a real user turn would also carry — so treating it as
   synthetic would let compaction blank out a delegation's actual result
   (see test_completion_survives_compaction_verbatim_after_blank_echo,
   bc48241). They're still excluded from titling, where the boilerplate
   wrapper text would make a bad title regardless of the payload.

4. Subagent attribution: completion/async_delegation events resolve their
   task_id (via tools/delegate_tool.py's _active_subagents registry) into a
   "Started by subagent ... of delegation ... Task: ..." provenance line.
   watch_match already carried this; watch_disabled never did, in either
   formatter. Fixed by stamping task_id on the watch_disabled event dict and
   adding the same attribution lookup to both formatters — gateway/run.py's
   kept additive (not delegated to the shared formatter) to avoid silently
   swapping its _redact_gateway_user_facing_secrets guarantee for
   redact_terminal_output.

Mutation-verified throughout, including the async-delegation exclusion
against both the original wrong code and each half of the fix independently
— all reproduce test_completion_survives_compaction_verbatim_after_blank_echo
failing. Full neighbor sweep green (tests/gateway/, tests/agent/
compress/compaction suite, tests/tools/test_watch_patterns.py,
tests/tools/test_process_registry.py, tests/tools/test_async_delegation.py).
ruff clean.

Adjacent open PRs checked, no semantic overlap:
- NousResearch#75719 restructures the same gateway/run.py formatter for an unrelated
  concern (a "supersession context" note on delayed notifications) —
  textual proximity only.
- NousResearch#61719 adds a different field (origin_ui_session_id) to the same
  watch_match/watch_disabled dict literals, for TUI/WebUI tab ownership —
  complementary, not overlapping.
- NousResearch#73469 is an alternative architecture for the same-tick
  completion-coalescing race this week's series already solved differently;
  different function region.
@pierrenode
pierrenode force-pushed the fix/watch-notification-parity branch from daa6c88 to 5e857b3 Compare August 19, 2026 22:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/compression Context compression and continuation sessions comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint comp/gateway Gateway runner, session dispatch, delivery P2 Medium — degraded but workaround exists sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state tool/terminal Terminal execution and process management type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants