feat(whatsapp): observe unmentioned group messages as session context

[FarelRA]

Stacked on #104133 (AST formatter). Merge #104133 first and this diff shrinks to just this feature.

What

Ports Telegram's observe_unmentioned_group_messages gate to WhatsApp — and generalizes the runner side so the separation is platform-neutral. In groups with require_mention: true, messages that pass the group allowlist but aren't addressed to the bot (no @mention, no reply-to-bot, no mention-pattern keyword) are no longer dropped silently — they're appended to the group's shared chat-scoped session transcript as observed context, with zero model/API cost. When a later mention/keyword/reply triggers a turn in that same group, the model sees the accumulated conversation flow.

Runner-neutral separation (fixes the replay bug this PR originally shipped with)

gateway/run.py's _build_gateway_agent_history previously recognized only Telegram's observed-context prompt marker. A WhatsApp adapter emitting the equivalent marker got no separation: its observed rows replayed as ordinary user history. This PR:

• moves the whole mechanism into a platform-neutral sibling, gateway/observed_context.py: marker constants (single source both adapters build their prompts from), injection limits, assembly, and compaction;
• generalizes the Telegram-only marker check to the shared predicate — any future platform adapter opts in by carrying its marker in the channel prompt; no per-platform runner code;
• makes the header platform-neutral: [Observed group context - context only, not requests]. Telegram behavior is unchanged (covered by the existing suite).

Without a recognized marker the behavior is fail-closed: observed rows replay as normal history (documented + tested).

Media in observations (real artifacts, not placeholders)

The observe path downloads media to the local cache (disk-only; no agent or API calls at observation time) and records inspectable references: [image: /path] plus a vision_analyze pointer, [voice note: /path], [audio: /path], [document: /path], [video: <url>]. At trigger time the model inspects via the normal vision_analyze / read_file contract. A failed download degrades to [image (unavailable: download failed)] — observation never raises.

Bounds: rolling compaction, not truncation

The observed block is API-only on the current user turn (the transcript persists the unwrapped message), which makes it prompt-cache-safe — but it also means context compression can never shrink it. Dropping oldest rows (truncation) destroys information. Instead:

• Rolling compaction at write time: when the verbatim observed set overflows roughly half the char budget, a background pass summarizes the OLDEST rows via the aux compression LLM (call_llm(task="compression") — same routing/limits as main compaction, secret redaction, per-row and total input bounds) into one iterative summary row appended to the transcript. Later passes update that summary in place (PREVIOUS SUMMARY + NEW TURNS, the same iterative shape the main compressor uses). No turn ever waits on the summary call (cold TTFB untouched).
• Read path: latest summary + verbatim rows after it (the summary row is the watermark). Originals stay in the transcript, searchable via FTS — nothing is destroyed.
• Hard cap as the valve: gateway.observed_context_max_chars (default 512,000 ~ 128K tokens, sized for common 256K-window models) / observed_context_max_rows (default 4,000). If compaction lags or the aux call fails, the cap drops oldest-first with an [... N older observed messages omitted ...] note. Both keys are config.yaml-only (no new env vars); 0 disables.
• Cheap scheduling: a per-session pending-chars gate means a busy group does not reload its transcript on every message; a per-session lock prevents duplicate passes; a failure cooldown avoids hammering the aux LLM.
• Platform-neutral: the same trigger fires from the shared WhatsApp mixin AND the Telegram adapter — Telegram's observed block gets the same bounds + compaction (it previously had none).

Attribution scoping

_apply_whatsapp_group_observe_attribution applies only to mention-gated, allowlisted groups: in group_policy: open groups or free_response_chats, triggered messages keep their real per-user source and clean text (ported from #38750's post-audit fix, generalized over the allowlist scope).

Cache/alternation invariants

• The wrapper prepends to the current turn only; the persisted row is the unwrapped ctx.message (run_turn_runner.py persist_user_message), so the transcript never stores the context block — cache-safe by construction. Compaction changes the block content between turns, which is exactly the sanctioned compression exception and never touches the cached prefix (system prompt + history).
• Removing observed user rows from replay cannot create same-role consecutive messages (alternation asserted in the runner test).
• System prompt untouched.

Interlock with the other observe PRs

The category currently has several open PRs (#38750 whatsapp — pre-plugin-layout; #74775 qqbot; #84592 signal; #85490 whatsapp — durable persistence + 50-row window; #101121 mattermost; this one). Per the rubric's consolidation rule, the sustainable shape is a shared observed-context mechanism in the runner + thin per-platform adapters — this PR is that shape (shared gateway/observed_context.py, shared whatsapp_common.py mixin reused by bridge + WAHA, Telegram trigger included). Relative to #85490 specifically: overlapping runner separation; if it merges first I'll rebase and drop my overlapping runner diff, keeping media references, compaction, and the scoping fix. The write side here is transcript-only (no new persistence layer, no dedup store): the summary row + watermark is the mechanism, which keeps the diff small and the storage format unchanged.

Tests

• E2E runner tests (tests/gateway/test_observed_context_runner.py) through the REAL _build_gateway_agent_history + _wrap_current_message_with_observed_context (no mocks): separation for both platforms, fail-closed replay without a marker, cap relationships, alternation survival, unwrapped persistence contract, defaults sized for large windows. Sabotage-checked: reverting the marker generalization fails 6 of these.
• Compaction tests (tests/gateway/test_observed_context_compaction.py): watermark assembly (summary leads, pre-watermark rows skipped), hard-cap valve relationships, iterative update over the previous summary (wrapper stripped), per-row + total input bounds with redaction (secret never reaches the summarizer), aux-failure leaves the transcript untouched, scheduling gate (no transcript reload per message), no-event-loop no-op. Sabotage-checked: breaking the watermark skip or the summary append fails the suite.
• Gating tests: 15 observe tests (observed-not-dispatched, trigger dispatch instead, allowlist enforcement, DM exclusion, chat-scoped source, attribution + safety prompt, media reference with cached path + vision note, download-failure degradation, scoping x3). Full tests/gateway/ + tests/plugins/ + tests/conformance/ green: 9646 passed, 0 failed on this branch.
• Honest limits: a live end-to-end compaction (real aux LLM + real transcript) is not exercised by CI; the LLM boundary is faked at _call_summary_llm and everything around it runs the real code. End-to-end send/download against a live WhatsApp bridge is likewise not exercised by CI; the media path is covered at the adapter boundary with mocked downloads.

Docs

website/docs/user-guide/messaging/whatsapp.md: opt-in, media references, compaction + bounds + retention honesty, scoping note.

[alt-glitch]

This was generated by AI during triage.

Related: this implements #38710 and competes with the older open #38750 for the same feature (observed unmentioned WhatsApp group context). Linking so reviewers can compare; #104247 (WAHA) stacks on this.

[FarelRA]

Reworked after the maintainer-standards pass: truncation replaced by rolling compaction

The earlier revision bounded the observed block at read time by dropping the oldest rows. That destroys information, so it is replaced with a rolling compaction mechanism in a new platform-neutral sibling, gateway/observed_context.py:

Write path (eager, background). When the verbatim observed set overflows roughly half the char budget, the observe appenders (shared WhatsApp mixin + Telegram adapter) schedule a fire-and-forget background pass. It summarizes the OLDEST rows via the aux compression LLM (call_llm(task="compression") — same routing, semaphore, and redaction the main compressor uses) into one iterative summary row appended to the transcript. Later passes feed PREVIOUS SUMMARY + NEW TURNS back in, so the summary updates in place instead of restarting. No turn ever waits on the summary call.

Read path. The latest summary row is the watermark: the injected block is the summary + verbatim rows after it. Originals stay in the transcript and searchable — nothing is deleted. A hard cap (gateway.observed_context_max_chars, default 512,000 chars ~ 128K tokens for common 256K-window models; observed_context_max_rows, default 4,000) remains purely as the last-resort valve with the omission note.

Platform-neutral. Marker constants now live in the sibling as the single source; the WhatsApp mixin and the Telegram adapter both build their prompts from them and both trigger compaction. Telegram's observed block previously had no bound at all — it gets the same treatment.

Scheduling hygiene. A per-session pending-chars gate (spawn only past half the budget) means a busy group does not reload its transcript on every message; a per-session lock prevents duplicate passes; a 5-minute failure cooldown avoids hammering the aux LLM.

Tests. New tests/gateway/test_observed_context_compaction.py (watermark assembly, iterative update, redaction bounds — a planted secret never reaches the summarizer prompt, aux-failure safety, scheduling gate) and the runner e2e file updated for the new semantics. Both sabotage-checked (breaking the watermark skip, the summary append, or the marker generalization fails the suite). Full tests/gateway/ + tests/plugins/ + tests/conformance/: 9646 passed, 0 failed.

Also in this push: defaults raised to 512K/4K per the sizing rationale above, and the stale 4,096 chunking paragraph in the docs fixed (on #104133, where the cap change lives).

[FarelRA]

Rebased onto current main (176 new commits). The only overlap was gateway/run.py: upstream's timestamp-replay rework (08f269f) touches the same loop this PR modifies. Resolution keeps both behaviors: observed rows render timestamps like upstream does, but onto a copied row so the transcript stays untouched and the sibling's assembly reads the rendered content. Full suite re-run green: 9673 passed, 0 failed (gateway + plugins + conformance). #104247 rebased on top via rebase --onto (skipping the superseded observe commit).