feat(gateway): session activity heartbeats, stall watchdog, bounded compression waits

[teknium1]

feat(gateway): session activity watchdog — clean re-land of #73031 with commit-phase ceiling fix and heartbeat write discipline

Summary

Re-lands the session activity watchdog (stall detection + one-shot /new notify + progress-aware compression timeouts, #72016) from @fangliquanflq's PR #73031 onto current main, composed with tonight's SessionDB retry/cooldown merges, and fixes the two known defects: the silently-unenforced context_total_ceiling_seconds during commit-phase hangs, and unpinned heartbeat write cadence.

Related to #72016, supersedes #73031.

Staged commit map

Contributor commits (cherry-picked in order, authorship preserved):

Commit	Author	What
60b7ec29	@fangliquanflq	feat: session activity watchdog, stall notify, compress timeout (#72424 base)
69bcebbb	@kshitijk4poor	refactor: reuse existing utilities in salvaged #72424
49b6315a	@kshitijk4poor	fix: propagate logging session context after daemon-pool compress_context
cd7cbc4e	@fangliquanflq	fix: silence detached compression heartbeat after host timeout
7676ba27	@fangliquanflq	chore: contributor email mapping
57f6e4ae	@fangliquanflq	fix: force-persist compression completed past SessionDB rate limit
0bb796ec	@fangliquanflq	fix: clarify compress_context ceiling is pre-commit only
ce8fe843	@fangliquanflq	fix: restore end_turn in run_conversation finally

Our follow-up fixes:

Commit	What
0b7666c5	Ceiling bug fix: bounded, surfaced commit-phase waits past context_total_ceiling_seconds
642e24be	Heartbeat write discipline: pinned cadence constant + best-effort hardening + tests
f997b5db	Deflake concurrent-fork suite (live aux feasibility probe inside stubbed-compressor tests; reproduced on #73031's own head)

Conflict resolutions during cherry-pick favored current main: hunks referencing the removed in-memory _hygiene_compression_failure_cooldowns dict were re-targeted to the DB-backed _record_hygiene_cooldown helpers from #74251; hermes_state.py deadline-patience retry work was preserved intact.

The timeout guarantee (exact wording, now in config comment + docs en/zh)

The summary phase is bounded by context_total_ceiling_seconds; the commit phase is logged and surfaced if it exceeds it. Once the worker has entered the compression commit fence and SessionDB mutation is in flight, the commit is never abandoned mid-flight (that would risk transcript divergence) — but the wait is no longer silent: on ceiling breach the overrun is logged (WARNING, escalating to ERROR on repeat), surfaced once through the user-visible warning channel (on_commit_overrun → _emit_warning), and the host keeps waiting in bounded 30s increments (_COMMIT_OVERRUN_WAIT_SLICE_SECONDS) until the commit completes.

Previously the post-begin_commit() waiter called unbounded future.result() — the advertised ceiling was simply not enforced (or even observed) for commit-phase hangs. The test that ACCEPTED silent over-ceiling waits (test_never_finishing_commit_waits_past_pre_commit_ceiling) now asserts the warning/log surfacing fires, and a new test proves a raising overrun callback can't break the commit wait.

Heartbeat cadence + failure semantics

• Cadence: pinned in SESSION_ACTIVITY_HEARTBEAT_MIN_INTERVAL_SECONDS = 60.0 (agent/session_activity.py) with a documented >= 30s contract. Deliberately a code constant, independent of all compression.*/agent.* config — no configuration can turn the heartbeat into a high-frequency writer. force_persist (terminal compression stamps) is the only bypass.
• Write path: verified (not assumed) — the heartbeat goes through SessionDB.touch_session_activity → self._execute_write(...), i.e. the standard patience path including tonight's deadline-patience retry, _sleep_before_write_retry, and no-more-rows message-scoped retry.
• Failure semantics: best-effort by contract. A failed heartbeat write NEVER raises into the agent loop — swallow + debug-log ("session activity heartbeat write failed (ignored)"), natural retry on the next due window. Proven by test_heartbeat_write_failure_never_propagates_direct (direct _persist_session_activity_if_due call with OSError side effect) plus the existing _touch_activity-level swallow test.

Watchdog semantics (verified)

• Fires WARNING log + one-shot /new notify only when busy + pending inbound + idle > agent.session_stall_timeout (default 300, 0 = disabled — test_session_stall_watcher_disabled_when_timeout_zero).
• One-shot dedup across ticks: test_check_session_stalls_notifies_once + test_check_session_stalls_does_not_renotify_after_summary_gap + re-notify-after-recovery episode semantics (test_check_session_stalls_renotifies_after_resume_then_restall).
• Single source of truth: idle derives exclusively from agent.get_activity_summary() / the shared agent.session_activity contract (resolve_session_idle_seconds_from_activity refuses turn-start/pending-inbound clocks; test_check_session_stalls_ignores_raw_clock_without_summary). Consistent with the PR fix: replace wall-clock agent timeout with inactivity-based timeout #4864 precedent — no parallel activity tracker.
• Fenceless in-agent compress timeout REUSES the existing CompressionCommitFence.touch_progress()/seconds_since_progress() machinery (32fd9d6); no second progress clock.

Invariants preserved

• No conversation-history injection. Grepped the full branch diff for messages.append / user-role injection in non-test code: none. The watchdog is notify-only (adapter send), never mutates the transcript.
• No prompt-cache break, no role-alternation break — nothing is inserted into the message list at any point.
• Notify-only: the stall watcher never kills the turn (contrast gateway_timeout / shutdown watchdog).
• DB-backed hygiene cooldowns (fix: persist session hygiene compression cooldown to state DB #74251) preserved — all cooldown writes go through _record_hygiene_cooldown.

Validation

Check	Result
PR's listed test set (compress-timeout, concurrent-fork, session-activity, agent-cache, cached-agent-max-iter, config-env-bridge, stall-watchdog, status, codex-compaction, activity-persist, hermes_state)	pass
tests/state/ (incl. test_no_more_rows_retry, test_session_model_usage_pk_heal, test_write_lock_patience)	pass
Hygiene cooldown tests (tests/gateway/test_session_hygiene.py) + compression lock/busy-retry suites	pass
Review-fix regression files (tests/agent/test_compression_review_76354.py, tests/agent/test_compression_worker_isolation_76354.py, tests/gateway/test_watchdog_review_76354.py) — F1–F6 + S1–S4, all asserting the BLOCKED state before releasing workers	18 tests, pass
Combined affected surface after the review-fix series (progress-timeout, concurrent-fork, logging-context, review regressions, session-activity, anti-thrash, telemetry, rotation-state, interrupt-protection, progress, fallback-budget, max-attempts, preflight-gate, manual-feedback, context-compressor, activity-persist, stall-watchdog, hygiene, tests/state/, hermes_state)	452 passed
Per-commit verification: each of the 10 review-fix commits checked out in a worktree; cumulative regression files green at every stage	17 → 33 tests, all green
Sabotage checks: every new regression flipped red with its fix reverted (F1/F2/F6/S3 batch, F4 hook + cooldown ordering, F3/F4-lease/F5 batch, S1/S2 batch), then restored green	verified
ruff check on all touched files	clean
git diff --check (incl. contributors/emails whitespace fix)	clean
Pre-push gate: rebased onto current origin/main; behind-count = 0	verified

Review-fix series (#76354 review by @helix4u)

All six blocking findings, the additional correctness/scope items, and all 10
minimum merge gates are addressed in 10 follow-up commits (one concern per
commit): F1 lock-free commit-phase observability, F2 commit-admission
revocation on every host unwind (reconciled with #74449 by composition), F3
worker transcript isolation + documented thread-safety contract for pooled
context engines / memory providers, F4 holder-qualified durable lease
cancellation + cooldown-clear ordering (transplanted from #71569,
Co-authored-by @ciabata-git), F5 caller-side session ContextVar rebind after
rotation, F6 bounded executor admission + stale-job cancellation, S1
sub-second busy budget for observational activity writes + no-op clear skip,
S2 pre-delivery stall revalidation, S3 idle wait charged from last progress,
S4 precise watchdog scope docs + explicit import-resets-activity contract.

Credit

Base implementation by @fangliquanflq (with refactor commits by @kshitijk4poor), authorship preserved per-commit via cherry-pick. The #72858 revert of the original landing was process-only (landing sequence), not a defect in the contributor's design — this branch re-lands that work composed with current main and hardens the two known gaps on top.

Infographic

[github-actions]

૮ >ﻌ< ა ci review

_{ran on 0c70ca3}

all good!

[alt-glitch]

This was generated by AI during triage.

Related: this current-main maintainer re-land broadens #73031 with a surfaced commit-phase overrun path and pinned heartbeat-write discipline; it is not a redundant same-policy patch.

[helix4u]

Reviewed current head f997b5dbc204653e715149ad2ec269e7007c1e31 against current main.

Overall, the session-activity/watchdog side looks coherent and deliberately scoped, but I do not think the bounded-compression portion is safe to merge yet. The central problem is that a timeout returns control to the caller while the detached compression worker can still hold durable resources, mutate shared state, and eventually commit. Several of the new guarantees therefore do not hold under the failure modes this PR is intended to handle.

What looks sound

• The re-land appears clean and preserves the prior contributor history.
• Activity snapshots are bounded and observational; they do not alter transcript, prompt, role alternation, or cache state.
• SessionDB updates keep timestamp, description, and provenance together and reject ordinary backward timestamp movement.
• Session recency correctly considers both message activity and the new heartbeat activity.
• The watchdog is notification-only, scans the default and multiplex adapters with deduplication, handles SendResult(success=False) as retryable, and clears its one-shot latch on recovery, drain, disable, and lifecycle boundaries.
• The existing database-backed gateway hygiene cooldown is preserved.
• The watchdog’s /new delivery bypasses the running agent, which is necessary for this recovery path.
• The added fields are backward-compatible at the storage/schema boundary.
• All required CI checks are currently green. The issues below are concurrency/lifecycle paths not exercised by those checks.

Blocking findings

1. The commit-phase overrun warning cannot fire while a commit is actually hung

begin_commit() retains the fence lock until finish_commit(). After the host-side ceiling expires, try_cancel_before_commit() also needs that lock.

The timeout path spins while try_cancel_before_commit() returns None, but the new overrun-warning loop is only reached after that spin. Therefore:

1. The worker enters commit and holds the lock.
2. The host ceiling expires.
3. The host cannot inspect/cancel the fence because the worker owns the lock.
4. The host spins indefinitely before reaching the warning code.
5. The warning becomes reachable only after the commit finishes and releases the lock.

That means a genuinely hung commit remains both unbounded and silent. The current regression test releases the commit before checking the callback/log, so it does not test the hung state: test_compress_context_progress_timeout.py.

The phase needs to be observable without acquiring the lock held throughout commit, for example through a separate event/atomic phase marker. The regression should assert that the warning fires while the commit remains blocked.

2. A non-timeout unwind can leave an unfenced worker that later commits

The whole compression operation is submitted to the daemon executor here, but the host only handles concurrent.futures.TimeoutError. There is no finally or broader unwind path that closes the worker’s commit authority.

If the caller receives KeyboardInterrupt, cancellation, or another non-timeout exception while waiting, the host can exit while the worker keeps running. The worker can subsequently enter the commit fence and mutate durable state, including in-place archival or session rotation/publication.

Every host exit path needs to revoke future commit admission, with tests for at least interruption/cancellation before commit. This overlaps the lifecycle and rollback work in #74449, so those designs need to be reconciled rather than treated independently.

3. A timed-out worker shares the caller’s live transcript and mutable agent state

The worker closure in run_agent.py captures the caller’s current messages object: run_agent.py. That live object is passed through to the compression implementation: conversation_compression.py.

The timeout path then returns the existing messages while deliberately leaving the worker alive. However, the implementation explicitly supports plugin/legacy context engines that mutate the input list in place, and the fence is not checked until after the engine returns.

A hung or merely late context engine can therefore mutate the transcript concurrently after the host has resumed normal conversation. This can alter roles, message ordering, context contents, or what is persisted. Similar concerns apply to mutable compressor, memory, and lifecycle fields touched before the commit fence.

The worker needs an isolated snapshot and worker-owned intermediate state. Only a successful, admitted commit should publish changes to caller-visible or durable state. A regression should use an in-place-mutating engine, trigger the timeout, and prove the caller’s transcript remains unchanged while the old worker continues.

4. Timeout does not release the durable compression lock or lease refresher

Compression acquires its durable lock before the potentially hanging summary operation, and starts the lease refresher before invoking the context engine.

When the host times out, it cannot call the worker-local lock cleanup. The worker releases the lock only after the summary returns and it observes cancellation at the later fence check. A truly hung summary therefore continues retaining and refreshing the durable lease indefinitely, preventing later compression attempts.

The current concurrency regression releases the old summary and joins the worker before checking the lock, so it does not establish that a new compressor can proceed immediately after host timeout: test_compression_concurrent_fork.py.

There is also a state-ordering issue: a late successful summary can call _clear_compression_failure_cooldown() before reaching the canceled commit fence, undoing the timeout cooldown recorded by the host.

The lock needs holder-qualified cancellation/release semantics that the host can invoke without letting the stale worker release a replacement worker’s lease. Late completion must not clear cooldown or publish any state. This substantially overlaps #71569; that PR is too stale to merge wholesale, but its lease-cancellation and executor-admission work should be reconciled here.

Required regression:

1. Block the original summary indefinitely.
2. Let the host timeout.
3. Prove another compressor can acquire the durable lock before releasing the original summary.
4. Release the old worker.
5. Prove it cannot clear cooldown, release the new holder’s lease, or publish stale state.

5. Out-of-place compression leaves the caller’s session ContextVar stale

On session rotation, the new session ID is set inside the compression worker’s copied context. After the worker completes, the caller repairs only the logging thread-local state: run_agent.py.

get_session_env() prefers the caller’s already-bound ContextVar over the process environment: session_context.py. Consequently, after a successful compression.in_place=false rotation, tools or subprocesses invoked on the caller thread can still receive the parent session ID even though self.session_id and the process environment point to the child.

The caller should explicitly bind the new session ID in its own context after successful rotation, alongside the logging repair. This needs a behavioral test in which a post-compression tool reads HERMES_SESSION_ID and receives the child ID.

6. Four hung summaries permanently exhaust the process-wide compression pool

The new executor is process-wide and fixed at four workers: conversation_compression.py. Its daemon behavior changes shutdown handling, but it retains the standard executor’s unbounded queue semantics.

Four hung summaries occupy all four workers. A fifth compression is queued, waits for its timeout, and returns without starting. Because the timeout path does not cancel a not-yet-running future, that stale queued job remains eligible to run later if any worker recovers. It can then perform an expensive summary even though its fence has already been canceled, because the first fence check is after the summary operation.

At minimum:

• Cancel futures that have not started.
• Check the canceled fence before doing expensive summary work.
• Add bounded admission/backpressure rather than an unbounded queue.
• Define the recovery behavior when every worker is wedged.
• Add a saturation test with four blocked summaries plus a fifth submission, including later worker recovery.

Additional correctness and scope concerns

Activity writes are described as best-effort but are synchronous on critical paths

Activity persistence and turn-end label clearing run synchronously from the conversation path: run_agent.py, including the finally path at turn completion.

Routine database writes can wait for the normal write-patience budget, currently up to roughly 20 seconds under contention: hermes_state.py. This means an otherwise completed reply can be delayed substantially just to clear observational activity labels, potentially creating the appearance of the stall this feature is intended to detect.

These writes should have a short/nonblocking busy budget or be moved off the response-critical path. The clear path should also avoid a transaction when the labels are already empty.

The watchdog can send /new using a stale activity snapshot

The watchdog gathers pending/activity candidates and later sends the recovery instruction without revalidating the selected session immediately before delivery: gateway/run.py.

An agent can make progress after the initial read but before the send, causing a false /new during an actively recovering turn. Re-read the pending state and current activity timestamp/provenance immediately before delivery, then abort and clear/re-arm the latch if the candidate is no longer stale. This race needs a behavioral regression.

The configured “idle timeout” can allow nearly twice that amount of silence

The worker wait uses a full timeout interval, then checks whether progress changed. If progress occurs shortly after the start of an interval, the first wait can consume nearly the whole interval and the next full wait starts from the check rather than from the actual progress timestamp. Silence since the last progress event can therefore approach two configured idle intervals.

The remaining wait should be computed as idle_timeout - elapsed_since_last_progress, or progress should wake the waiter directly.

The watchdog scope is narrower than “gateway session stall” in general

Pending work is inferred from adapter _pending_messages and _queued_events. This does not necessarily include all states that can make a durable session busy or waiting, such as startup restoration, build sentinels, the shared turn lease, debounce state, direct/API waiters, or pending work owned by another process. Routing-key aliases may also represent the same durable session in separate adapter queues.

That may be a valid initial scope, but the contract should be described precisely as recovery for an AIAgent with an adapter-queued follow-up, rather than a general detector for every gateway/session stall. The 60-second cadence is also per AIAgent instance, not globally coordinated per durable session.

Running all compression on pooled threads introduces an undocumented thread-safety requirement

The change moves previously synchronous compression, including plugin/legacy context engines and memory providers, onto arbitrary pooled threads. The relevant extension interfaces do not currently appear to promise thread safety or thread affinity. The session-context issue above demonstrates that the caller/worker distinction already affects real state.

Either audit and document the thread-safety contract for these components, or constrain the detached work so only isolated computation runs in the pool and stateful publication remains on the owning thread.

Import/export behavior should be explicit

Session export includes the new activity fields, while import does not restore them. Resetting live activity during import is probably preferable to resurrecting stale “working” state, but it should be an explicit contract with a regression rather than an accidental asymmetry.

PR/history integration

• This is the replacement for #73031. The old duplicate should be closed as superseded only after this replacement is accepted.
• The lifecycle behavior materially overlaps #74449.
• Durable lease cancellation and executor saturation materially overlap #71569. That branch is stale, so the relevant invariants should be transplanted/reconciled rather than merging it blindly.
• If this PR is intended to finish the earlier session-resilience umbrella, the cumulative SSE retry-deadline item is not implemented here and should remain separately tracked.
• git diff --check also reports trailing whitespace in contributors/emails/fangliquan@qq.com; that is minor housekeeping.

Minimum merge gates

1. Make commit-phase observation independent of the commit-held lock and prove the overrun warning fires while commit remains blocked.
2. Revoke commit admission on every host unwind path, not only the idle-timeout path.
3. Isolate worker inputs and intermediate state; publish only during an admitted commit.
4. Allow host timeout to cancel/release the timed-out holder’s durable lease without creating an ABA/stale-holder release race.
5. Prevent late workers from clearing cooldown or publishing session/compression state.
6. Repair the caller’s session ContextVar after successful out-of-place rotation.
7. Add executor saturation/backpressure and cancel queued stale futures.
8. Revalidate activity and pending work immediately before watchdog delivery.
9. Keep observational heartbeat writes from blocking the response-critical path for the normal database write-patience interval.
10. Add behavioral regressions for all of the above, especially while the blocked worker remains blocked rather than only after the test releases it.

The activity/watchdog work is directionally solid, but the compression timeout architecture currently changes “the caller can hang forever” into “the caller may return while an unbounded stale worker still owns shared and durable state.” I would keep this in draft/request-changes until those ownership and cancellation guarantees are made explicit and tested.

[teknium1]

Response to @helix4u's review of #76354 — point-by-point

All six blocking findings, all "additional correctness and scope" items, and
all 10 minimum merge gates are addressed in 10 reviewable commits atop the
rebased branch (one logical concern per commit). Every new regression asserts
the BLOCKED/hung state itself — workers are released only after the
assertions — and each one was sabotage-verified (fix reverted → test red;
fix restored → green).

Commit map

Commit	Concern	Files
fix(agent): observe commit phase without the fence lock (review F1)	F1	conversation_compression.py, gateway/run.py, tests/agent/test_compression_review_76354.py
fix(agent): revoke commit admission on every host unwind (review F2)	F2	conversation_compression.py, gateway/run.py, tests
fix(agent): isolate the pooled compression worker from the live transcript (review F3)	F3	run_agent.py, conversation_compression.py (docstring contract), context-engine-plugin.md, tests/agent/test_compression_worker_isolation_76354.py
fix(agent): holder-qualified durable lease cancellation, cooldown ordering (review F4)	F4 (Co-authored-by: ciabata-git, PR #71569 transplant)	conversation_compression.py, context_compressor.py, gateway/run.py, tests
fix(agent): rebind the caller's session ContextVar after out-of-place rotation (review F5)	F5	run_agent.py, tests
fix(agent): bounded admission + stale-job cancellation for the compress pool (review F6)	F6	conversation_compression.py, tests
fix(agent,gateway): charge the idle wait from the last progress event (review S3)	S3	conversation_compression.py, gateway/run.py, tests
fix(state): sub-second busy budget for observational activity writes (review S1)	S1	hermes_state.py, tests/gateway/test_watchdog_review_76354.py
fix(gateway): revalidate stall candidate immediately before /new delivery (review S2)	S2	gateway/run.py, tests
docs(gateway): precise watchdog scope, explicit import-resets-activity contract (review S4)	S4 + housekeeping	config_defaults.py, hermes_state_portability.py, contributors/emails/fangliquan@qq.com, tests

Findings → what changed

F1 — overrun warning couldn't fire while a commit was hung. Added a
lock-free phase marker (threading.Event) set inside begin_commit() while
the fence lock is held, cleared in finish_commit(), exposed as
CompressionCommitFence.commit_in_flight. Both host waits (sync wrapper +
gateway hygiene) break their cancel spin on commit_in_flight, so the
bounded overrun loop — WARNING/ERROR log + one-shot on_commit_overrun — is
reachable WHILE the commit is blocked.
Regression: TestF1CommitOverrunWhileHung::test_overrun_warning_fires_while_commit_still_blocked
asserts callback + WARNING with the event-gated fake commit still blocked
(release only after asserting — the exact test-shape you flagged).

F2 — non-timeout unwind left an unfenced worker. The whole host wait is
now try/finally; any exit that didn't settle the worker calls the new
lock-free fence.revoke_commit_admission() (begin_commit re-checks under
the fence lock; an admitted commit is never abandoned). The gateway hygiene
wait gets a BaseException handler doing the same + deferring agent cleanup.
Reconciliation with #74449: that PR routes explicit host interrupts into
aux-call cancellation (the fast path); this admission revocation is the outer
host-side safety net — they compose, we did not duplicate its fence
extensions.
Regressions: KI + generic-exception unwinds assert fence.is_cancelled while
the worker is provably still blocked pre-commit, then prove begin_commit()
is refused.

F3 — worker shared the live transcript. _compress_context's pooled
worker now deep-snapshots messages on the worker thread; the caller's list
object never reaches pooled engine code. Publication to caller-visible state
happens only via the returned value of an admitted commit; durable mutation
was already fence-gated. Thread-safety contract for context engines / memory
providers documented in the module docstring and
website/docs/developer-guide/context-engine-plugin.md.
Regression: in-place-mutating engine + host timeout → live transcript
byte-identical while the engine is still blocked (test_f3_...).

F4 — durable lock retained by a hung worker; late cooldown clear.
Transplanted #71569's holder-qualified lease invariants with credit
(Co-authored-by: ciabata-git): begin_lock_setup/finish_lock_setup fence
the acquire→hook-publication gap, register_cancelled_lock_release publishes
an idempotent holder-scoped release, the refresher start is serialized with
the release path, and the host invokes the hook on idle timeout, hygiene
timeout, and every unwind. No ABA: release_compression_lock is
holder-qualified (DELETE … WHERE holder = ?), so a stale release can never
free a new holder's lease. Ordering: the compressor consults a
fence-cancellation check BEFORE _clear_compression_failure_cooldown, and the
fence is checked before any publication.
Regression: test_f4_five_step_stale_holder_regression is your exact 5-step
scenario — new compressor acquires the durable lock while the old summary is
still event-blocked (asserted before release), then the released old worker
can't clear cooldown / release the new lease / publish.

F5 — stale caller ContextVar after rotation. After the pooled wrapper
returns, the caller rebinds HERMES_SESSION_ID in its own context via
set_current_session_id (alongside the existing logging repair).
Regression: test_f5_session_contextvar_rebound_after_rotation — with a
gateway-style bound session context, a post-compression
get_session_env("HERMES_SESSION_ID") on the caller thread returns the child.

F6 — pool exhaustion / stale queued jobs. Bounded admission at the worker
count: a saturated pool refuses the submission fast (unchanged messages, loud
warning) instead of queueing; slots free via a future done-callback;
not-yet-started futures are cancel()ed on timeout; the cancelled fence is
checked BEFORE expensive summary work both in the pooled wrapper and inside
compress_context. Recovery contract documented at
_COMPRESS_EXECUTOR_MAX_WORKERS: while wedged, new work fails fast; wedged
workers are fence-cancelled and restore service on return; a worker that
never returns costs its slot (bounded, observable degradation).
Regression: test_saturated_pool_fails_fast_and_never_runs_stale_job —
4 event-blocked summaries, 5th refused in <1s while the 4 are provably still
blocked, refused job never runs after recovery, fresh submission then
succeeds. Plus test_cancelled_fence_skips_summary_work_before_start.

S1 — activity writes on the critical path. touch_session_activity and
clear_session_activity_labels use a dedicated _ACTIVITY_WRITE_PATIENCE_S = 0.5 budget (not the 20s routine patience); a no-op label clear skips the
transaction entirely. Regressions: contended-DB writes give up well under the
short budget; no-op clear performs zero write transactions.

S2 — stale snapshot before /new delivery. _check_session_stalls now
re-reads the adapter pending slot, overflow queue, and live activity snapshot
immediately before adapter.send, aborts and re-arms (pops) the latch when
the candidate is no longer stale. Race regressions: progress-between-scan-
and-send aborts; pending-drained aborts; genuinely-stale still delivers once.

S3 — ~2x idle silence. Both progress-aware waits compute each slice as
idle_timeout - elapsed_since_last_progress. Regression: progress early in
an interval then silence → timeout at ~1x idle, not ~2x.

S4 — scope + contracts. session_stall_timeout config docs now state the
precise contract (recovery notifier for an in-process AIAgent with an
adapter-queued follow-up; per-AIAgent cadence; notify-only). import_sessions
documents the export-includes/import-resets activity asymmetry with a
regression pinning both halves. Trailing whitespace in
contributors/emails/fangliquan@qq.com fixed.

Thread-safety of pooled extension points — documented (module docstring +
plugin guide) rather than constrained: the snapshot isolation (F3) plus
commit-fence publication gate means only isolated computation's EFFECTS can
escape the pool, which is the "state the contract" option the review offered.

Merge gate → commit → test

Gate	Commit	Test
0. Rebase onto current origin/main	branch rebased before the fix series (was 22 behind)	full suite re-run post-rebase
1. Commit-phase observation without the lock; warning fires while blocked	F1 commit	TestF1CommitOverrunWhileHung::test_overrun_warning_fires_while_commit_still_blocked, test_commit_in_flight_marker_is_lock_free
2. Revoke commit admission on every host unwind	F2 commit	TestF2HostUnwindRevokesAdmission (ki + generic)
3. Isolate worker inputs/intermediate state; publish only on admitted commit	F3 commit	test_f3_mutating_engine_cannot_touch_live_transcript_after_timeout
4. Host timeout releases the holder's lease without ABA	F4 commit	test_f4_five_step_stale_holder_regression (steps 2–3, 5b)
5. Late workers can't clear cooldown or publish	F4 commit	test_f4_five_step_stale_holder_regression (steps 5a, 5c), TestF4CooldownClearOrdering
6. Repair caller's session ContextVar after rotation	F5 commit	test_f5_session_contextvar_rebound_after_rotation
7. Executor saturation/backpressure + cancel queued stale futures	F6 commit	TestF6ExecutorSaturation (both tests)
8. Revalidate activity/pending immediately before watchdog delivery	S2 commit	test_s2_progress_between_scan_and_send_aborts_delivery, test_s2_pending_drained_..., test_s2_still_stale_...
9. Heartbeat writes off the write-patience interval	S1 commit	test_s1_contended_activity_write_gives_up_within_short_budget, test_s1_contended_clear_..., test_s1_clear_labels_noop_skips_transaction
10. Behavioral regressions asserting the blocked state	all test files above	every hung-state test asserts before releasing its worker; all sabotage-verified

Validation

• Full affected surface (PR's listed set + compression suites + watchdog
  suites + tests/state/ + hygiene + the three new regression files): green.
• Per-commit: each of the 10 commits was checked out in a worktree and its
  cumulative regression files + the progress-timeout suite pass at every
  stage (17 → 33 tests, all green).
• ruff check on all touched files: clean. git diff --check: clean.
• Sabotage runs: F1/F2/F6/S3 (5 red), F4-hook/F4-cooldown (red),
  F3/F4-lease/F5 (3 red), S1/S2 (5 red) — all restored green.

Gaps / caveats (nothing silently skipped)

• Gate 0: rebase done; if origin/main moves again before merge a fresh
  rebase may be needed (mechanical).
• F6 "recovery when a worker NEVER returns": defined and documented as
  bounded degradation (slot lost for process lifetime) rather than thread
  murder — Python offers no safe way to kill a wedged thread; this is the
  documented contract the review asked for, not an evasion.
• S3: the waiter is not woken instantly ON progress (we compute the
  remaining wait instead, the review's first suggested option); worst-case
  silence is now bounded by ~1x idle + one scheduling quantum.
• fix(compression): let explicit interrupts cancel safely #74449 reconciliation is by composition (its aux-cancel fast path +
  our fence revocation safety net), not by merging its diff into this PR.