fix(gateway): gateway sessions lack activity watchdog - agent loop stalls silently with no detection and notification

[fangliquanflq]

What does this PR do?

Re-lands the session activity watchdog work after #72858 reverted #72817 (salvage of #72424).

Maintainer note on the revert: the original work is solid; the revert was a process correction because #72817 landed before a full review. This PR restores the same change set and is intended to go through the normal review cycle before merge.

Cherry-picks 3 commits from #72817 (skips the redundant noreply contributor mapping; fangliquan@oppo.com is already mapped):

1. feat: session activity heartbeats, stall notify, compress_context timeout
2. refactor: reuse _relative_time, hygiene provenance helper, record_timeout_failure()
3. fix: propagate logging session context after daemon-pool compress_context

Behavior (unchanged from #72817 / #72424):

1. Mid-turn activity heartbeats to SessionDB so hermes sessions list / hermes status show progress during long turns
2. Stall watchdog: busy session + pending inbound + idle past agent.session_stall_timeout (default 300) -> WARNING + one-shot /new notify (notify-only)
3. Progress-aware compaction timeout for fenceless compress_context callers (compression.context_timeout_seconds / compression.context_total_ceiling_seconds)

Does not include #72016 item 4 (cumulative SSE stream-retry deadline); that remains a follow-up.

Related Issue

Related to #72016 (closed when #72817 merged; code was later reverted by #72858)
Supersedes / re-opens the review path for #72424 and #72817

Type of Change

• Bug fix (non-breaking change that fixes an issue)
• New feature (non-breaking change that adds functionality)

Changes Made

• Same file set as fix(gateway): session activity watchdog, stall notify, compress timeout (#72424 salvage) #72817 (activity contract, stall watcher, compress timeout, config/docs/tests, salvage refactors, CI logging-context fix)

How to Test

1. Activity: during a long gateway turn, confirm hermes sessions list / hermes status advance last_active without a new message row
2. Stall notify: busy stalled turn with queued follow-up -> WARNING + one /new notification; no spam on later ticks
3. Compaction: hung/silent auxiliary model during auto-compress -> warning, no message drop, turn continues
4. Automated:

scripts/run_tests.sh \
  tests/agent/test_compress_context_progress_timeout.py \
  tests/agent/test_compression_concurrent_fork.py \
  tests/agent/test_session_activity.py \
  tests/run_agent/test_session_activity_persist.py \
  tests/gateway/test_session_stall_watchdog.py \
  tests/gateway/test_config_env_bridge_authority.py \
  tests/hermes_cli/test_status.py \
  tests/test_hermes_state.py \
  tests/run_agent/test_compression_boundary_hook.py \
  -q

Checklist

Code

• I've read the Contributing Guide
• My commit messages follow Conventional Commits
• I searched for existing PRs to make sure this isn't a duplicate
• My PR contains only changes related to this fix/feature
• I've run scripts/run_tests.sh on relevant tests and they pass
• I've added tests for my changes
• I've tested on my platform: Windows 10/11 (WSL test runner)

Documentation & Housekeeping

• I've updated relevant documentation
• I've updated cli-config.yaml.example if I added/changed config keys
• I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows - N/A
• I've considered cross-platform impact
• I've updated tool descriptions/schemas if I changed tool behavior - N/A

[fangliquanflq]

@kshitijk4poor thanks again for the salvage/refactor work on the #72817 re-land path.

Could you take another look when you have a chance? A few follow-ups landed on top of the restored watchdog set:

1. Silence detached compression heartbeat after host timeout, so a timed-out daemon compress_context cannot re-arm agent.compression / poison stall clocks.
2. Force-persist compression completed past the SessionDB 60s rate limit, so /compress does not leave durable activity stuck on "in progress".

Happy to address any further review notes.

[teknium1]

Thanks for restoring the watchdog and activity work. The current-main premise remains valid: active sessions queue follow-ups in gateway/platforms/base.py:5712-5724, while current startup has no session-stall watcher (gateway/run.py:10991-11073).

Problems

• agent/conversation_compression.py:458-464 does not honor the advertised total compression ceiling after begin_commit(): it spins until the fence lock is available and then calls unbounded future.result(). The commit fence covers SessionDB mutation such as archive_and_compact() at agent/conversation_compression.py:2349 and publish_compression_child() at line 2412, so a hang there can still stall the turn indefinitely. The new test explicitly accepts an over-ceiling commit wait at tests/agent/test_compress_context_progress_timeout.py:125-149.

Suggested changes

• Bound or otherwise make the commit phase recoverable, or narrow the documented/configured timeout guarantee and add a test for a commit-phase hang.

This is an automated hermes-sweeper review.

[kshitijk4poor]

Closing as superseded by #76354 — @teknium1's clean re-land of the session activity watchdog onto current main, with the commit-phase ceiling fix and heartbeat write discipline composed with recent SessionDB merges. Your original work was solid (Teknium noted the revert was a process correction); #76354 carries it forward. Thanks!

[teknium1]

Superseded by PR #76354 (#76354), now merged — your full commit sequence was cherry-picked as its base with your authorship preserved via rebase-merge (the #72858 revert was process-only, and your work survived it intact).

On top of your base, the merged PR carries two adversarial review rounds' worth of hardening (bounded commit-phase waits, worker transcript isolation, holder-qualified lease cancellation, bounded pool admission, watchdog send bounds) plus live-test verification of the heartbeat cadence, stall notify, and compression timeout behavior.

Thanks for building the watchdog and for persisting through the revert — the activity/heartbeat design was validated as-is by both reviewers. Closing this re-land as superseded.