Skip to content

fix(gateway): session activity watchdog, stall notify, compress timeout (#72424 salvage) - #72817

Merged
kshitijk4poor merged 4 commits into
NousResearch:mainfrom
kshitijk4poor:salvage/72424-activity-watchdog
Jul 27, 2026
Merged

fix(gateway): session activity watchdog, stall notify, compress timeout (#72424 salvage)#72817
kshitijk4poor merged 4 commits into
NousResearch:mainfrom
kshitijk4poor:salvage/72424-activity-watchdog

Conversation

@kshitijk4poor

@kshitijk4poor kshitijk4poor commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Summary

Gateway sessions now detect and notify when the agent loop stalls silently — activity heartbeats, a stall watchdog, and a progress-aware compress_context timeout.

Cherry-picked from #72424 by @fangliquanflq with three code-reuse fixes applied on top.

Changes

  • agent/session_activity.py (new): shared activity observation contract — timestamp + bounded description/provenance
  • gateway/session_stall.py (new): notify-once stall policy for pending inbound + stale activity
  • agent/conversation_compression.py: progress-aware timeout wrapper for in-agent compress_context (daemon pool + fence cancel)
  • run_agent.py: _touch_activity gains provenance + rate-limited SessionDB heartbeat; _compress_context wraps fenceless callers
  • hermes_state.py: touch_session_activity / clear_session_activity_labels / get_session_activity; freshest-of last_active SQL helper
  • gateway/run.py: stall watcher, hygiene compression provenance stamps
  • hermes_cli/config.py / status.py / dump.py: new config keys, status surfacing
  • Tests: 690 passed across 11 files

Salvage fixes (our commits)

  1. Reuse _relative_time from hermes_cli/main.py instead of duplicating in status.py
  2. Extract _stamp_hygiene_compression_provenance helper in gateway/run.py to deduplicate two copy-paste blocks
  3. Add ContextCompressor.record_timeout_failure() and delegate to it from the compress_context timeout callback instead of re-implementing the (60, 300, 900) cooldown ladder inline

Validation

Before After
Tests — 690 passed, 0 failed
E2E — All real-import tests passed
Lint — ruff clean

Closes #72016 (slices 1-3; slice 4 cumulative SSE stream-retry deadline remains a follow-up).

fangliquanflq and others added 3 commits July 27, 2026 23:16
@fangliquanflq @kshitijk4poor
feat(gateway): session activity watchdog, stall notify, compress time…
de7c5f0 
…out (NousResearch#72424)

Three mechanisms to detect and notify when gateway sessions stall silently:

1. Mid-turn activity heartbeats stamped to SessionDB so hermes sessions list
   and hermes status show progress during long turns without new message rows.

2. Stall watchdog: when a busy session has pending inbound and the shared
   activity clock is idle past agent.session_stall_timeout (default 300),
   log a WARNING and notify the user once to try /new. Notify-only; does
   not kill the turn.

3. Compaction timeout: fenceless compress_context callers get a progress-aware
   host budget (compression.context_timeout_seconds default 120 idle,
   compression.context_total_ceiling_seconds default 600 ceiling). On timeout,
   cancel via commit fence, skip compaction without dropping messages, and
   continue the turn.

Closes NousResearch#72016 (slices 1-3; slice 4 cumulative SSE stream-retry deadline
remains a follow-up).

Cherry-picked from PR NousResearch#72424 by @fangliquanflq.
@kshitijk4poor
refactor: reuse existing utilities in salvaged PR NousResearch#72424
0590200 
Three code-reuse fixes applied during salvage:

1. Reuse _relative_time from hermes_cli/main.py instead of duplicating
   the relative-time formatting logic in hermes_cli/status.py.

2. Extract _stamp_hygiene_compression_provenance helper in gateway/run.py
   to deduplicate the two nearly-identical try/except blocks that stamp
   compression timeout/abort provenance in the hygiene path.

3. Add ContextCompressor.record_timeout_failure() method and use it from
   the in-agent compress_context timeout callback instead of re-implementing
   the (60, 300, 900) cooldown ladder inline. The existing summary-LLM
   exception handler already has this ladder — now both paths share one
   method.
@kshitijk4poor
chore: add contributor mapping for fangliquanflq (NousResearch#72424)
f99b1a4
@alt-glitch alt-glitch added type/bug Something isn't working P1 High — major feature broken, no workaround comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint comp/gateway Gateway runner, session dispatch, delivery comp/cli CLI entry point, hermes_cli/, setup wizard area/config Config system, migrations, profiles sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages labels Jul 27, 2026
@kshitijk4poor
fix: propagate logging session context after daemon-pool compress_con…
b2253f4 
…text

compress_context now runs on a daemon pool worker thread (via
run_compress_context_with_progress_timeout). The session id rotation
updates hermes_logging._session_context (a threading.local) on the
WORKER thread, not the caller thread. After the wrapper returns,
propagate self.session_id back to the caller's logging context so
subsequent log lines carry the rotated id (NousResearch#34089).

Fixes CI failure in test_compression_logging_session_context.
@kshitijk4poor
kshitijk4poor merged commit 1f405aa into NousResearch:main Jul 27, 2026
38 checks passed
This was referenced Jul 27, 2026
Closed
Merged
kshitijk4poor added a commit that referenced this pull request Jul 27, 2026
@kshitijk4poor
revert: PR #72817 — session activity watchdog, stall notify, compress…
2c1809e 
… timeout

Reverting #72817 (salvage of #72424) pending further review.
All 4 commits reverted: feat, refactor, chore (contributor map), CI fix.
kshitijk4poor added a commit that referenced this pull request Jul 27, 2026
@kshitijk4poor
Merge pull request #72858 from kshitijk4poor/revert/72817
5646fed 
revert: PR #72817 — session activity watchdog, stall notify, compress timeout
@fangliquanflq fangliquanflq mentioned this pull request Jul 28, 2026
Closed
14 tasks
@kshitijk4poor
kshitijk4poor deleted the salvage/72424-activity-watchdog branch August 5, 2026 07:10
randlee pushed a commit to randlee/hermes-agent that referenced this pull request Aug 11, 2026
@kshitijk4poor
revert: PR NousResearch#72817 — session activity watchdog, stall noti…
d22f462 
…fy, compress timeout

Reverting NousResearch#72817 (salvage of NousResearch#72424) pending further review.
All 4 commits reverted: feat, refactor, chore (contributor map), CI fix.
randlee pushed a commit to randlee/hermes-agent that referenced this pull request Aug 11, 2026
@kshitijk4poor
Merge pull request NousResearch#72858 from kshitijk4poor/revert/72817
d735cd3 
revert: PR NousResearch#72817 — session activity watchdog, stall notify, compress timeout
33hodl pushed a commit to 33hodl/hermes-agent that referenced this pull request Aug 12, 2026
@kshitijk4poor
revert: PR NousResearch#72817 — session activity watchdog, stall noti…
1b07e20 
…fy, compress timeout

Reverting NousResearch#72817 (salvage of NousResearch#72424) pending further review.
All 4 commits reverted: feat, refactor, chore (contributor map), CI fix.
33hodl pushed a commit to 33hodl/hermes-agent that referenced this pull request Aug 12, 2026
@kshitijk4poor
Merge pull request NousResearch#72858 from kshitijk4poor/revert/72817
111db9b 
revert: PR NousResearch#72817 — session activity watchdog, stall notify, compress timeout
prmartinow pushed a commit to prmartinow/hermes-agent that referenced this pull request Aug 26, 2026
@kshitijk4poor
revert: PR NousResearch#72817 — session activity watchdog, stall noti…
fe8c91e 
…fy, compress timeout

Reverting NousResearch#72817 (salvage of NousResearch#72424) pending further review.
All 4 commits reverted: feat, refactor, chore (contributor map), CI fix.
prmartinow pushed a commit to prmartinow/hermes-agent that referenced this pull request Aug 26, 2026
@kshitijk4poor
Merge pull request NousResearch#72858 from kshitijk4poor/revert/72817
127c0a1 
revert: PR NousResearch#72817 — session activity watchdog, stall notify, compress timeout
melon-xf added a commit to melon-xf/hermes-agent that referenced this pull request Sep 3, 2026
@melon-xf
revert: PR NousResearch#72817 — session activity watchdog, stall noti…
317abc8 
…fy, compress timeout

Reverting NousResearch#72817 (salvage of NousResearch#72424) pending further review.
All 4 commits reverted: feat, refactor, chore (contributor map), CI fix.
melon-xf added a commit to melon-xf/hermes-agent that referenced this pull request Sep 3, 2026
@melon-xf
Merge pull request NousResearch#72858 from kshitijk4poor/revert/72817
6208d2e 
revert: PR NousResearch#72817 — session activity watchdog, stall notify, compress timeout
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

area/config Config system, migrations, profiles comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint comp/cli CLI entry point, hermes_cli/, setup wizard comp/gateway Gateway runner, session dispatch, delivery P1 High — major feature broken, no workaround sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state type/bug Something isn't working

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Gateway sessions lack activity watchdog — agent loop stalls silently with no detection or notification

3 participants

@kshitijk4poor @alt-glitch @fangliquanflq