Skip to content

feat(hermes_state): Gateway session activity tracking so silent stalls are visible - #72039

Closed
fangliquanflq wants to merge 8 commits into
NousResearch:mainfrom
fangliquanflq:fix/gateway-session-activity-tracking
Closed

feat(hermes_state): Gateway session activity tracking so silent stalls are visible#72039
fangliquanflq wants to merge 8 commits into
NousResearch:mainfrom
fangliquanflq:fix/gateway-session-activity-tracking

Conversation

@fangliquanflq

@fangliquanflq fangliquanflq commented Jul 26, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?

Adds durable mid-turn Gateway session activity tracking so operators can see when a gateway agent turn is still making API/tool/compaction progress, even before new message rows land. This implements suggested feature from #72016 (activity tracking + CLI exposure). It does not add the stall watchdog, compaction timeout, or cumulative stream-retry deadline from that issue.

Motivation

AIAgent already updated an in-memory activity clock on API/tool/compaction work, and the gateway used that for in-process inactivity timeouts. SessionDB last_active for hermes sessions list / hermes status came only from message timestamps (or started_at). During a long or stalled turn with no new messages, CLI surfaces looked idle and gave operators no durable activity signal.

How to see the gap

  1. Run hermes gateway with a messaging platform configured.
  2. Start a turn that spends a long time in API retries, tools, or compaction without writing assistant messages yet.
  3. In another terminal, run hermes sessions list and hermes status.

Desired: listings/status reflect recent agent activity (API/tool/compaction).
Without this feature: Last Active / session status stayed frozen on the last message time; status only showed an active session count.

What this feature adds

  • sessions.last_activity_at and SessionDB.touch_session_activity.
  • Rate-limited (60s) durable stamps from AIAgent._touch_activity (same cadence posture as the kanban heartbeat bridge).
  • Listing last_active as freshest of heartbeat and latest message timestamp (so a lagging heartbeat cannot rank behind newer messages).
  • Freshest gateway Last activity line in hermes status.

Related Issue

Related to #72016 (implements suggested item #1: Gateway session activity tracking). Does not close the issue; stall watchdog / compaction timeout / cumulative retry deadline remain open.

Type of Change

  • ✨ New feature (non-breaking change that adds functionality)

Changes Made

  • hermes_state.py - last_activity_at column, touch helper, freshest-of last_active SQL for list/status/archive paths
  • run_agent.py / agent/agent_init.py - rate-limited durable persist from _touch_activity
  • hermes_cli/status.py - print freshest gateway last activity
  • tests/run_agent/test_session_activity_persist.py, tests/test_hermes_state.py, tests/hermes_cli/test_status.py - coverage for persist cadence, freshest-of ranking, status line

How to Test

  1. Manual: during a long gateway turn, confirm hermes sessions list advances Last Active from heartbeats and hermes status shows Last activity.
WechatIMG685
  1. Automated (already run locally, 23 passed):
scripts/run_tests.sh \
  tests/run_agent/test_session_activity_persist.py \
  tests/hermes_cli/test_status.py \
  tests/test_hermes_state.py::TestListSessionsRich \
  -q

Checklist

Code

  • I've read the Contributing Guide
  • My commit messages follow Conventional Commits
  • I searched for existing PRs to make sure this isn't a duplicate
  • My PR contains only changes related to this fix/feature
  • I've run scripts/run_tests.sh on relevant tests and they pass
  • I've added tests for my changes
  • I've tested on my platform: Windows 11

Documentation & Housekeeping

  • I've updated relevant documentation - N/A (docstrings only)
  • I've updated cli-config.yaml.example if I added/changed config keys - N/A
  • I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows - N/A
  • I've considered cross-platform impact - N/A (SQLite + CLI paths are cross-platform)
  • I've updated tool descriptions/schemas if I changed tool behavior - N/A

Stamp sessions.last_activity_at from AIAgent._touch_activity (rate-limited)
so hermes sessions list and hermes status observe API/tool/compaction
progress while a gateway turn is still writing no message rows (NousResearch#72016).
Prefer max(last_activity_at, latest message) for session last_active so a
rate-limited mid-turn heartbeat cannot rank a session behind newer message
timestamps in listings, status, and idle archive.
…t_active

Document that last_active is max(heartbeat, message time), and separate the
status last-activity test from the preceding class with PEP8 spacing.
Clarify that archive_stale_sessions ages on max(last_activity_at, latest
message), matching the shared last_active SQL helper.
@alt-glitch alt-glitch added type/feature New feature or request P3 Low — cosmetic, nice to have comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint comp/cli CLI entry point, hermes_cli/, setup wizard area/sessions Session lifecycle, resume, persistence, history sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades labels Jul 26, 2026
@fangliquanflq fangliquanflq changed the title feat(hermes_state): Gateway session activity tracking so silent stalls are visible in CLI feat(hermes_state): Gateway session activity tracking so silent stalls are visible Jul 26, 2026
Extend the shared session activity observation contract so durable SessionDB heartbeats carry description/provenance alongside the timestamp, matching in-memory get_activity_summary for NousResearch#72039 consumers.
Keep gateway cache turn resets on the ts/desc/provenance triple, and drop the TypeError persist fallback that could silently omit description/provenance.
Keep last_activity_at for idle/watchdog continuity, but wipe description/provenance in memory and SessionDB so idle sessions do not keep advertising the last mid-turn stamp.
fangliquanflq added a commit to fangliquanflq/hermes-agent that referenced this pull request Jul 27, 2026
…chdog

Use AIAgent.get_activity_summary() / agent.session_activity as the only
progress source for pending-inbound stall notify. Drop turn-start and
pending-event clocks so NousResearch#72079 stays composable with NousResearch#72039.
…ctivity-tracking

Resolved hermes_state.py search_sessions docstring conflict: keep freshest-of last_active semantics from this branch and workspace_key docs from main.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/sessions Session lifecycle, resume, persistence, history comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint comp/cli CLI entry point, hermes_cli/, setup wizard P3 Low — cosmetic, nice to have sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state type/feature New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants