Skip to content

fix(agent): terminal activity stamp to prevent stuck last_activity_at - #80832

Open
yingliang-zhang wants to merge 1 commit into
NousResearch:mainfrom
yingliang-zhang:fix/turn-end-activity-stamp
Open

fix(agent): terminal activity stamp to prevent stuck last_activity_at#80832
yingliang-zhang wants to merge 1 commit into
NousResearch:mainfrom
yingliang-zhang:fix/turn-end-activity-stamp

Conversation

@yingliang-zhang

@yingliang-zhang yingliang-zhang commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

Problem

The Desktop UI depends on last_activity_at in the session DB to detect session freshness and trigger message list reloads. When this timestamp gets stuck, the UI stops showing new messages — the user sees "output swallowed" even though the messages are correctly persisted in the DB.

Root Cause

The turn finally block called _reset_activity_labels_after_turn() which cleared last_activity_description and last_activity_provenance but never force-persisted last_activity_at.

Mid-turn heartbeats via _touch_activity() are rate-limited to one write per 60s (SESSION_ACTIVITY_HEARTBEAT_MIN_INTERVAL_SECONDS), and each write has a 0.5s patience budget (_ACTIVITY_WRITE_PATIENCE_S) to avoid blocking the response-critical path. Under DB contention (large state.db, many read connections), the 0.5s budget is exhausted and the write is silently dropped (fail-open by design).

When ALL intermediate heartbeats in a turn are either rate-limited or dropped, and the turn finally doesn't force-persist, last_activity_at stays at the last successful heartbeat — potentially many messages behind.

Fix

Replace _reset_activity_labels_after_turn() in the turn finally block with the new _finalize_activity_after_turn():

SessionDB.finalize_session_activity() (new, hermes_state.py)

  • One atomic UPDATE that stamps last_activity_at AND clears labels in the same write
  • Uses elevated write patience (2.0s vs 0.5s) — the turn is over, no response-critical path to protect
  • Monotonic guard (WHERE last_activity_at < ?) prevents backwards movement

AIAgent._finalize_activity_after_turn() (new, run_agent.py)

_reset_activity_labels_after_turn() — retained without a production caller

Once this PR lands, _reset_activity_labels_after_turn() has no production callers (compression's label handling goes through _touch_activity(..., force_persist=True) and clear_session_activity_labels, not through this function). It is retained for test-only usage and as a utility for future callers that need label-only clearing without timestamp stamping.

Tests

New test file tests/run_agent/test_finalize_activity_after_turn.py (8 tests):

All 22 tests pass (8 new + 14 existing in test_session_activity_persist.py).

@alt-glitch alt-glitch added type/bug Something isn't working comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint P2 Medium — degraded but workaround exists sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state labels Aug 7, 2026
@spfcraze

spfcraze commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

This was generated by AI during triage.

Summary:
The description says _reset_activity_labels_after_turn() is still used by compression, but compression's label handling goes through _touch_activity(..., force_persist=True) and clear_session_activity_labels — the turn finally replaced here is its only production call site.

Problems:

  • agent/conversation_compression.py:1456 stamps the compression terminal label via _touch_activity(..., force_persist=True); agent/conversation_compression.py:3430 clears the archived parent's labels via clear_session_activity_labels — neither path calls _reset_activity_labels_after_turn().
  • At main, _reset_activity_labels_after_turn() has one production caller, the turn finally in run_agent.py that this PR replaces with _finalize_activity_after_turn(); the only other reference is a test file.

Solution:
As a wording that matches the code: _reset_activity_labels_after_turn() is retained without a production caller once this lands, and compression's label paths are _touch_activity(..., force_persist=True) and clear_session_activity_labels.


Checked against de26441 — the tip of fix/turn-end-activity-stamp when this was written — and 427584b, main at the same moment.

@yingliang-zhang
yingliang-zhang force-pushed the fix/turn-end-activity-stamp branch from de26441 to 92ca79d Compare August 8, 2026 04:10
@yingliang-zhang

Copy link
Copy Markdown
Contributor Author

Thanks @spfcraze for the accurate triage — good catch. I've updated the PR description to reflect the actual code structure: _reset_activity_labels_after_turn() is retained without a production caller once this PR lands. Compression's label paths go through _touch_activity(..., force_persist=True) and clear_session_activity_labels, not through this function. The function is kept for test-only usage and as a utility for future callers that need label-only clearing without timestamp stamping.

@yingliang-zhang

Copy link
Copy Markdown
Contributor Author

Rebased onto latest main (bccb7972df).

@yingliang-zhang
yingliang-zhang force-pushed the fix/turn-end-activity-stamp branch from 47862b2 to 441dccd Compare August 19, 2026 03:15
The turn finally block called _reset_activity_labels_after_turn() which
cleared description/provenance but never force-persisted last_activity_at.
When all intermediate heartbeats were rate-limited (60s cadence) or
silently dropped (0.5s write patience under DB contention), the timestamp
stuck at the last successful heartbeat — breaking Desktop UI freshness
detection and hiding the last N messages from the user.

Fix: replace _reset_activity_labels_after_turn() in the turn finally with
_finalize_activity_after_turn(), which:
- Calls new SessionDB.finalize_session_activity() — one atomic UPDATE
  that stamps last_activity_at AND clears labels in the same write
- Uses elevated write patience (2.0s vs 0.5s) since the turn is over
  and there is no response-critical path to protect
- Does NOT call _touch_activity (avoids kanban bridge side-effect that
  could trigger a spurious continuation turn on a finished turn)
- Does NOT bump in-memory _last_activity_ts (preserves NousResearch#15654
  interrupt-recursive watchdog continuity)

_reset_activity_labels_after_turn() is kept unchanged — compression and
other consumers still use it.

Tri-model review: K3/GLM/DSF 3/3 NEEDS_FIXES on original proposal,
converged on (b)+(c) combined approach (dedicated method + atomic write).
@yingliang-zhang
yingliang-zhang force-pushed the fix/turn-end-activity-stamp branch from 1286f49 to 14cac94 Compare August 20, 2026 16:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint P2 Medium — degraded but workaround exists sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants