Repository navigation
fix(agent-server): keep streaming deltas feeding the activity heartbeat - #4712
Draft
all-hands-bot wants to merge 1 commit into
Draft
all-hands-bot wants to merge 1 commit into
all-hands-bot wants to merge 1 commit into
Conversation
PR #4689 made streaming deltas opt-in for subscribers, but the internal _EventSubscriber was not opted in. That subscriber doubles as the activity heartbeat: it refreshes both the idle-eviction clock and the runtime-api idle timer (update_last_execution_time). Detaching it from the token stream means a long-running streaming generation never refreshes the runtime idle clock, so the runtime-api can kill the pod mid-generation (~20 min idle timeout). Opt _EventSubscriber into streaming deltas so it keeps refreshing activity, while webhooks/telemetry still skip deltas. Co-authored-by: openhands <openhands@all-hands.dev>
Contributor
Python API breakage checks — ✅ PASSEDResult: ✅ PASSED |
Contributor
REST API breakage checks (OpenAPI) — ✅ PASSEDResult: ✅ PASSED |
Contributor
Coverage Report •
|
||||||||||||||||||||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
HUMAN:
AGENT:
Why
PR #4689 ("stop fanning streaming deltas out to every subscriber") made
StreamingDeltaEventdelivery opt-in per subscriber, defaulting to off. Theinternal
_EventSubscriber(inconversation_service.py) is the conversation'sactivity heartbeat — its
__call__refreshes both the idle-eviction clock(
EventService.touch()) and the runtime-api idle timer(
server_details_router.update_last_execution_time()). It was not opted in, sostreaming deltas silently stopped reaching it.
Impact (lens #9 semantic assertions / #6 background work): a long-running
streaming generation (stream=True) no longer refreshes the runtime idle clock.
After the ~20-minute idle timeout the runtime-api kills the pod mid-generation.
This is the exact failure the ACP activity heartbeat was added to prevent (see
_setup_acp_activity_heartbeat); regular SDK LLM agents relied on_EventSubscriberreceiving the token stream and now regress.Root cause: the opt-in filter in
PubSub.__call__drops every non-websocketsubscriber for
StreamingDeltaEvent, including the activity heartbeat, whichwas never marked as a delta consumer.
Summary
_EventSubscriberinto streaming deltas (receives_streaming_deltas = True)so the activity heartbeat keeps firing on the token stream.
deltas (the intended behavior of fix(agent-server): stop fanning streaming deltas out to every subscriber #4689 is preserved).
Issue Number
N/A
How to Test
Source PR reviewed: #4689. Risk lenses: #9 (semantic assertions) and #6
(background work / idle-eviction clocks).
Deterministic reproduction — a
StreamingDeltaEventpublished through theconversation's
PubSubmust advance both idle clocks:main, this test fails:idle_seconds()keeps advancing andtime.time() - server_details_router._last_event_timestays at 10000+ secondsbecause
_EventSubscriber.__call__never runs for a delta.Before/after test results:
Before (main, without fix):
After (with fix):
Full affected suites still green:
End-to-end validation: the regression test exercises the real
ConversationService.start_conversation→EventService._pub_sub→_EventSubscriber.__call__path (no mocks), publishing a realStreamingDeltaEventand asserting the observable invariant — the runtime idleclock is reset — rather than just "the event was emitted".
Why the original tests did not catch this: #4689's tests assert only that
deltas are filtered out of non-opted-in subscribers (webhook/telemetry) and
delivered to opted-in websockets. They never assert the activity-heartbeat
side effect of the internal subscriber, which is the mechanism that keeps the
pod alive during long streaming runs.
Video/Screenshots
N/A (server-side behavior; command output captured above).
Design Doc
Type
Notes
This PR was created by an AI agent (OpenHands) on behalf of the weekly
high-impact regression hunt. Source PR #4689 was authored by @VascoSch92 and
approved by @enyst; both are requested as reviewers.
🐳 Agent Server images for this PR — GHCR package, pull/run commands, and all pushed tags (click to expand)
• GHCR package: https://github.com/OpenHands/agent-sdk/pkgs/container/agent-server
Variants & Base Images
eclipse-temurin:17-jdknikolaik/python-nodejs:python3.13-nodejs22-slimgolang:1.21-bookwormPull (multi-arch manifest)
# Each variant is a multi-arch manifest supporting both amd64 and arm64 docker pull ghcr.io/openhands/agent-server:685a469-pythonRun
All tags pushed for this build
About Multi-Architecture Support
685a469-python) is a multi-arch manifest supporting both amd64 and arm64685a469-python-amd64) are also available if needed