fix(stream): bound a single streaming call and ignore keep-alive frames (#83657)

[JoaoMarcos44]

Fixes #83657.

What the issue really contains

Two independent failures were reported together:

Symptom	Owner	Verdict
502 origin_bad_gateway cluster from inference-api.nousresearch.com	Provider edge (Cloudflare/origin)	Upstream. Already handled: classify_api_error marks it retryable and the fallback chain swaps providers. Nothing to fix client-side.
One chat.completions call ran 1239s and pushed a cron run past its window	Hermes	Real client-side bug. Two guards were missing. Fixed here.

Root cause

1. Keep-alive frames counted as progress

The stale-stream detector's own comment promises it kills "connections kept alive by SSE pings but no actual data". It could not: the poll loop stamped last_chunk_time on every event, so an Anthropic ping or an empty OpenAI-style delta refreshed its patience. A provider heartbeating into the void was indistinguishable from one generating tokens.

stream_chunk_is_progress now decides what counts:

• progress — content, reasoning / reasoning_content, tool-call or function-call deltas, a role opener, a finish_reason, usage, or any non-empty vendor field in model_extra
• not progress — Anthropic ping events, deltas whose every field is empty, chunks with neither choices nor usage
• unknown shapes count as progress, so an unfamiliar provider is never starved of patience

2. Nothing bounded a single streaming call

Every guard Hermes had measures silence and resets on provider traffic:

• the stale-stream detector (HERMES_STREAM_STALE_TIMEOUT)
• the agent activity tracker (_touch_activity)
• the gateway inactivity monitor (HERMES_AGENT_TIMEOUT)
• the cron leash (HERMES_CRON_TIMEOUT)

A provider dripping real tokens resets all four and can hold one call open indefinitely. That is the exact shape of the 1239s call — and the streaming path was the only one without a ceiling: the non-streaming path has its 90s stale budget, the codex path has a hard ceiling (#64507), and auxiliary_client has total_timeout.

resolve_stream_hard_timeout adds the one bound activity cannot push back.

%%{init: {'theme': 'dark', 'themeVariables': { 'primaryColor': '#00f0ff', 'mainBkg': '#0a0a16', 'primaryTextColor': '#ffffff', 'primaryBorderColor': '#ff007f', 'lineColor': '#00f0ff'}}}%%
graph TD
    A[🔌 Provider Stream Event] --> B{⚡ stream_chunk_is_progress?}
    B -->|Content / Reasoning / Tool Call / Finish / Usage| C[🟢 Refresh Idle Window]
    B -->|Anthropic ping / Empty Delta| D[🚫 Ignored - Window Keeps Aging]
    C --> E{⏱️ Idle > Stale Timeout?}
    D --> E
    E -->|Yes| F[🔪 Stale Kill - Reconnect + Breaker Bump]
    E -->|No| G{🧭 Total Elapsed > Wall-Clock Ceiling?}
    G -->|No| A
    G -->|Yes| H[🛑 Mark Request Cancelled #6600]
    H --> I[🔌 Abort Transport - Stranger-Thread Safe #29507/#67142/#70773]
    I --> J[🚀 TimeoutError to Retry / Provider Fallback]

Loading

Infographic :

Design decisions

• Default = HERMES_API_TIMEOUT (1800s), floored at the stale timeout. The ceiling must never preempt the stale detector, which owns reconnect + diagnostics. The floor applies only to the derived default — an explicit max_call_seconds / HERMES_STREAM_MAX_CALL_SECONDS is honored verbatim, including small values.
• Local endpoints are exempt unless configured explicitly (is_local_endpoint), so long local generations on Ollama / llama.cpp are not newly killed.
• 0 disables it.
• Raise, don't reconnect. The stale path reconnects in place; the ceiling does not. Reconnecting would hand the same wedged provider a brand-new budget — retry / provider fallback is the correct owner.
• No new watchdog thread. The check rides the existing 0.3s poll loop, so there is nothing extra to leak or join.
• Teardown follows the established contracts: _request_cancelled is set before the abort (fix(agent,gateway): voice interrupts + cascading interrupt hang #6600) so the worker reads its own torn transport as a cancellation, and the abort uses the stranger-thread path that never releases FDs from the poll thread (Interrupted OpenAI/httpx request thread survives across turns and writes TLS record bytes to unrelated file descriptors on delayed close #29507 / Anthropic stale-stream watchdog can still corrupt SQLite via TLS FD reuse #67142 / Custom OpenAI-compatible provider still corrupts SQLite via TLS FD reuse in v0.18.2 — streaming stale watchdog closes shared client pool from non-owner thread #70773).

hermes_cli/timeouts.py gained get_provider_max_call_timeout on top of a _lookup_provider_timeout helper extracted from the two existing lookups — same behavior, same module paths, so existing patch("hermes_cli.timeouts.get_provider_request_timeout") call sites are unaffected.

Regression surface checked

• Sessions permanently stuck in Stream stale loop - never recovers (OpenAI-compatible provider) #58962 cross-turn stale breaker — keep-alive frames now reach the breaker instead of bypassing it; reset paths untouched.
• Reasoning models pausing mid-stream — unchanged: the stale timeout and its reasoning floor still govern silence, and the ceiling is floored at it.
• Local providers — exempt by default (HERMES_LOCAL_STREAM_STALE_TIMEOUT behavior untouched).
• Usage-only final chunks and role-opener deltas explicitly classified as progress so a well-behaved tail is never mistaken for a heartbeat.
• Relay / MoA _accept_stream_chunk shares the same predicate, so the two paths cannot drift.

Test plan

New: tests/agent/test_stream_progress_and_ceiling_83657.py (26 unit tests) and tests/run_agent/test_stream_drip_watchdog_83657.py (4 behavioral tests driving the real poll loop with self-bounding fake streams, so a regression fails instead of hanging).

• Keep-alive-only drip trips the stale detector and bumps _consecutive_stale_streams
• Content drip keeps the idle window open (no regression for slow models)
• Endless productive stream is aborted by the ceiling with a TimeoutError and a torn-down transport
• Fast stream under the ceiling completes untouched
• Ceiling resolution: config > env > derived default, 0 disables, local exempt, explicit local value honored, config-lookup failure degrades to the default
• pytest tests/agent/test_stream_progress_and_ceiling_83657.py tests/run_agent/test_stream_drip_watchdog_83657.py tests/run_agent/test_streaming.py tests/run_agent/test_stream_stale_breaker_reset.py tests/hermes_cli/test_timeouts.py tests/agent/test_stream_read_timeout_floor.py tests/agent/test_local_stream_timeout.py → 114 passed
• Neighboring suites (test_reasoning_stale_timeout_floor, test_non_stream_stale_timeout, test_relay_llm, test_moa_trace_streamed_capture, test_aux_progress_streaming, test_anthropic_adapter, test_nous_portal_anthropic_wire) → 174 passed, 2 skipped

Docs

website/docs/reference/environment-variables.md — new HERMES_STREAM_MAX_CALL_SECONDS row; the HERMES_STREAM_STALE_TIMEOUT row now states that keep-alive frames do not reset it.
website/docs/user-guide/configuration.md — new "Streaming call ceiling" row in the API Timeouts table plus a section explaining why an activity-based guard cannot catch this failure shape.

[enzo-adami]

Reproduced against head 3fda9ec88e: provider/model config value max_call_seconds: 0 does not disable the wall-clock ceiling as the PR documents.

Minimal counterexample:

configured: 0
get_provider_max_call_timeout(...): None
resolve_stream_hard_timeout(...): 1800.0

The cause is that the new generic _lookup_provider_timeout() still routes through _coerce_timeout(), which maps every value <= 0 to None. The resolver therefore cannot distinguish “unset” from the documented explicit zero and falls through to HERMES_API_TIMEOUT / 1800 seconds.

Minimal fix: let only the max-call lookup preserve zero (for example an allow_zero coercion/lookup option), while request/stale timeout lookups keep their current positive-only contract. Please add both provider-level and model-level zero tests asserting the resolver returns inf.

The existing 30 targeted tests pass; this case is not covered. The PR also now conflicts with current main in agent/chat_completion_helpers.py.

[enzo-adami]

I prepared the minimal follow-up directly on this PR's source HEAD (3fda9ec88e), preserving the original commit author and without rebasing or opening an upstream replacement:

• Draft follow-up: fix(stream): honor zero max-call config JoaoMarcos44/hermes-agent#15
• Commit: 362d030ba3b83bb15528fff72469fd1748b2a864
• Target: JoaoMarcos44:fix/83657-stream-wallclock-ceiling

It only preserves max_call_seconds: 0 through provider/model config lookup and adds the two regressions. Request/stale timeout semantics are unchanged.

Fresh validation on the exact composed branch: 32 targeted tests pass; Ruff, Windows-footgun check, and git diff --check pass.

[enzo-adami]

Current-main handoff is now pushed without opening a duplicate upstream PR: https://github.com/enzo-adami/hermes-agent/tree/agent/pr84066-main-refresh

Base is exact main 165c889; commit 83f281c preserves joaomarcos as primary author and includes the narrow max_call_seconds=0 correction from JoaoMarcos44#15. Net scope remains the six watchdog/config/docs files.

Fresh validation: 32/32 dedicated first-byte/idle/max-call/keepalive tests, Ruff, Windows-footgun checker, and diff-check all pass. The branch is a force-update/cherry-pick handoff for #84066, not a replacement PR.

[JoaoMarcos44]

Current-main handoff is now pushed without opening a duplicate upstream PR: https://github.com/enzo-adami/hermes-agent/tree/agent/pr84066-main-refresh

Base is exact main 165c889; commit 83f281c preserves joaomarcos as primary author and includes the narrow max_call_seconds=0 correction from JoaoMarcos44#15. Net scope remains the six watchdog/config/docs files.

Fresh validation: 32/32 dedicated first-byte/idle/max-call/keepalive tests, Ruff, Windows-footgun checker, and diff-check all pass. The branch is a force-update/cherry-pick handoff for #84066, not a replacement PR.

imma fix this now, tysm