fix(agent): prevent auto retry loop from slow LLM backend prefilling large contexts (#69424)

[webtecnica]

Problem

When a slow local LLM backend (e.g., 122B Qwen model on AMD Ryzen AI Max+ 395) processes a large context (~140K tokens), prompt prefill can take well over 180 seconds. The stale-stream detector kills the connection at the timeout threshold, and the conversation loop's retry mechanism immediately restarts the same large-context request from scratch — creating an infinite retry loop where prompt processing never completes.

Reported in #69424.

Root Cause

Three interacting issues in the stale-stream timeout computation:

1. Local endpoints skipped context-size scaling. The local-endpoint branch (is_local_endpoint() → default 900s) jumped straight to the local stale timeout without applying the context-token-based scaling that the cloud path used. A 900s ceiling would still fire before a 122B model finishes prefilling 140K+ tokens on modest hardware.

2. No backoff between retries. After a stale-stream kill, the conversation loop retries immediately with the identical timeout. If the prefill genuinely takes longer than the timeout, every retry hits the same wall → infinite loop. The existing circuit breaker (_stale_streak ≥ 5) eventually gives up, but that's 5 full timeout waits rather than progressive scaling.

3. Cloud path tiers were too conservative. The >100K tier capped at 300s and the >50K tier at 240s — too short for very large models processing dense contexts.

Fix (3 changes)

1. Context-size scaling for ALL paths (local + cloud)

Move the scaling out of the else branch so both local and cloud endpoints get proportional timeouts:

Estimated tokens	Minimum stale timeout
>200K	1800s (30 min)
>100K	1200s (20 min)
>50K	600s (10 min)
>10K	base (180s cloud / 900s local)
≤10K	base

Applied to _stream_with_stale_detection (main streaming), _derive_stream_stale_timeout (Bedrock streaming), and _compute_non_stream_stale_timeout (non-streaming).

2. Stale-streak backoff

After 2+ consecutive stale-stream kills on the same agent (tracked by _stale_streak), multiply the stale timeout by a progressive factor:

• streak 2: 2.5x
• streak 3: 4.0x
• streak 4: 5.5x
• streak 7+: 10x (cap)

This breaks the infinite retry loop: each retry waits longer, eventually outlasting the prefill. The multiplier resets on the next successful response (handled by the existing _reset_stale_streak on any successful API call).

3. Updated test assertions

test_non_stream_stale_timeout.py assertions updated to match the new conservative floor values (600s and 1200s instead of 150s and 240s).

Testing

• All 153 stale/timeout-related tests pass
• All 90 chat completions transport tests pass
• All 12 test_timeouts.py tests pass

[teknium1]

Thanks for the detailed investigation and the user confirmation on #69424. This is an automated hermes-sweeper review: current main already provides the PR's central protection against repeated stale-stream retries.

• agent/chat_completion_helpers.py:334 implements the cross-turn stale-stream circuit breaker; after the default five consecutive stale kills, it raises an actionable error rather than allowing another full timeout/retry cycle.
• agent/chat_completion_helpers.py:2748 invokes that guard before opening the next streaming request, and agent/chat_completion_helpers.py:4137 increments the same streak on every stale-stream cancellation.
• tests/run_agent/test_stream_stale_circuit_breaker.py:65 verifies that the threshold short-circuits without opening a new stream.
• The behavior shipped in 985e19c110f19e1131b52f78c6505e2a06837127 (v2026.7.7). website/docs/reference/environment-variables.md:802 also documents the configurable local stale ceiling.