fix(stream): ask the backend before killing a slow local request (progress-aware watchdog for W2)

[OmarB97]

The bug

Hermes killed a healthy 90.7k-token turn on deepseek-v4-flash-w2 at 240s, while vLLM was still prefilling it:

API call failed after 1 retries: Local dflash stream produced no first chunk after 240s (threshold: 240s)
Fallback policy any: deepseek-v4-flash-w2 via ai-router failed (reason: timeout); no usable configured fallback remained.

Two defects, one root cause: the watchdog was a blind stopwatch.

Defect 1 — the budget was an uncalibrated step ladder

est_tokens > 100_000 -> 300.0
est_tokens >  50_000 -> 240.0     # <- 90.7k landed here
else                 -> 180.0

• Discontinuous. 99,999 tokens got 240s; 100,001 got 300s.
• Capped at 300s with no headroom above 100k.
• Calibrated against nothing. Those numbers correspond to no measurement of any backend.
• Its 180s floor did not even clear a cold model load. Measured on taro: a llama-swap W2 load takes 138.5s for a 4-token prompt, before any prefill. llama-swap evicts W2 whenever another model is requested on the same GPU, so the very next W2 turn pays that load and re-prefills the whole context (the eviction wipes the KV cache). The old floor left ~40s for everything else.

Now continuous, and never narrower than the ladder it replaces:

budget = cold_start_allowance + base + per_1k * (tokens / 1000)   [clamped to a ceiling]

tokens	old	new
10,000	180s	400s
50,000	180s	560s
90,700	240s	723s
131,072	300s	884s

Defect 2 — the real one: it never asked

A stopwatch cannot distinguish "the server is wedged" from "the server is healthily prefilling 90k tokens." So it guesses, and at some context size it always guesses wrong.

It no longer has to guess. ai-gate proxies every generation request to llama-swap, which makes it the one component that knows the truth, and it already exposes it:

GET /_gate/status
{ "active_generation_count": 1,
  "active_generations": [{"id","model","path","client_host","started_at","age_s"}] }

When the no-first-chunk budget expires, the watchdog now asks the backend whether it is still working. If it is, the request is extended and the wait is surfaced to the user (⏳ Still prefilling ~90,700 tokens (312s elapsed). The server reports it is working — waiting.) instead of dying silently. Only a genuinely idle backend — or the absolute ceiling — ends the request.

This is what makes it hold up as contexts grow: it is no longer a prediction.

Note a deliberate subtlety: if the gate reports a generation for a different model, we still treat the backend as busy. On a single-GPU llama-swap host that means our model was evicted and is being swapped back in — precisely when patience is required.

Why there is no fitted TTFT curve in this diff

I measured, and the honest answer is that a single constant would be fiction:

• Prefix-cached (the normal agentic case — each turn appends to an already-cached conversation): 18.8s @ 4k, 87.5s @ 32k, then flat ~100s at 64k/96k/128k. TTFT stops growing because vLLM serves the shared prefix from KV cache. This is why W2 feels fast turn-to-turn.
• Uncached (after an eviction wipes the cache): the same 4k prompt with unique content took 62.6s — 3.3× the cached figure.

The cost of a healthy request spans more than an order of magnitude depending on cache state, and nothing client-side can know which regime a given turn is in. That is the argument for liveness over prediction, and the budget is now just a safety net for backends that expose no signal.

Evidence

$ python3 -m pytest tests/agent/test_local_stream_timeout.py -q
95 passed in 3.27s

$ python3 -m pytest tests/agent/ -q -k "stream or timeout or interrupt or chat_completion or watchdog"
509 passed, 5079 deselected

Mutation checks — neither new guard is vacuous:

# restore the old step ladder:
FAILED ... test_budget_clears_a_cold_model_load_at_every_context

# remove the progress-aware extension (kill without asking):
FAILED ... test_busy_backend_extends_instead_of_killing

Backwards compatibility

• Every existing override still wins: HERMES_DFLASH_FIRST_CHUNK_TIMEOUT, HERMES_DFLASH_TTFB_TIMEOUT, config.yaml provider/model timeouts. An operator who pinned a number still gets that number.
• Providers without a gate are untouched. The probe returns None on any error, non-200, missing endpoint, or unexpected body shape — never False — so a remote API or bare llama.cpp server keeps the legacy stopwatch exactly as before. There is a test for this.
• New knobs (all optional): HERMES_DFLASH_PREFILL_SECONDS_PER_1K, HERMES_DFLASH_COLD_START_ALLOWANCE, HERMES_DFLASH_FIRST_CHUNK_CEILING, HERMES_GATE_PROBE_TIMEOUT.
• The watchdog still bounds the worst case: the ceiling (default 1800s) is measured on a non-resettable clock, so a backend that lies about being busy cannot park a request forever.

Risks / gaps

• The probe adds one short HTTP GET per expired budget window (default 4s timeout), not per poll. Negligible, and only on local endpoints.
• This does not make W2 faster. It stops Hermes from killing it. Making large-context turns genuinely quicker is a serving-side question (expert cache sizing, KV dtype), out of scope here.
• The fallback policy is still willing to fall back to another local model, which on a single-GPU llama-swap host would evict W2 and make recovery strictly worse. Not addressed in this PR; worth a follow-up.