Skip to content

fix(agent): name reasoning_effort in custom-provider stale timeouts - #101672

Open
itsflownium wants to merge 1 commit into
NousResearch:mainfrom
itsflownium:fix/100841-reasoning-effort-error
Open

itsflownium wants to merge 1 commit into
NousResearch:mainfrom
itsflownium:fix/100841-reasoning-effort-error

Conversation

@itsflownium

Copy link
Copy Markdown
Contributor

Root cause (the diagnostic gap, not a code-change to the request)

vLLM / llama.cpp / SGLang endpoints can't be capability-probed like Ollama's /api/show (#63315), so Hermes forwards the top-level reasoning_effort field to provider: custom backends unconditionally (#90057). When the backend honours it as extended thinking, a request that took seconds can stall for minutes — with no error, no log line, and the only visible symptom a generic Non-streaming API call timed out after 180s. The reporter lost a fleet of cron jobs to this and had to trace it via manual curl timing experiments.

The issue explicitly does NOT ask to gate the parameter (that would kill the feature for vLLM servers that do serve reasoning well) — it asks for the causal link to be visible when the stall happens (their suggestion 1).

Fix

  • run_agent.py: new _custom_reasoning_hang_hint(api_kwargs) — mirrors the existing _codex_silent_hang_hint shape ([Bug]: All openai-codex / gpt-5.5 primary calls hang silently for full stale timeout #21444). Returns an actionable message when a stale non-streaming call went to a custom/vLLM-family provider carrying a real reasoning_effort (top-level or nested in extra_body; "none"/empty is ignored), naming the effort value, the likely cause (extended-thinking stall vs Ollama's clean 400), the exact workaround (agent.reasoning_effort: ''), and the issue number.
  • agent/chat_completion_helpers.py: new _stale_hang_hint(agent, api_kwargs) resolver (Codex heuristic first — the sharper diagnosis — then the custom-reasoning hint). Wired into all three stale-kill paths: the inline direct_api_call watchdog and both worker-poll branches (TTFB and generic stale detector), replacing the two duplicated codex-only hint blocks. Behaviour for non-matching requests is byte-identical (hint is None).

Why this works

When the 180s/240s timeout now fires on a vLLM-backed custom provider with reasoning_effort: medium set, the RuntimeError, the warning log, and the status-line message all say This request forwarded reasoning_effort='medium' to a custom OpenAI-compatible endpoint … set agent.reasoning_effort: '' instead of the bare timeout — seconds to diagnose instead of days, and the field forwarding itself is untouched (vLLM reasoning servers keep working).

Verification

uv run python -m pytest tests/run_agent/test_custom_reasoning_hang_hint.py tests/run_agent/test_codex_silent_hang_hint.py -o 'addopts=' -q9 passed (7 new: hint fires with value, silent on none/empty/no-effort, non-custom provider excluded, extra_body nesting, codex precedence, fall-through, no-match → None; plus the 2 pre-existing Codex hint tests to prove no regression in the shared slot).

Also run: tests/agent/test_cascading_interrupt_6600.py → 15 passed; tests/plugins -k custom → 33 passed, 3 skipped.

Not claimed: I could not reproduce the actual vLLM hang locally (no GPU/vLLM server here) — the tests pin the decision logic and message content, and the reporter's production evidence already documents the stall side.

Closes #100841

@alt-glitch alt-glitch added type/bug Something isn't working P3 Low — cosmetic, nice to have comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint area/config Config system, migrations, profiles labels Sep 2, 2026
@Enough1122

Copy link
Copy Markdown
Contributor

AI code review — automated review for reference; please use your judgment.

Summary: Adds a _custom_reasoning_hang_hint (custom/vLLM/llama.cpp/SGLang + forwarded reasoning_effort explains generic stale-timeout stalls) and composes it with the existing Codex silent-hang heuristic via _stale_hang_hint, deduplicating three call sites. Well-tested including priority ordering.

Findings (all Non-blocking):

  • agent/chat_completion_helpers.py:17-30 — Resolver iterates attribute names and special-cases calling conventions per name (model= kwarg vs full dict). Works, but the convention is implicit; a wrong-signature helper added later fails silently via the blanket except Exception. Consider a uniform (agent, api_kwargs) signature. Nit.
  • run_agent.py:111 — Provider set {"custom","vllm","llamacpp","llama.cpp","llama-cpp","local"} is a hardcodedUI list; a custom provider configured under another name carrying reasoning_effort gets no hint. Acceptable heuristic — just note the allowlist may need extending (e.g. SGLang named explicitly in the docstring but not in the set — is "sglang" missing intentionally?). Non-blocking question.
  • run_agent.py:119effort_str == "none" treated as no-effort; confirm the backend treats "none" the same as unset (rather than as an unknown level that still triggers thinking). If the transport forwards the string verbatim, "none" could still stall. Non-blocking verification.
  • run_agent.py:121model falls back to getattr(self, "model", ""); hint interpolates it — no secret leakage (model name only). No issue.
  • Refactor at chat_completion_helpers.py:57,72 replaces two duplicated inline blocks with one helper — behavior preserved, Codex hint keeps priority per test. No issue.

LGTM — mainly asking whether "sglang" belongs in the provider set.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/config Config system, migrations, profiles comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint P3 Low — cosmetic, nice to have type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

reasoning_effort silently hangs vLLM (provider: custom) instead of erroring, unlike Ollama's clean 400

3 participants