Skip to content

fix(zai): set tool_stream=true to avoid api.z.ai 30s idle timeout - #12758

Open
aarongxa wants to merge 1 commit into
NousResearch:mainfrom
aarongxa:aaron/glm-zai-tool-stream
Open

fix(zai): set tool_stream=true to avoid api.z.ai 30s idle timeout#12758
aarongxa wants to merge 1 commit into
NousResearch:mainfrom
aarongxa:aaron/glm-zai-tool-stream

Conversation

@aarongxa

@aarongxa aarongxa commented Apr 20, 2026

Copy link
Copy Markdown

Summary

GLM models (4.6, 4.7, 5, 5.1) batch tool_call argument generation in a single chunk by default. When the tool args are large (long file writes, big shell commands), the model goes silent for 30+ seconds while generating them, and api.z.ai's 30-second server-side idle timeout kills the connection with ECONNRESET — surfacing as a "timeout" inside Hermes mid-tool-call.

Z.AI's documented fix is to send tool_stream=true alongside stream=true. That makes the model emit tool args incrementally and keeps chunks flowing within the 30s window.

What this changes

In _build_api_kwargs (chat-completions branch), inject extra_body["tool_stream"] = True whenever:

  • self.provider == "zai", OR the base URL contains z.ai or bigmodel.cn, AND
  • self.tools is non-empty (no benefit to streaming tool args when no tools are bound)

A user-supplied extra_body.tool_stream wins via setdefault, so it's overrideable.

Why it's safe

  • Gated by provider/base-URL — no behavioral change for OpenRouter, Anthropic, OpenAI, Ollama, etc.
  • Gated by self.tools — never sent on toolless calls.
  • Z.AI ignores unknown extra_body fields gracefully on routes that don't support it (legacy GLM-4.5 and earlier), so even an over-broad match would be a no-op there. Per Z.AI docs, tool_stream is supported on GLM-4.6, 4.7, 5, and 5.1 — i.e. every currently-shipping GLM coding model.
  • setdefault preserves any explicit override.

Test plan

  • pytest tests/run_agent/test_run_agent.py -k "ZaiToolStream or zai or glm" — 8 passed (5 new + 3 pre-existing)
  • pytest tests/run_agent/test_provider_parity.py tests/run_agent/test_strict_api_validation.py — 80 passed
  • New tests cover: zai provider, api.z.ai global URL, bigmodel.cn URL, no-tools omission, non-zai omission

References

GLM models (4.6, 4.7, 5, 5.1) batch tool_call argument generation in a
single chunk by default. When the tool args are large (long file writes,
big shell commands), the model goes silent for 30+ seconds while
generating them, and api.z.ai's 30s server-side idle timeout kills the
connection with ECONNRESET — surfacing as a "timeout" in Hermes.

Z.AI's documented fix is to send tool_stream=true alongside stream=true,
which makes the model emit tool args incrementally and keeps chunks
flowing within the 30s window.

This patch injects extra_body.tool_stream=true in _build_api_kwargs
whenever the provider is "zai" or the base URL points at api.z.ai or
open.bigmodel.cn, and only when tools are present (no benefit otherwise).
A user-supplied extra_body.tool_stream wins via setdefault.

Refs:
- https://docs.z.ai/guides/capabilities/stream-tool
- vercel/ai#12949 (30s idle timeout root cause analysis)
- opencode#15350 (ECONNRESET on zai-coding-plan)

Made-with: Cursor
@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint provider/zai ZAI provider labels Apr 23, 2026
@teknium1

Copy link
Copy Markdown
Contributor

Thanks for identifying the Z.AI idle-timeout failure mode. The requested wire behavior is still absent on current main, but this patch targets a request-builder body that was extracted after the branch was created.

Problems

  • run_agent.py:5328-5331 is now only a forwarder to agent.chat_completion_helpers.build_api_kwargs (extracted in 4b25619bc4770396faf206429ddc180ad02231a9), so the proposed insertion point is no longer live.
  • Direct Z.AI requests resolve through the registered profile path (agent/chat_completion_helpers.py:849-888; agent/transports/chat_completions.py:316-321). ZaiProfile.build_api_kwargs_extras currently emits thinking and reasoning_effort, but not tool_stream (plugins/model-providers/zai/__init__.py:88-108).

Suggested changes

  • Salvage this into the current profile-aware transport path, preserving the tools-only gate and explicit request_overrides.extra_body precedence (agent/transports/chat_completions.py:537-606).
  • Cover both direct provider="zai" and the PR's claimed custom Z.AI endpoint route with current-path tests.

Automated hermes-sweeper review.

@teknium1 teknium1 added sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform labels Jul 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint P2 Medium — degraded but workaround exists provider/zai ZAI provider sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants