fix(zai): bound in-flight model calls to Z.AI under subagent swarms

[Da7-Tech]

Summary

Adds a strict, process-local concurrency bound for Z.AI / Zhipu model calls made by direct agents, delegated subagents, and Mixture-of-Agents (MoA). The goal is to stop a fan-out from sustaining HTTP 429 / code 1305 overload retries against the Coding Plan endpoint.

The important correction in this revision is that saturation now queues instead of silently proceeding uncapped. The previous best-effort timeout path could exceed the configured ceiling exactly when the provider was already overloaded.

Why this matters

A 1305 retry storm is more than a latency problem. In the field report on this PR, overloaded Coding Plan calls carried roughly 130k tokens and could be attempted up to six times, repeatedly re-sending the same large context and consuming the rolling quota. A five-profile deployment sharing one key reported zero 1305 responses in its active observation window after setting the cap to 2, while MoA continued to run normally.

MoA makes the bound especially important: reference models fan out in parallel and the acting aggregator uses its resolved provider route on every relevant iteration. If that aggregator is GLM on Z.AI, the outer virtual moa provider is not enough to identify or limit the real requests.

What changed

• Added one lazy, process-wide BoundedSemaphore for resolved Z.AI destinations.
• Changed the default cap from 4 to 2; HERMES_ZAI_MAX_CONCURRENT=0 disables it.
• Moved acquisition into context entry so an unused handle cannot leak a slot.
• Made the cap strict: a positive HERMES_ZAI_ACQUIRE_TIMEOUT_S raises locally without contacting the provider; the default 0 waits for a slot.
• Made waiting interruptible and closed the race where an interrupt arrives at the same instant a slot is granted.
• Preserved the current codex_responses preflight while rebasing the main call path.
• Applied the same shared gate to MoA reference calls and the resolved acting aggregator.
• Kept a Z.AI streaming slot for the stream's full iterator lifetime, not only until the SDK returns the stream object.
• Added operator documentation, including the multi-process fleet caveat and the separate five-hour usage quota.

Non-Z.AI providers still take the direct no-op path. A bare glm-* model name is intentionally insufficient for detection because local and third-party endpoints can serve GLM models without being Z.AI.

Scope and limitations

The semaphore is process-local. If several CLI or gateway processes share one key, the effective ceiling is:

process count × HERMES_ZAI_MAX_CONCURRENT

Fleet operators should normally keep the per-process value at 1 or 2. Cross-process coordination remains separate work.

This patch does not reinterpret the hard Usage limit reached for 5 hour response; that is a rolling spend quota, not concurrency overload. The adjacent client-identity concern raised in the field report is also separate from concurrency and is intentionally not mixed into this focused patch.

Verification

Three independent verification layers passed on the rebased final commit:

1. Focused gate and MoA tests: 25 passed, including acquire-on-enter, strict timeout, foreign-slot safety, interrupt races, six parallel MoA references, and full streaming-lifetime ownership.
2. Behavioral integration: 152 related MoA/retry/interrupt/system-prompt tests passed; the complete streaming suite passed with 44 tests.
3. Broad regression pass: 729 agent, auxiliary-client, compression-timeout, and run-agent tests passed.

Additional checks:

• 100-thread stress simulation held the peak at exactly 2 and restored full capacity after injected exceptions.
• ruff, py_compile, and git diff --check passed.
• Manual diff/security review found no secret handling, persistence, command execution, or network destination changes outside the intended provider calls.
• Added lines and commit metadata contain no personal name, email, or local filesystem path; the commit uses the pseudonymous GitHub noreply identity.

[ahmadalzaro1]

Ran this on a 5-profile Hermes fleet — one z.ai Coding Plan key, glm-minimax MoA presets. Applied the gate at HERMES_ZAI_MAX_CONCURRENT=2 and restarted: in the active window since, zero 1305s (baseline was 30–40/day/profile, 172 on the busiest), with MoA still firing normally (MiniMax references + GLM aggregator). Short window so far — I'll follow up with a full-day figure — but the drop under active load is unambiguous. A few observations that might strengthen the PR:

The retry storm is the real damage, and this gate breaks it. Before the gate, each 1305'd call retried up to maxAttempts (6×), and because coding-plan calls carry the full ~130k-token context, every failed attempt re-ships ~130k input. So the concurrency overload wasn't just latency — it was multiplying token spend ~6× on every throttled call and burning through the 5-hour usage quota. Bounding in-flight calls stops the overload → stops the retries → the token bleed goes away too, not only the 429s. Might be worth calling out in the PR description, since the win is bigger than "fewer 429s."

MoA 2.0 is what makes this urgent. Since #54016 (references fire on every user/tool response, in parallel, through the provider's real route), a single task's tool loop becomes a burst of concurrent z.ai aggregator calls. On glm-* aggregator presets that's exactly what pegs the coding plan at its ~1-in-flight ceiling — this gate is what makes MoA-on-coding-plan viable.

Caveat for single-key fleets: the gate is process-local, so N gateway processes sharing one key get N independent semaphores → effective ceiling N × HERMES_ZAI_MAX_CONCURRENT. I run =2; the default 4 may still saturate a multi-process fleet. A doc note for fleet operators to scale it down per-process would help (cross-process being #7479's domain, as you note).

One adjacent gap, separate from this PR: the opencode-identity work (#51792) is only wired into the main OpenAI client, not auxiliary_client.py — so MoA reference/aggregator zai calls (which route through the auxiliary path) don't get the opencode UA / X-Stainless-* strip. Relevant because MoA is the heaviest zai caller and it's the one going out un-disguised.

And to avoid conflation in testing: distinct from 1305, the coding plan also returns a hard HTTP 429 "Usage limit reached for 5 hour" (rolling quota) — a spend cap, not concurrency, which this gate correctly leaves alone.

[Da7-Tech]

Final audit revision pushed as 1e1bd8ee9 and rebased onto current main.

The field report in #issuecomment-4881661503 exposed two material gaps in the earlier patch, both now fixed:

• The gate is now a strict bound. A saturated timeout raises locally instead of proceeding uncapped, so overload cannot defeat the configured ceiling.
• MoA is covered at the real call sites: Z.AI reference calls and the resolved acting aggregator share the same process-wide semaphore, and streaming holds its slot until iterator completion.

The default is now 2, waits are interruptible, the acquire/interrupt race returns the granted slot before raising, and the docs explain multi-process fleet scaling plus the unrelated five-hour quota. The current-main codex_responses preflight was preserved during conflict resolution.

Final verification: 25 focused tests, 152 related integration tests, 44 streaming tests, and a 729-test broad regression pass all green; ruff, py_compile, git diff --check, a 100-thread stress run, manual security review, and privacy/metadata checks also passed.

The adjacent client-identity item from the field report remains intentionally separate from this concurrency-focused PR.

[teknium1]

Thanks for the focused concurrency investigation and for incorporating the MoA, streaming-lifetime, interrupt-race, fleet-scaling, and operator-documentation feedback.

This automated hermes-sweeper review is closing this under the standing configuration policy:

• PR commit 1e1bd8ee97852506216f4c13fa256bd2f80fa080 makes HERMES_ZAI_MAX_CONCURRENT and HERMES_ZAI_ACQUIRE_TIMEOUT_S the user-facing controls (agent/zai_concurrency.py:33-38, documented in website/docs/reference/environment-variables.md:762-763 in the PR diff).
• The maintainer rubric explicitly reserves .env / HERMES_* for secrets and requires behavioral thresholds and timeouts to live in config.yaml (AGENTS.md:102-107).
• Current main does have the underlying overload path and its retry handling (agent/conversation_loop.py:3177-3188, 4139-4159), so this is not a judgment that the operational report is invalid.

A viable follow-up would re-scope the controls to a profile-safe config.yaml setting, with the normal config/setup integration, rather than introduce new user-facing environment variables.

---

Closed as not-planned per standing maintainer policy (env-var-for-config). This is a design-direction decision, not a code-quality judgment — see the Contribution Rubric in AGENTS.md for what the project is looking for. If you believe this policy was misapplied to your change, comment here and a maintainer will take a look.

[Da7-Tech]

Implemented every requested configuration-policy change in #64911.\n\nThe user-facing controls now live only in the active profile's config.yaml under providers.zai, with normal hermes config set integration. The new PR adds no behavioral HERMES_* variables, keeps the strict direct-agent and MoA concurrency bound, holds streaming slots for the full iterator lifetime, preserves interruptible waits, and adds profile-scoped real-path tests plus process stress and live Z.AI validation.\n\nClosing the loop here for traceability; review can continue on #64911.