feat(sse): per-egress pacing + fleet-wide backoff for opencode rotation (opt-in) - #14290
Merged
diegosouzapw merged 3 commits intoSep 21, 2026
Conversation
added 2 commits
September 20, 2026 11:09
…replay one capped leg
maxmad64bis
force-pushed
the
feat/egress-throttle-fleet-backoff
branch
from
September 20, 2026 22:35
7bbf228 to
e2bda0a
Compare
maxmad64bis
marked this pull request as ready for review
September 20, 2026 22:36
5 tasks
…w#13924 rewrite Merges origin/release/v3.8.51 (which already carries diegosouzapw#13924's rewrite, squash-merged as 893fef9) into this PR branch. The three conflicting paths (open-sse/executors/opencode.ts, tests/unit/opencode-429-park-resume.test.ts, config/quality/file-size-baseline.json) are resolved to base's diegosouzapw#13924 semantics plus only diegosouzapw#14290's own egress-throttle additions (opencodeEgressThrottle.ts + its dedicated test file, untouched). Default-off (OPENCODE_EGRESS_THROTTLE_ENABLED unset): behavior is byte-identical to the tip, proven by the existing "env-module off by default: everything is a no-op" test plus the full opencode-* suite (101/101 passing). With the flag on, the new fleet-suspect slot-budget hands over to diegosouzapw#13924's park-and-replay path instead of returning the last 429, proven by the added handover test. Also fixes a rebaseline-note label carried over from the PR's own commit (wrong PR number, stray non-English word) in config/quality/file-size-baseline.json. Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
diegosouzapw
merged commit Sep 21, 2026
3411317
into
diegosouzapw:release/v3.8.51
7 of 16 checks passed
This was referenced Sep 21, 2026
diegosouzapw
added a commit
that referenced
this pull request
Sep 21, 2026
Keep the drain's t06 Zod validation on /v1/responses/input_tokens (ours) over the tip's empty passthrough schema. Bump the server-owned-tool-loop flag-count snapshot 74 → 75 to match the #14290 flag that landed on the tip. Env-doc-sync is clean after the merge (0 code vars missing from .env.example).
diegosouzapw
added a commit
to aldoeliacim/OmniRoute
that referenced
this pull request
Sep 24, 2026
…alog fix Neither diegosouzapw#14290 (an unrelated merged PR) nor diegosouzapw#5460/diegosouzapw#5465 (unrelated closed issues about Reka and t3.chat) documents this bug — no filed issue exists for it. Drop the bogus links: rename the changelog fragment to the PR's own number (14448, matching the repo's fragment-naming convention) and drop the diegosouzapw#14290 link from its body, and remove the diegosouzapw#5460/diegosouzapw#5465 reference from the code comment in modelRouteProjection.ts. Also ran the two regression suites named in review (tests/unit/noauth-catalog-only-providers.test.ts and tests/unit/noauth-local-catalog-intentional.test.ts) in isolation to confirm both pass; the earlier failure seen when run together was SQLITE_BUSY DB contention from the shared devbox, not a real defect. Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
diegosouzapw
added a commit
to aldoeliacim/OmniRoute
that referenced
this pull request
Sep 24, 2026
The first commit only carried the file rename; this carries the actual content change it described — drop the diegosouzapw#14290 link from the changelog fragment body and the diegosouzapw#5460/diegosouzapw#5465 reference from the code comment. Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
When several concurrent requests share the same network egress, multi-account rotation fires them all at once, so a burst of 429s hits the provider simultaneously. After ten burst 429s in a minute the fleet is treated as suspect for 60-120 s, during which each request tries at most two accounts before handing over to the wait-and-replay path of #13924 (heartbeat plus one capped leg; last upstream answer only when that path is off), and the first success clears the state. Real rate limits (parseable Retry-After or rate-limit wording) never feed the suspect counter and keep the existing early-stop path; everything fails open and the behavior is opt-in via
OPENCODE_EGRESS_THROTTLE_ENABLED=1(default off: rotation unchanged), pacing dispatches per egress for #10880 (IP-bucketed 429s waste one upstream call per sibling account) and backing the fleet off for #10920 (stop retrying sibling connections after an IP-scoped 429) instead of locking accounts after the fact.Related Issues
Validation
npm run lintTests Added Or Updated
tests/unit/opencode-egress-throttle.test.ts(new, 22 tests: config defaults and bounds, cap and FIFO, jitter ranges, fleet window and expiry, fail-open paths, first-success reset, simulated burst wave, per-arm leak guards, caps eviction)tests/unit/opencode-429-park-resume.test.ts(handover case: a fleet-suspect slot budget runs the park-and-replay instead of returning the last 429)Coverage Notes
open-sse/executors/opencodeEgressThrottle.ts(new, ~545 lines) is covered by the 22 tests above;open-sse/executors/opencode.tsseam (slot-budget arm plus handover) is covered by the handover test. No existing test needed changes.Reviewer Notes
OPENCODE_EGRESS_THROTTLE_ENABLEDunset: rotation byte-identical); tuning viaOPENCODE_EGRESS_THROTTLE_{CAP,WAIT_MIN_MS,WAIT_MAX_MS,WAIT_BUDGET_MS,FLEET_WINDOW_MS,FLEET_THRESHOLD,SUSPECT_MIN_MS,SUSPECT_MAX_MS,SUSPECT_SLOTS}with bounds, never throws. Alongside fix(sse): park request with heartbeat after repeated transient 429s, replay one capped leg #13924, the slot-budget arm hands over to its wait-and-replay instead of returning the last 429.open-sse/executors/opencode.ts(1233 to 1247: irreducible seam, handover arm; logic lives in the leaf modules under the cap).api-typecheckshows 2 pre-existing errors (rerankProviderNodes.ts,antigravity.ts, both TS2677, files outside this PR, unchanged since merge-base) — inherited from the red base (🔴 Release branch not green: release/v3.8.51 #13866), not this PR.