fix(sse): re-select pool member per attempt after per-address 429

.env.example

@@ -1840,6 +1840,7 @@ CURSOR_USER_AGENT="Cursor/3.4"
 # OPENCODE_PARK_AND_RESUME=false  # #13924 feature flag (Settings → Feature Flags wins): park the request with a heartbeat after repeated transient 429s, then replay one capped leg of up to 3 accounts
 # STREAM_READINESS_STALL_RETRY=false  # feature flag (Settings → Feature Flags wins): when a streaming first body stalls before usable output, issue one bounded second attempt through the same routing path
 #OPENCODE_POOL_STRAIN_MARKER_PATH=/tmp/opencode-pool-strain.json  # #13924: pool-strain marker path (JSON {since, reason, ttl_s}); fresh marker parks without recounting
+# OPENCODE_POOL_RESELECT=false  # feature flag (Settings → Feature Flags wins): after a 429 from an egress-bucketed provider on a proxy-less account under an ambient pool context, ask the pool for another member for the next attempt instead of retrying the same address
 # RESPONSES_FIRST_BYTE_TIMEOUT_MS=15000  # #13484: OpenCode Responses first-byte window, only used when the OPENCODE_RESPONSES_STALL_ROTATION flag is on (0 disables)
 # OPENCODE_RESPONSES_HEADERS_WAIT_MS=30000  # opt-in bound on waiting for upstream response headers on streamed Responses calls before moving to the next account (0 = off, suggested 30000)
 # OPENCODE_RESPONSES_HEADERS_WAIT_MAX_ROTATIONS=2  # rotation budget for the headers-wait bound above

changelog.d/fixes/14659-opencode-pool-reselect-per-attempt.md

@@ -0,0 +1 @@
+- **fix(sse):** the opencode rotation may serve the next attempt from another pool member after a 429 from an egress-bucketed provider, instead of retrying the refused egress address ([#14659](https://github.com/diegosouzapw/OmniRoute/pull/14659)) — thanks @maxmad64bis

config/quality/eslint-suppressions.json

@@ -1962,7 +1962,7 @@
   },
   "src/sse/handlers/chatHelpers.ts": {
     "@typescript-eslint/no-unused-vars": {
-      "count": 3
+      "count": 2
     }
   },
   "tests/benchmarks/pipeline-accuracy.test.ts": {
@@ -3795,11 +3795,6 @@
       "count": 3
     }
   },
-  "tests/unit/gemini-business-provider.test.ts": {
-    "@typescript-eslint/no-unused-vars": {
-      "count": 1
-    }
-  },
   "tests/unit/gemini-finish-reason-normalization.test.ts": {
     "@typescript-eslint/no-explicit-any": {
       "count": 3

docs/reference/ENVIRONMENT.md

@@ -809,6 +809,7 @@ REQUEST_TIMEOUT_MS (global override)
 | `OPENCODE_RESPONSES_HEADERS_WAIT_MAX_ROTATIONS` | `2`            | OpenCode executor only: how many times per request the headers-wait bound above may move to the next account. The last remaining account always keeps the full headers window. |
 | `OPENCODE_PARK_AND_RESUME`                    | `false`              | OpenCode executor only: park the request with a heartbeat after repeated transient 429s (or a fresh pool-strain marker), then replay one capped leg of up to 3 sequential accounts instead of fanning out the whole fleet (#13924). Off by default: every 429 rotates to the next account exactly as before. |
 | `STREAM_READINESS_STALL_RETRY` | `false` | Streaming chat only: when the first upstream body stalls before producing a usable event, issue one bounded second attempt through the same routing path with the same readiness budget and no account penalty. Off by default: a stalled first body fails the request without a retry. |
+| `OPENCODE_POOL_RESELECT`                    | `false`              | OpenCode executor only: after a 429 from an egress-bucketed provider on a proxy-less account under an ambient pool context, ask the connection pool for another member for the next attempt instead of retrying the same egress address. Orders, never excludes: an exhausted pool keeps the current behavior. Off by default: every 429 rotates to the next account exactly as before. |
 | `OPENCODE_POOL_STRAIN_MARKER_PATH`              | _(unset)_            | OpenCode executor only: override path of the pool-strain marker read before parking (`{since, reason, ttl_s}`, default `/tmp/opencode-pool-strain.json`, #13924). A fresh marker parks without recounting; absent or stale falls back to the burst counter. |
 | `API_BRIDGE_PROXY_TIMEOUT_MS`             | `30000`              | Proxy hop timeout for `/v1` bridge requests.                                                                                                                    |
 | `FIRECRAWL_BASE_URL`                      | `https://api.firecrawl.dev` | Point the Firecrawl web-fetch executor at a self-hosted instance (API key optional off-cloud).                                                          |

docs/reference/FEATURE_FLAGS.md

@@ -46,7 +46,7 @@ A boolean flag is considered **enabled** when its effective value is `"true"`,
 
 ## Flag Catalog
 
-78 flags across 6 categories. **Default** is the definition default — the value
+79 flags across 6 categories. **Default** is the definition default — the value
 used when neither a DB override nor an environment variable is present.
 
 ### Security (10)
@@ -64,7 +64,7 @@ used when neither a DB override nor an environment variable is present.
 | `AUTH_LOG_INCLUDE_ACCOUNT_ID`           | boolean | `false`  | Include account prefix in AUTH log lines (e.g. "Using <provider> account: abc12345..."). Disabled by default so account identifiers are redacted from shared/multi-tenant process logs. Independent from Debug Mode; flipping Debug Mode does not reveal this. |
 | `OMNIROUTE_OIDC_DISABLE_PASSWORD_LOGIN` | boolean | `false`  | When OIDC is enabled, disable password login so users can only authenticate via OIDC Single Sign-On. When disabled (default), both password login and OIDC are available.                                                                                      |
 
-### Network (20)
+### Network (21)
 
 | Key                                             | Type    | Default | Restart | Description                                                                                                                                                                                                                                                                                                                                                                                                                                             |
 | ----------------------------------------------- | ------- | ------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
@@ -75,14 +75,15 @@ used when neither a DB override nor an environment variable is present.
 | `OMNIROUTE_CONTROL_PLANE_PROXY_DIRECT_FALLBACK` | boolean | `false` |         | Allow OAuth and provider validation flows to bypass a pinned proxy and connect directly when proxy reachability pre-checks fail. Off by default because this can change egress IP.                                                                                                                                                                                                                                                                      |
 | `NETWORK_ROTATION_SHARED_EGRESS_GUARD`          | boolean | `true`  |         | On a network exception (timeout, connection refused/reset) for a multi-account rotation executor, when the failing account has no dedicated proxy, apply a short cooldown and skip other proxy-less accounts for the rest of the request instead of retrying each one. On by default (safe: no egress IP change, only reduces latency/cooldown risk on shared-egress accounts). Disable to restore immediate propagation on the first proxy-less throw. |
 | `ROTATION_ATTRIBUTION`                          | boolean | `false` |         | Opencode rotation records which account served or was skipped (masked ids only, never full account ids) and links proxy log entries to their request, so the operator can tell skipped accounts apart from unused ones. Off by default.                                                                                                                                                                                                                 |
-| `PROXY_SKIP_RECENTLY_FAILED`                    | boolean | `true`  |         | Proxy pools and the per-account rotation of opencode stop re-serving a proxy that just failed (refused TCP probe, or a 429 received through it) for a per-process period that doubles on each repeat, up to a cap. No proxy status is written; with every candidate set aside the choice is unchanged. On by default; `false` restores plain selection.                                                                                                                       |
+| `PROXY_SKIP_RECENTLY_FAILED`                    | boolean | `true`  |         | Proxy pools and the per-account rotation of opencode stop re-serving a proxy that just failed (refused TCP probe, or a 429 received through it) for a per-process period that doubles on each repeat, up to a cap. No proxy status is written; with every candidate set aside the choice is unchanged. On by default; `false` restores plain selection.                                                                                                 |
 | `PROXY_POOL_EGRESS_OBSERVATION`                 | boolean | `false` |         | Show, under a proxy pool in the dashboard, how many observed egress IPs served its members over the last 24 h and how many connections used them. Read-only, computed from the proxy log, never used for routing. Off by default.                                                                                                                                                                                                                       |
 | `OPENCODE_RESPONSES_STALL_ROTATION`             | boolean | `false` |         | For the OpenCode executor, watch the first body byte of a streamed Responses reply (window: `RESPONSES_FIRST_BYTE_TIMEOUT_MS`, default `15000`). A 2xx Responses stream that stays silent past the window is treated as stalled: the account is cooled down and the request rotates to the next account once; a second stall fails fast. Off by default: stalled streams keep today's wait until the stream readiness timeout.                          |
 | `OPENCODE_USER_BLOCKED_ROTATION`                | boolean | `false` |         | OpenCode executor: on a 403/451 carrying a `user_blocked` refusal (not geo, not a Cloudflare fingerprint rejection), cool the refused account down and rotate to the next account at most once per request; a second refusal is returned as-is, without a success mark. Off by default: routing around an upstream user block can look like evasion and spread the flag across the fleet.                                                               |
 | `OPENCODE_TRANSIENT_FAILOVER_BACKOFF`           | boolean | `false` |         | OpenCode rotation: after two consecutive transient upstream failures (5xx or an empty 400), pause before the next account — 1.5s doubling per further failure, capped at 6s per pause and 10s per request, skipped on client disconnect; the failed body is released before waiting. Off by default: failover stays immediate.                                                                                                                          |
 | `OPENCODE_PARK_AND_RESUME`                      | boolean | `false` |         | OpenCode rotation: park the request after repeated transient 429s (or a fresh pool-strain marker) with a heartbeat, then replay one capped leg of up to 3 sequential accounts instead of fanning out the whole fleet. Off by default: every 429 rotates to the next account exactly as before.                                                                                                                                                          |
 | `STREAM_READINESS_STALL_RETRY`                  | boolean | `false` |         | Streaming chat: when the first upstream body stalls before producing a usable event, issue one bounded second attempt through the same routing path with the same readiness budget and no account penalty. Off by default: a stalled first body fails the request without a retry.                                                                                                                                                                      |
 | `FLUSH_EMPTY_RETRY_ENABLED`                     | boolean | `false` |         | On translated streaming turns, when the upstream turn carries no usable content (reasoning-only completion or zero valuable chunks), issue bounded retries through the normal credential path (up to `STREAM_RECOVERY.EMPTY_TURN_RETRY_MAX`) before anything is exposed to the client. Off by default: empty turns keep the current behavior (empty 200 or empty-content 502).                                                                          |
+| `OPENCODE_POOL_RESELECT`                        | boolean | `false` |         | OpenCode rotation: after a 429 from an egress-bucketed provider on a proxy-less account under an ambient pool context, ask the connection pool for another member for the next attempt instead of retrying the same egress address. Orders, never excludes: an exhausted pool keeps the current behavior. Off by default: every 429 rotates to the next account exactly as before.                                                                      |
 | `OPENCODE_RATE_LIMITED_429_EARLY_STOP`          | boolean | `false` |         | OpenCode rotation: stop the account wave at the first 429 classified as a real rate limit (parseable `Retry-After`, or a body naming a rate/usage limit) and return that upstream 429 unchanged. Unclassified 429s keep rotating. Off by default: the free tier is limited per egress IP (#9611), so every 429 rotates and an exhausted wave returns the last upstream 429.                                                                             |
 | `MITM_DISABLE_TLS_VERIFY`                       | boolean | `false` | ✓       | Disable TLS certificate verification for the MITM proxy. **Danger.**                                                                                                                                                                                                                                                                                                                                                                                    |
 | `OMNIROUTE_ALLOW_PRIVATE_PROVIDER_URLS`         | boolean | `false` |         | Allow provider URLs pointing to private/internal networks.                                                                                                                                                                                                                                                                                                                                                                                              |
@@ -133,7 +134,7 @@ used when neither a DB override nor an environment variable is present.
 | `RETRY_AFTER_PROVENANCE_ENABLED`            | boolean | `false` |         | On aggregated 429/503 unavailable responses, omit `Retry-After` when no concrete future retry time is known (instead of a synthetic 1s), add `error.retry_after_provenance` (`signal` \| `none`), and let combo drain paths read prose retry hints from JSON and plain-text upstream bodies. The field only appears on responses built by `unavailableResponse()`; other 429/503 bodies are unchanged.                                                                                                                                                                                                                                          |
 | `PROTECTED_PRIORITY_INFRA_502_ENABLED`      | boolean | `false` |         | When a `priority` combo target marked fallback-only-on-quota-exhaustion stops the combo for a cause that is provably not quota (provider circuit breaker open, predictive latency skip), answer 502 instead of the quota-looking 503. Lockout, cooldown, unavailable, exhaustion and concurrency-cap stops keep 503.                                                                                                                                                                                                                                                                                                                            |
 | `MISTRAL_AMBIGUOUS_401_SOFT_LOCKOUT`        | boolean | `false` |         | A bare Mistral 401 (`{"detail":"Unauthorized"}`, no explicit auth signal) is identical for a revoked key and for exhausted quota. When on, it cools the connection down instead of parking it as `expired`, at most 3 times per hour per connection; the next one parks it, so a revoked key still converges. Off by default: every bare Mistral 401 parks the connection as before.                                                                                                                                                                                                                                                            |
-| `XAI_OAUTH_LIVE_MODEL_DISCOVERY`            | boolean | `true` |         | Fetch the live xAI model catalog for xai-oauth connections from https://api.x.ai/v1/models using the OAuth bearer token, instead of the frozen static seed. On by default. Set the flag to false to keep serving the static seed. HTTP failures fall back to the seed in the discovery route; the flag getter itself does not issue HTTP.                                                                                                                                                                                                                                                                              |
+| `XAI_OAUTH_LIVE_MODEL_DISCOVERY`            | boolean | `true`  |         | Fetch the live xAI model catalog for xai-oauth connections from https://api.x.ai/v1/models using the OAuth bearer token, instead of the frozen static seed. On by default. Set the flag to false to keep serving the static seed. HTTP failures fall back to the seed in the discovery route; the flag getter itself does not issue HTTP.                                                                                                                                                                                                                                                                                                       |
 | `BATCH_AND_FILE_AUTO_CLEANUP_ENABLED`       | boolean | `false` |         | Let the automatic cleanup sweep delete terminal (completed/failed/cancelled/expired) Batch API jobs older than `OMNIROUTE_BATCH_RETENTION_DAYS`, along with their per-line checkpoints, and clear the BLOB content of uploaded files past their own `expires_at`. Off by default: every existing install keeps this data exactly as before until an operator opts in. The operator-triggered `DELETE /api/v1/batches/delete-completed` route is unaffected either way — it is a separate, unconditional public API contract.                                                                                                                    |
 | `ANTIGRAVITY_ACCOUNT_LEASE_ENABLED`         | boolean | `false` |         | Reserve the selected Antigravity account for the streaming lifecycle of the request that picked it, so a concurrent retry or the credential handoff cannot re-pick an account already committed to an in-flight stream. The reservation is scoped to (connection, callable upstream model), so one account can still serve two different models at once. When every eligible account is already leased for that model, the request returns a structured 503 `antigravity_pool_busy` with a bounded `Retry-After` instead of piling onto a busy account. Off by default: account selection stays exactly as before, and no reservation is taken. |