feat(resilience): stop retrying sibling connections after an IP-bucketed 429

[maxmad64bis]

⚠️ base-red inherited: #9985

Summary

This removes upstream calls, it doesn't add capacity. Nothing here creates an identity, rotates an IP, or synthesizes a header — it just stops hammering a bucket that already said no.

You established in #9611 that the opencode free tier is IP-based, not account-based, and that real rotation needs a distinct proxy IP per account. Taking you at your word: when connections do share an IP, the current rotation still tries each of them after the first 429, so one exhausted bucket costs one guaranteed-failed call per remaining connection. Those calls hit an upstream that has already refused.

So on a 429 from an allowlisted provider, cool every connection sharing the failing one's last known egress IP. Rotation skips them instead of trying them.

Unlike #10357, nothing here depends on how the upstream responds — the branch runs after a 429 is already in hand and only writes local cooldown state, so there's no upstream round-trip to validate.

One behavior change to weigh: opencode is passthroughModels, so a 429 used to lock a single model and now cools the connection, sibling or not. That's what the opencode rule table already declares (scope: "connection") and never got to apply. Removing the provider from the allowlist reverts it. Limits are in docs/architecture/RESILIENCE_GUIDE.md §7.

Related Issues

• Closes feat(resilience): lock all connections of a shared egress IP together on IP-bucketed 429 #10880
• Related to question: Cannot use Claude Code or OpenCode — all providers return 429 rate limit #9611 (your IP-based diagnosis, which this builds on), fix(backend): redact client IPs, account prefixes, and internal URLs in default logs #10348, feat(resilience): Surface anonymous egress-IP sharing in the proxy health sweep and the egress API #10677

Validation

• Change type: routing / resilience
• Focused tests and category gates from the golden path
• npm run lint
• Reconciled with the current active release base; focused checks rerun afterward
• Production-code changes include a new or updated automated test in this PR

node --import tsx/esm --test tests/unit/egress-lock-allowlist-10880.test.ts \
  tests/unit/proxy-logs-egress-lookup-10880.test.ts tests/unit/egress-ip-lock-10880.test.ts
# tests 23  # pass 23  # fail 0

# suites touching this path, unchanged:
agentrouter-lock-scope-10334 · agentrouter-error-rules · proxy-logs-egress-ip ·
proxy-10348-log-redaction · account-fallback-service · combo-lockout-quota-reset-6863 ·
noauth-autocombo-lockout-7623 · auth-disable-cooling-2997 · auth-terminal-status
# tests 151  # pass 151  # fail 0

sse-auth · chat-cooldown-aware-retry · model-lockout-max-cooldown ·
10347-embed-402-cooldown · account-fallback-lockout-eviction ·
auth-opencode-zen-noauth-fallback
# tests 86  # pass 86  # fail 0

npm run typecheck:core  # exit 0
npm run check:cycles    # exit 0, 424 files
npm run lint            # exit 0

The "one call instead of N" claim is easy to fake, so the test walks the whole pool with no early exit and skips only what credential selection skips. The same rotation on a provider that didn't opt in gets 3 — that gap is the measurement, not the 1. A third case runs it with isCombo: true and also gets 1.

Cost, on a real-traffic DB copy: 4708 rows in the 24h window out of 104624 in proxy_logs, sibling query ~14 ms. Two window-bounded scans at 429 frequency, in a try/catch — a DB failure never reaches the caller.

Inherited from the base: check:mutation-test-coverage (entries owned by the base and other branches; the 3 files here are registered, delta 0) and check:docs-all, which stops at check:env-doc-sync over BOT_TOKEN/BOT_URL from scripts/ad-hoc/mesh-*.mjs. The two docs gates behind it were run alone and pass.

Tests Added Or Updated

• tests/unit/egress-lock-allowlist-10880.test.ts — the allowlist predicate.
• tests/unit/proxy-logs-egress-lookup-10880.test.ts — the egress-IP lookup and its window.
• tests/unit/egress-ip-lock-10880.test.ts — the branch: both accepted reasons, never terminal, longer sibling cooldowns never shortened, terminal siblings untouched, unresolvable IP, disableCooling opt-out, the 1/3/combo rotation trio, cross-provider isolation, 401/403 still landing on per-model lockout.

Coverage Notes

auth.ts, providerErrorRules.ts and proxyLogs.ts are each covered by the matching file above. No touched file loses coverage; mutation-gate delta is 0.

Reviewer Notes

• Siblings stay inside the opencode family, and the query binds that allowlist instead of repeating it as SQL text, so widening it stays one line.
• Never terminal, terminal siblings left alone, disableCooling opts out.
• The lookup reads 24h of proxy_logs while the egress-IP cache lives 5 minutes, so it's history: a connection that rotated IPs inside the window can be missed, or cooled after moving off the exhausted IP. §7 says so.
• rate_limit_exceeded is accepted next to quota_exhausted because the opencode rules can't match on this path: markAccountUnavailable passes checkFallbackError neither headers nor structuredError, and opencode isn't in FULL_TEXT_RULE_PROVIDERS. Threading the headers through is the real fix — say the word and I'll do that one first instead.