Skip to content

fix(providers): lock opencode model on upstream 400 model-unavailable - #13146

Merged
diegosouzapw merged 1 commit into
diegosouzapw:release/v3.8.51from
maxmad64bis:fix/opencode-400-model-unavailable-lock
Sep 11, 2026
Merged

diegosouzapw merged 1 commit into
diegosouzapw:release/v3.8.51from
maxmad64bis:fix/opencode-400-model-unavailable-lock

Conversation

@maxmad64bis

@maxmad64bis maxmad64bis commented Sep 10, 2026 •

Copy link
Copy Markdown
Contributor

⚠️ base-red inherited: #12732

Summary

When the opencode upstream reports a model as down (400 "Model is unavailable"), that model stays served: every later request burns a full round-trip to get the same 400 back. Observed case: a free-tier model failed every call for days while remaining selectable.

This PR adds a catalog rule for the opencode family that locks the MODEL (not the connection) with a declared 1h cooldown, self-healing through the operator-controlled clamp (~30min default, escalating in-window). Malformed 400s keep the existing zero-cooldown guard (#2101) — no lock, no escalation.

Two allowlist-adjacent changes, both headlined for review: HONORS_RULE_LOCK_SCOPE_PROVIDERS gains the opencode family so the persistence layer honors the rule's model scope (per #10334), and markAccountUnavailable gains a narrow ruleScope === "model" branch gated on status === 400 (every other status keeps its path — the agentrouter 403 forbidden flow is untouched). FULL_TEXT_RULE_PROVIDERS is deliberately NOT widened: that would short-circuit the #10880 egress-bucketed block, so quota-body 429s keep their current classification. The lockout cooldown is read from the rule itself, not a literal at the call site.

Related Issues

Related to #12777 (upstream 400s scored as account failures — answered here for one narrow, proven-dead case: a multi-day upstream outage is not a shape error, and a self-healing cooldown is not a terminal verdict). Same effect is already available without code via settings.providerErrorRules; this PR is only the catalog default. Catalog eviction is out of scope.

Validation

  • Change type: provider
  • Focused tests and category gates from the golden path
  • npm run lint
  • Reconciled with the current active release base; focused checks rerun afterward
  • Production-code changes include a new or updated automated test in this PR

Ten focused suites, 121/121 pass on the release base (suites named in Tests below). Probe: quota-body 429s stay rate_limit_exceeded with no scope, the pinned 400 returns model_capacity/model. typecheck:core clean; eslint clean on touched files (full-repo lint:json OOMs locally — CI-owned; wider-sweep red is pre-existing, see Reviewer Notes).

Tests Added Or Updated

New tests/unit/opencode-400-model-unavailable.test.ts (verbatim locks, malformed-400 guard, isModelLocked round-trip, 429 headers rule pinned for all four family members, monthly-quota 13-day cooldown with no scope, quota-body markers inert, verbatim terminal off-family, combo classifier regression). agentrouter-error-rules.test.ts: A14 only (HONORS family).

Coverage Notes

The new markAccountUnavailable branch is covered by the isModelLocked round-trip; the status === 400 narrowing is covered by the agentrouter 403 suite (it caught the hijack as a real RED during development).

Reviewer Notes

Three deliberate points. The HONORS widening lets the headers-only quota rule surface its declared connection scope on header-passing paths — the #10880 egress block stays reachable (FULL_TEXT untouched, proven by the marker loop). Branch-ordering note: that branch now runs before the egress block, so an opencode 429 with upstream quota headers takes the connection cooldown and skips sibling cooling — latent today (only chat-completions forwards headers, and the observed opencode 429 envelope is headers-less). The new branch uses a model && provider && guard, no non-null assertion. Wider-sweep red is pre-existing (identical on the unmodified base), so those boxes stay unchecked.

@maxmad64bis
maxmad64bis force-pushed the fix/opencode-400-model-unavailable-lock branch from 91b5330 to 6300969 Compare September 10, 2026 01:09
@maxmad64bis
maxmad64bis marked this pull request as draft September 10, 2026 02:18
@maxmad64bis
maxmad64bis marked this pull request as ready for review September 10, 2026 09:16
@maxmad64bis
maxmad64bis force-pushed the fix/opencode-400-model-unavailable-lock branch 2 times, most recently from 2464ba1 to e15bcf7 Compare September 10, 2026 09:40
@maxmad64bis
maxmad64bis force-pushed the fix/opencode-400-model-unavailable-lock branch from e15bcf7 to 6ce9d86 Compare September 10, 2026 11:50
diegosouzapw added a commit to maxmad64bis/OmniRoute that referenced this pull request Sep 11, 2026
…zapw#13141, diegosouzapw#13146, diegosouzapw#12975)

Final combined values for the batch, set on the first PR merged so every
intermediate merge state is covered. stream.ts is deliberately left alone:
it is already over its freeze on the pure tip with zero contribution from
this batch (base-red diegosouzapw#12732).
@diegosouzapw
diegosouzapw merged commit a19bb22 into diegosouzapw:release/v3.8.51 Sep 11, 2026
9 of 16 checks passed
Githab-capibara added a commit to Githab-capibara/OmniRoute that referenced this pull request Sep 17, 2026
…diegosouzapw#13146)

Scoping the lock to the MODEL rather than the connection is the right layer for a `400 "Model is unavailable"` — a multi-day upstream outage on one model should not darken the account. Good discipline on the two allowlist-adjacent changes: gating the `markAccountUnavailable` branch on `ruleScope === "model"` AND `status === 400` leaves every other status on its existing path, and deliberately not widening `FULL_TEXT_RULE_PROVIDERS` keeps the diegosouzapw#10880 egress-bucketed 429 classification intact. Reading the cooldown from the rule instead of a literal at the call site is what makes it self-healing.

---

Validated in one consolidated worktree cut from `release/v3.8.51`, boarded together with the rest of this batch — zero conflicts between them.

- `typecheck:core` clean; `check:changelog-integrity` OK
- complexity 2799 / baseline 3218 and cognitive-complexity 1265 / baseline 1437 — both under baseline
- 86 focused assertions green across the batch's 10 unit test files, plus 16/16 on the v1 plugin option schema and 16/16 on the v2 option tests
- `check-file-size` rebaselined for this batch's real growth (annotation `_rebaseline_2026_09_11_mergebatch_v3851_maxmad_opencode`, landed on diegosouzapw#13141). `open-sse/utils/stream.ts` was deliberately left frozen: it is already 3115 > 3098 on the pure tip with zero contribution from this batch.

⚠️ base-red inherited: diegosouzapw#12732 — `Docs Gates`, `Merge integrity`, `No new ESLint warnings`, `Unit Tests fast-path` and `Fast Quality Gates` all reproduce on the pure `release/v3.8.51` tip (provider count 356 vs the 358 the modules define, SKILL.md drift, and the `stream.ts` freeze above). None of them touch these diffs.

Thanks @maxmad64bis.
diegosouzapw pushed a commit that referenced this pull request Sep 22, 2026
…egistry level (#14251)

Merged. Thank you, @maxmad64bis.

A registry-level pin for a rule that shipped in #13146 is exactly the kind of guard that stops a future edit from silently widening or dropping it, and the three cases (marked 400 locks the model, unmarked 400 stays terminal, cooldown comes from the rule) pin the behaviour at the right boundary.

Validation before merge: boarded onto the current release tip alongside four sibling PRs of yours — merged clean, no conflict. `tests/unit/opencode-400-model-unavailable-registry.test.ts` green on the combined tree, together with the other boarded suites (36 pass / 0 fail). Test-only change, no production file touched, so nothing to weigh on the size or complexity ratchets.
@maxmad64bis
maxmad64bis deleted the fix/opencode-400-model-unavailable-lock branch September 24, 2026 21:14
muhamadgalihsaputra pushed a commit to niyatna/NiyatnaRoute that referenced this pull request Sep 27, 2026
…diegosouzapw#13146)

Scoping the lock to the MODEL rather than the connection is the right layer for a `400 "Model is unavailable"` — a multi-day upstream outage on one model should not darken the account. Good discipline on the two allowlist-adjacent changes: gating the `markAccountUnavailable` branch on `ruleScope === "model"` AND `status === 400` leaves every other status on its existing path, and deliberately not widening `FULL_TEXT_RULE_PROVIDERS` keeps the diegosouzapw#10880 egress-bucketed 429 classification intact. Reading the cooldown from the rule instead of a literal at the call site is what makes it self-healing.

---

Validated in one consolidated worktree cut from `release/v3.8.51`, boarded together with the rest of this batch — zero conflicts between them.

- `typecheck:core` clean; `check:changelog-integrity` OK
- complexity 2799 / baseline 3218 and cognitive-complexity 1265 / baseline 1437 — both under baseline
- 86 focused assertions green across the batch's 10 unit test files, plus 16/16 on the v1 plugin option schema and 16/16 on the v2 option tests
- `check-file-size` rebaselined for this batch's real growth (annotation `_rebaseline_2026_09_11_mergebatch_v3851_maxmad_opencode`, landed on diegosouzapw#13141). `open-sse/utils/stream.ts` was deliberately left frozen: it is already 3115 > 3098 on the pure tip with zero contribution from this batch.

⚠️ base-red inherited: diegosouzapw#12732 — `Docs Gates`, `Merge integrity`, `No new ESLint warnings`, `Unit Tests fast-path` and `Fast Quality Gates` all reproduce on the pure `release/v3.8.51` tip (provider count 356 vs the 358 the modules define, SKILL.md drift, and the `stream.ts` freeze above). None of them touch these diffs.

Thanks @maxmad64bis.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants