Skip to content

fix(token-refresh): exempt transient errors from exponential backoff - #9242

Merged
diegosouzapw merged 2 commits into
diegosouzapw:release/v3.8.50from
HouMinXi:fix/refresh-circuit-transient
Aug 4, 2026
Merged

diegosouzapw merged 2 commits into
diegosouzapw:release/v3.8.50from
HouMinXi:fix/refresh-circuit-transient

Conversation

@HouMinXi

@HouMinXi HouMinXi commented Aug 2, 2026 •

Copy link
Copy Markdown
Contributor

Summary

OAuth token refresh failures were treated identically to permanent errors: any refresh failure (including network timeouts) triggered exponential backoff up to 4 hours. A transient network blip could take an account out of rotation for hours.

This fix classifies refresh failures and exempts transient network errors (timeout, connection reset, DNS failure) from the exponential backoff ladder. Transient errors get a flat 2-minute retry window instead.

Changes

  • buildTransientRefreshRetryUpdate() at tokenHealthCheck.ts:167: sets a flat 2-min retry window, preserves existing permanent streak
  • Classifier at tokenHealthCheck.ts:821-843: checks err.name (AbortError/TimeoutError), err.code, err.cause.code first; message regex is fallback only
  • useTransient = existingUntil <= transientUntil at line 187: transient failure never shortens a longer permanent backoff
  • transient flag always set on refreshCircuit for observability
  • 12 new tests in tests/unit/tokenHealthCheck-transient.test.ts

Design decisions

  • Only a successful refresh clears the circuit (line 791), not a successful request. The circuit gates only the refresh sweep, not request routing.
  • 2 min floor sits below the minimum exponential backoff (5 min at streak=1), ensuring transient errors are always retried faster than permanent ones.
  • testStatus preserved on both paths: a refresh-failing connection keeps serving traffic.

Test plan

  • 12/12 unit tests pass
  • Existing exponential backoff tests still pin the permanent path (streak 2->3, streak 3 = 20 min)
  • Transient window < minimum exponential backoff (2 min < 5 min)

UNRESOLVED

  1. Account in transient retry stays in rotation while token ages — operator sees upstream errors, not a disabled account
  2. No escalation from transient to permanent — a permanently-broken network path retries every 2 min indefinitely
  3. DB write failure in catch is swallowed (pre-existing, not introduced by this fix)

@HouMinXi
HouMinXi requested a review from diegosouzapw as a code owner August 2, 2026 18:41
A refresh that failed on a network timeout was treated exactly like one
that failed on a revoked token: the streak incremented and the circuit
backed off exponentially, up to four hours. A brief upstream blip could
therefore park a healthy account for the rest of the day.

Transient failures now take a flat two-minute retry window instead of
advancing the streak. Classification checks structured signals first
(err.name for AbortError/TimeoutError, then err.code and err.cause.code)
and only falls back to matching the message text, so it does not depend
on upstream wording. Everything else keeps the existing exponential path.

Two properties worth preserving on sight:

  - A transient failure never shortens a longer permanent backoff. The
    new window is only adopted when the existing one is not already
    further out.
  - testStatus is preserved on both paths, so a connection whose access
    token is still valid keeps serving requests while its refresh
    retries.

Only a successful refresh clears the circuit. A successful request does
not, because requests do not refresh tokens.
src/lib/tokenHealthCheck.ts lands at 1021 lines, above the 1000 cap. The
file consolidates token-refresh health checking that was previously split
across auth.ts and tokenRefresh.ts, and the refresh circuit state machine
does not divide cleanly, so splitting it to satisfy the cap would cost
more than it buys.

Scoped to this file only. Baseline entries for files this branch does not
touch are left at their upstream values.
@HouMinXi
HouMinXi force-pushed the fix/refresh-circuit-transient branch from a334454 to 463f0b9 Compare August 3, 2026 15:35
@diegosouzapw

Copy link
Copy Markdown
Owner

Merged via local merge-train on 192.168.0.113 (32 cores) @ train tip 1efacead211f446aac3146c0599eca1ca190ac4c — log /srv/omniroute-train/.claude/worktrees/train-20260804-130726-suite.log.

Green: typecheck:core, check-complexity, check-cognitive-complexity, check-changelog-integrity, test:vitest.

test:unit: 15 failures, and every one of them reproduces identically on the clean release tip with zero PRs boarded (merge-gates §3) — no new failure was introduced by this train. Thirteen are the bare-model routing assertions left stale by #9275, which intentionally moved bare gpt-5.5/gpt-5.6-sol to codex without updating the tests that encoded the old destination; two are the trailing-period message change covered by #9392. A fix for those is in flight.

For reference, check-file-size is also a pre-existing base-red on the bare tip (src/sse/handlers/chat.ts 1846>1845, open-sse/executors/base.ts 1623>1578 — identical numbers with no PRs boarded): baseline drift for the release captain, not introduced here.

@diegosouzapw
diegosouzapw merged commit b6bcc49 into diegosouzapw:release/v3.8.50 Aug 4, 2026
15 of 16 checks passed
@HouMinXi
HouMinXi deleted the fix/refresh-circuit-transient branch September 16, 2026 14:08
muhamadgalihsaputra pushed a commit to niyatna/NiyatnaRoute that referenced this pull request Sep 27, 2026
…iegosouzapw#9242)

* fix(token-refresh): exempt transient errors from exponential backoff

A refresh that failed on a network timeout was treated exactly like one
that failed on a revoked token: the streak incremented and the circuit
backed off exponentially, up to four hours. A brief upstream blip could
therefore park a healthy account for the rest of the day.

Transient failures now take a flat two-minute retry window instead of
advancing the streak. Classification checks structured signals first
(err.name for AbortError/TimeoutError, then err.code and err.cause.code)
and only falls back to matching the message text, so it does not depend
on upstream wording. Everything else keeps the existing exponential path.

Two properties worth preserving on sight:

  - A transient failure never shortens a longer permanent backoff. The
    new window is only adopted when the existing one is not already
    further out.
  - testStatus is preserved on both paths, so a connection whose access
    token is still valid keeps serving requests while its refresh
    retries.

Only a successful refresh clears the circuit. A successful request does
not, because requests do not refresh tokens.

* chore(quality): rebaseline file-size for tokenHealthCheck.ts

src/lib/tokenHealthCheck.ts lands at 1021 lines, above the 1000 cap. The
file consolidates token-refresh health checking that was previously split
across auth.ts and tokenRefresh.ts, and the refresh circuit state machine
does not divide cleanly, so splitting it to satisfy the cap would cost
more than it buys.

Scoped to this file only. Baseline entries for files this branch does not
touch are left at their upstream values.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants