Repository navigation
fix(resilience): skip terminal connections in token health check sweep (#8182) - #8286
Merged
diegosouzapw merged 1 commit intoJul 23, 2026
Conversation
#8182) Background token health check sweep was probing connections with terminal statuses (credits_exhausted / banned / expired) on every cycle, wasting CPU and network. These connections can never self-heal via token refresh — they need manual re-auth or credit top-up. Add a terminal status guard in checkConnection() that mirrors the existing isTerminalConnectionStatus() in auth.ts and TERMINAL_CONNECTION_STATUSES in connectionRecovery.ts. Verified by reporter: after manually disabling 11 credits_exhausted connections, CPU dropped from ~53% to ~2.7%. This fix automates that skip so the sweep never touches terminal connections in the first place.
Collaborator
|
Thanks, @Dingding-leo! 🙏 Merged — skips terminal connections in the token health check (no point probing banned/expired/exhausted). Validated on the merge-train. |
diegosouzapw
merged commit Jul 23, 2026
99ad15a
into
cheaper-inference:release/v3.8.49
5 checks passed
diegosouzapw
added a commit
that referenced
this pull request
Jul 24, 2026
…no_refresh_token state (#8389) Cause: #8182's terminal-connection guard in checkConnection() returns early for any testStatus in {credits_exhausted, banned, expired} to stop the sweep from wasting CPU/network probing connections that can never self-heal. But testStatus="expired" + errorCode="no_refresh_token" is exactly the state the pre-existing GitHub Copilot self-heal targets (isGitHubAccessTokenOnlyConnection + canClearGitHubNoRefreshTokenState, ~line 83-97 / 413-492): a Copilot connection with no OAuth refresh token but a still-valid copilotToken, which the sweep is supposed to flip back to "active". With the new guard placed ahead of that block unconditionally, the self-heal became unreachable for exactly the state it exists to clear. Impact: healthy GitHub Copilot connections that once lost their OAuth refresh token got stuck at testStatus="expired" in the dashboard forever, even though their Copilot sub-token kept working and the sweep would have cleared the stale status back to "active" every cycle before #8182. Fix: carve out the exact recoverable shape from the terminal-skip guard — testStatus==="expired" && errorCode==="no_refresh_token" && isGitHubAccessTokenOnlyConnection(conn) — so the guard still skips every other terminal case (credits_exhausted, banned, and "expired" for any other reason) untouched, matching #8182's original intent. Validation: tests/unit/token-health-no-refresh-token-expired-5326.test.ts was red (1 fail / 4 pass) before the fix — "checkConnection clears stale no_refresh_token state for usable GitHub Copilot connections" asserted testStatus flips back to "active" but got "expired". Green after the fix (6/6, including a new boundary test proving a GitHub Copilot connection expired for any OTHER reason, e.g. errorCode "invalid_grant", is still skipped untouched). Also reran the adjacent checkConnection/tokenHealthCheck suites (token-health-check.test.ts, token-health-check-circuit-breaker.test.ts, apikey-connection-health-check.test.ts, token-health-check-sweep.test.ts, token-health-check-tickms-defined.test.ts, tokenHealthCheck-batchSize.test.ts, codex-oauth-refresh-persist-6352.test.ts, oauth-providers-error-handling.test.ts) — all green, confirming #8182's terminal-skip behavior is otherwise unchanged. npm run typecheck:core clean. Refs #8182 Refs #8286 Refs #5326
Merged
HouMinXi
pushed a commit
to HouMinXi/OmniRoute
that referenced
this pull request
Aug 2, 2026
cheaper-inference#8182) (cheaper-inference#8286) Background token health check sweep was probing connections with terminal statuses (credits_exhausted / banned / expired) on every cycle, wasting CPU and network. These connections can never self-heal via token refresh — they need manual re-auth or credit top-up. Add a terminal status guard in checkConnection() that mirrors the existing isTerminalConnectionStatus() in auth.ts and TERMINAL_CONNECTION_STATUSES in connectionRecovery.ts. Verified by reporter: after manually disabling 11 credits_exhausted connections, CPU dropped from ~53% to ~2.7%. This fix automates that skip so the sweep never touches terminal connections in the first place. Co-authored-by: Austin Liu <austinliu@Austins-MacBook-Air-3.local>
HouMinXi
pushed a commit
to HouMinXi/OmniRoute
that referenced
this pull request
Aug 2, 2026
…no_refresh_token state (cheaper-inference#8389) Cause: cheaper-inference#8182's terminal-connection guard in checkConnection() returns early for any testStatus in {credits_exhausted, banned, expired} to stop the sweep from wasting CPU/network probing connections that can never self-heal. But testStatus="expired" + errorCode="no_refresh_token" is exactly the state the pre-existing GitHub Copilot self-heal targets (isGitHubAccessTokenOnlyConnection + canClearGitHubNoRefreshTokenState, ~line 83-97 / 413-492): a Copilot connection with no OAuth refresh token but a still-valid copilotToken, which the sweep is supposed to flip back to "active". With the new guard placed ahead of that block unconditionally, the self-heal became unreachable for exactly the state it exists to clear. Impact: healthy GitHub Copilot connections that once lost their OAuth refresh token got stuck at testStatus="expired" in the dashboard forever, even though their Copilot sub-token kept working and the sweep would have cleared the stale status back to "active" every cycle before cheaper-inference#8182. Fix: carve out the exact recoverable shape from the terminal-skip guard — testStatus==="expired" && errorCode==="no_refresh_token" && isGitHubAccessTokenOnlyConnection(conn) — so the guard still skips every other terminal case (credits_exhausted, banned, and "expired" for any other reason) untouched, matching cheaper-inference#8182's original intent. Validation: tests/unit/token-health-no-refresh-token-expired-5326.test.ts was red (1 fail / 4 pass) before the fix — "checkConnection clears stale no_refresh_token state for usable GitHub Copilot connections" asserted testStatus flips back to "active" but got "expired". Green after the fix (6/6, including a new boundary test proving a GitHub Copilot connection expired for any OTHER reason, e.g. errorCode "invalid_grant", is still skipped untouched). Also reran the adjacent checkConnection/tokenHealthCheck suites (token-health-check.test.ts, token-health-check-circuit-breaker.test.ts, apikey-connection-health-check.test.ts, token-health-check-sweep.test.ts, token-health-check-tickms-defined.test.ts, tokenHealthCheck-batchSize.test.ts, codex-oauth-refresh-persist-6352.test.ts, oauth-providers-error-handling.test.ts) — all green, confirming cheaper-inference#8182's terminal-skip behavior is otherwise unchanged. npm run typecheck:core clean. Refs cheaper-inference#8182 Refs cheaper-inference#8286 Refs cheaper-inference#5326
muhamadgalihsaputra
pushed a commit
to niyatna/NiyatnaRoute
that referenced
this pull request
Sep 27, 2026
cheaper-inference#8182) (cheaper-inference#8286) Background token health check sweep was probing connections with terminal statuses (credits_exhausted / banned / expired) on every cycle, wasting CPU and network. These connections can never self-heal via token refresh — they need manual re-auth or credit top-up. Add a terminal status guard in checkConnection() that mirrors the existing isTerminalConnectionStatus() in auth.ts and TERMINAL_CONNECTION_STATUSES in connectionRecovery.ts. Verified by reporter: after manually disabling 11 credits_exhausted connections, CPU dropped from ~53% to ~2.7%. This fix automates that skip so the sweep never touches terminal connections in the first place. Co-authored-by: Austin Liu <austinliu@Austins-MacBook-Air-3.local>
muhamadgalihsaputra
pushed a commit
to niyatna/NiyatnaRoute
that referenced
this pull request
Sep 27, 2026
…no_refresh_token state (cheaper-inference#8389) Cause: cheaper-inference#8182's terminal-connection guard in checkConnection() returns early for any testStatus in {credits_exhausted, banned, expired} to stop the sweep from wasting CPU/network probing connections that can never self-heal. But testStatus="expired" + errorCode="no_refresh_token" is exactly the state the pre-existing GitHub Copilot self-heal targets (isGitHubAccessTokenOnlyConnection + canClearGitHubNoRefreshTokenState, ~line 83-97 / 413-492): a Copilot connection with no OAuth refresh token but a still-valid copilotToken, which the sweep is supposed to flip back to "active". With the new guard placed ahead of that block unconditionally, the self-heal became unreachable for exactly the state it exists to clear. Impact: healthy GitHub Copilot connections that once lost their OAuth refresh token got stuck at testStatus="expired" in the dashboard forever, even though their Copilot sub-token kept working and the sweep would have cleared the stale status back to "active" every cycle before cheaper-inference#8182. Fix: carve out the exact recoverable shape from the terminal-skip guard — testStatus==="expired" && errorCode==="no_refresh_token" && isGitHubAccessTokenOnlyConnection(conn) — so the guard still skips every other terminal case (credits_exhausted, banned, and "expired" for any other reason) untouched, matching cheaper-inference#8182's original intent. Validation: tests/unit/token-health-no-refresh-token-expired-5326.test.ts was red (1 fail / 4 pass) before the fix — "checkConnection clears stale no_refresh_token state for usable GitHub Copilot connections" asserted testStatus flips back to "active" but got "expired". Green after the fix (6/6, including a new boundary test proving a GitHub Copilot connection expired for any OTHER reason, e.g. errorCode "invalid_grant", is still skipped untouched). Also reran the adjacent checkConnection/tokenHealthCheck suites (token-health-check.test.ts, token-health-check-circuit-breaker.test.ts, apikey-connection-health-check.test.ts, token-health-check-sweep.test.ts, token-health-check-tickms-defined.test.ts, tokenHealthCheck-batchSize.test.ts, codex-oauth-refresh-persist-6352.test.ts, oauth-providers-error-handling.test.ts) — all green, confirming cheaper-inference#8182's terminal-skip behavior is otherwise unchanged. npm run typecheck:core clean. Refs cheaper-inference#8182 Refs cheaper-inference#8286 Refs cheaper-inference#5326
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #8182. Background token health check sweep was probing connections with terminal statuses (
credits_exhausted/banned/expired) on every cycle, wasting CPU and network. These connections can never self-heal via token refresh — they need manual re-auth or credit top-up.Root Cause
checkConnection()insrc/lib/tokenHealthCheck.tsonly checkedisActivebefore probing. A connection withtestStatus: "credits_exhausted"andisActive: truewas still probed every sweep cycle (~60s). The request path (auth.ts) and recovery path (connectionRecovery.ts) both skip terminal statuses, but the health check sweep did not.Reporter confirmed: after manually disabling 11
credits_exhaustedconnections, CPU dropped from ~53% to ~2.7%.Changes
src/lib/tokenHealthCheck.ts: Add terminal status guard incheckConnection()— skip connections withtestStatusin{credits_exhausted, banned, expired}. Mirrors the existingisTerminalConnectionStatus()insrc/sse/services/auth.tsandTERMINAL_CONNECTION_STATUSESinsrc/lib/quota/connectionRecovery.ts.Verification
npx tsc --pretty false -p tsconfig.typecheck-core.json— zero errorstests/unit/auth-terminal-status.test.ts+tests/unit/apikey-connection-health-check.test.ts— 15/15 pass1 file changed, +11