fix(combo): bound the pre-dispatch unavailable skip so a stale label cannot dark a pool (#12168) - #12285
Merged
Conversation
…cannot dark a pool (#12168) getPersistedConnectionCooldownSkipReason() returned a skip for ANY connection whose testStatus was `unavailable`, with no elapsed-cooldown check: if (status === "unavailable") return `Skipping ...`; That is the raw-label anti-pattern AGENTS.md warns about ("check whether code is reading raw state instead of using getStatus()/canExecute()") — the resilience layers are meant to recover lazily. The sibling helper directly above it, getConnectionStatusQuotaCutoffReason(), does require hasFutureRateLimitUntil() before treating `unavailable` as blocking. Its stated justification — "Lazy recovery is unaffected: clearAccountError() resets the status on first success" — does not hold on this path. This gate runs BEFORE dispatch, so it prevents the very successful request that would call clearAccountError(). And a row whose rateLimitedUntil is absent cannot be rescued by the out-of-band recovery job either, because hasElapsedCooldown() there requires a timestamp to be present. Net effect reported in #12168: an entire combo pool answering ALL_TARGETS_SKIPPED with recordedAttempts === 0 — zero upstream attempts, no path back to healthy. The original intent (do not burst into a connection AUTH just retired, before the timestamp lands) is preserved, but bounded: the bare label is honoured only while lastErrorAt is inside a grace window, mirroring ERROR_LABEL_GRACE_MS in src/lib/quota/connectionRecovery.ts so the two never disagree about whether a label is still meaningful. Past the window the request goes through, and one real attempt either succeeds (clearing the status) or re-arms the cooldown with a fresh timestamp. Regression introduced by #11360, shipped in v3.8.50. Two assertions in repro-combo-persisted-cooldown-preskip.test.ts encoded the buggy behavior as intended ("skips an unavailable connection whose cooldown already expired") and are realigned to the corrected contract, plus a case for the orphan state (unavailable with no timestamps at all).
This was referenced Sep 15, 2026
muhamadgalihsaputra
pushed a commit
to niyatna/NiyatnaRoute
that referenced
this pull request
Sep 27, 2026
…cannot dark a pool (diegosouzapw#12168) (diegosouzapw#12285) getPersistedConnectionCooldownSkipReason() returned a skip for ANY connection whose testStatus was `unavailable`, with no elapsed-cooldown check: if (status === "unavailable") return `Skipping ...`; That is the raw-label anti-pattern AGENTS.md warns about ("check whether code is reading raw state instead of using getStatus()/canExecute()") — the resilience layers are meant to recover lazily. The sibling helper directly above it, getConnectionStatusQuotaCutoffReason(), does require hasFutureRateLimitUntil() before treating `unavailable` as blocking. Its stated justification — "Lazy recovery is unaffected: clearAccountError() resets the status on first success" — does not hold on this path. This gate runs BEFORE dispatch, so it prevents the very successful request that would call clearAccountError(). And a row whose rateLimitedUntil is absent cannot be rescued by the out-of-band recovery job either, because hasElapsedCooldown() there requires a timestamp to be present. Net effect reported in diegosouzapw#12168: an entire combo pool answering ALL_TARGETS_SKIPPED with recordedAttempts === 0 — zero upstream attempts, no path back to healthy. The original intent (do not burst into a connection AUTH just retired, before the timestamp lands) is preserved, but bounded: the bare label is honoured only while lastErrorAt is inside a grace window, mirroring ERROR_LABEL_GRACE_MS in src/lib/quota/connectionRecovery.ts so the two never disagree about whether a label is still meaningful. Past the window the request goes through, and one real attempt either succeeds (clearing the status) or re-arms the cooldown with a fresh timestamp. Regression introduced by diegosouzapw#11360, shipped in v3.8.50. Two assertions in repro-combo-persisted-cooldown-preskip.test.ts encoded the buggy behavior as intended ("skips an unavailable connection whose cooldown already expired") and are realigned to the corrected contract, plus a case for the orphan state (unavailable with no timestamps at all).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
getPersistedConnectionCooldownSkipReason()skipped any connection whosetestStatuswasunavailable, with no elapsed-cooldown check:This is the raw-label anti-pattern
AGENTS.mdexplicitly warns about ("check whether code is reading rawstateinstead of usinggetStatus()/canExecute()") — the resilience layers are designed to recover lazily. The sibling helper immediately above it,getConnectionStatusQuotaCutoffReason(), gets this right: it requireshasFutureRateLimitUntil()before treatingunavailableas blocking.Why the original justification does not hold
The in-code comment argued:
But this gate runs before dispatch — so it blocks the very successful request that would call
clearAccountError(). Chicken-and-egg.And the out-of-band recovery job cannot rescue it either:
hasElapsedCooldown()insrc/lib/quota/connectionRecovery.tsrequires arateLimitedUntilto be present to consider a cooldown elapsed. A row with a staleunavailablelabel and no timestamp is unreachable by both paths.Net effect, exactly as reported in #12168: a whole combo pool answering
ALL_TARGETS_SKIPPEDwithrecordedAttempts === 0— zero upstream attempts made, and no route back to healthy.Regression introduced by #11360, shipped in v3.8.50.
How
The original intent is preserved — do not burst into a connection AUTH just retired, before its timestamp lands — but bounded: the bare label is honoured only while
lastErrorAtis inside a grace window, mirroringERROR_LABEL_GRACE_MSinconnectionRecovery.tsso the two never disagree about whether a label is still meaningful. Past that window the request goes through, and one real attempt either succeeds (clearing the status) or re-arms the cooldown with a fresh timestamp.A missing/unparseable
lastErrorAtis treated as stale, not blocking — an unbounded skip is precisely the reported failure, and one extra upstream attempt is far cheaper than a permanently dark pool.Validation (TDD, Hard Rule #18)
Two assertions in
repro-combo-persisted-cooldown-preskip.test.tsencoded the buggy behavior as intended ("skips an unavailable connection whose cooldown already expired"). They are realigned to the corrected contract — not weakened: the recent-failure case still asserts the skip fires, it just now requireslastErrorAtto make the label meaningful.#12168cases fail againstorigin/release/v3.8.51(verified by restoring the base file and re-running) — 12 pass / 2 failcombo-cooldown-retry,combo-provider-cooldown,combo-provider-cooldown-sibling,chatcore-compression-combo-predicates): 40/40typecheck:coreclean ·eslint(with project suppressions) cleanScope note
Refs #12168rather thanCloses. The validation also flagged a plausible but unproven second path that can produce the same orphan state:markAccountUnavailable()'s zero-cooldown branch (src/sse/services/auth.ts) nullsrateLimitedUntilwhileupdateProviderConnectionmerges, leaving a pre-existingunavailablelabel in place. This PR makes such a row recoverable, but hardening that write so the state is never produced deserves its own change with its own repro.Refs #12168