Skip to content

fix(providers): stop the loopback readiness gate from memorizing failure - #10903

Merged
diegosouzapw merged 2 commits into
diegosouzapw:release/v3.8.50from
maxmad64bis:fix/loopback-gate-memory-success
Aug 21, 2026
Merged

fix(providers): stop the loopback readiness gate from memorizing failure#10903
diegosouzapw merged 2 commits into
diegosouzapw:release/v3.8.50from
maxmad64bis:fix/loopback-gate-memory-success

Conversation

@maxmad64bis

@maxmad64bis maxmad64bis commented Aug 20, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

⚠️ base-red inherited: #9985

Summary

ensureLoopbackServerReady caches __loopbackReadyPromise and never forgets a rejection. The only reset path is a test helper. One failed probe at boot sends the whole process to the in-process fallback forever, and no probe ever runs again.

Waiting callers also log the failure, each of them. That is 270 log lines in 26h for one probe, in five 54-line bursts.

This is the follow-up to P#2221. It removed the 17 concurrent probes; it left the single probe memorizing its failure and logging once per caller. This PR fixes both.

Related Issues

Validation

  • Change type: other (internal self-fetch readiness gate)
  • Focused tests and category gates from the golden path
  • npm run lint
  • Reconciled with the current active release base; focused checks rerun afterward
  • Production-code changes include a new or updated automated test in this PR

Tests Added Or Updated

  • tests/unit/api/sync-models-readiness.test.ts — three new cases: a failed probe is re-attempted after the retry window, 54 concurrent callers inside the window share one rejection, and a readiness failure is logged once per probe instead of once per caller.

Run: node --import tsx/esm --test tests/unit/api/sync-models-readiness.test.ts

# tests 11

# pass 11

# fail 0

Coverage Notes

The three new tests cover the gate, the retry window and the transition log. selfFetchWithRetry keeps its existing coverage through the shared gate. The only mocks are an injected fetch and a console.warn spy.

Reviewer Notes

  • Behavior change: after a failed probe, the next caller re-probes after 30s instead of never. A caller arriving inside the window gets the same rejection it would have gotten, but no new probe is started.
  • The warning text lost the connection label. The transition is process-wide; one line per campaign is the point of this change.
  • Success path unchanged: a resolved probe is still memorized and shared by all callers.
  • No migration, no feature flag.

@maxmad64bis
maxmad64bis force-pushed the fix/loopback-gate-memory-success branch from 756e299 to 8e596fa Compare August 20, 2026 22:20
Max added 2 commits August 21, 2026 00:23
Max
fix(providers): stop the loopback readiness gate from memorizing failure
9808da4 
No-verify approved by the operator (2026-08-20): the check-docs-sync hook is
broken on the base tip (8c4a219 desynchronized the i18n mirrors, 40+
locales, unrelated to this PR). The remaining hook gates were run manually:
lint-staged PASS, any-budget PASS, tracked-artifacts PASS.
Max
chore(changelog): fragment for loopback gate failure memory
640b120 
No-verify approved by the operator (2026-08-20): check-docs-sync is broken
on the base tip (8c4a219), see previous commit.
@maxmad64bis
maxmad64bis force-pushed the fix/loopback-gate-memory-success branch from 8e596fa to 640b120 Compare August 20, 2026 22:23
arminanton added a commit to arminanton/OmniRoute that referenced this pull request Aug 21, 2026
@arminanton
fix(providers): stop the loopback readiness gate from memorizing failure
424a1fb 
Re-implements upstream diegosouzapw#10903 against the current release/v3.8.50 tip.

ensureLoopbackServerReady cached __loopbackReadyPromise unconditionally, so a
single failed readiness probe at boot cached a REJECTED promise forever. Every
subsequent model-sync self-fetch then got that same rejection back (line 188
short-circuit), fell through to the in-process fallback permanently, and no probe
ever ran again — the process never recovered even after the loopback listener
came up moments later. Each waiting caller also logged the failure independently
(~270 log lines in 26h, in bursts).

Fix (src/app/api/providers/[id]/sync-models/route.ts):
- Attach a .catch() to the readiness attempt that nulls __loopbackReadyPromise on
  rejection (only if it is still the current attempt), so the NEXT caller
  re-probes and the process self-heals. A resolved probe stays cached, so a
  healthy process still runs exactly one probe sequence (the boot-race dedup from
  P#2221 is preserved).
- Throttle the readiness-failure warning to at most once per 60s across callers
  (warnReadinessFailureThrottled), so the now-possible re-probes don't amplify
  log volume. __resetLoopbackReadinessForTests resets the throttle too.

Adds two regression tests to tests/unit/api/sync-models-readiness.test.ts: a
failed probe is NOT memoized (a later call re-probes and resolves), and a
successful probe IS memoized (one probe for a healthy process). Verified:
sync-models-readiness 10/10 (8 existing + 2 new), model-sync-route 18/18,
zai-web-model-sync-route 1/1; typecheck:core and lint clean.

Fixes diegosouzapw#10903. Refs diegosouzapw#2221.
@arminanton arminanton mentioned this pull request Aug 21, 2026
Merged
@diegosouzapw
diegosouzapw merged commit 9b95282 into diegosouzapw:release/v3.8.50 Aug 21, 2026
4 of 5 checks passed
arminanton added a commit to arminanton/OmniRoute that referenced this pull request Aug 21, 2026
@arminanton
fix(providers): stop the loopback readiness gate from memorizing failure
b123635 
Re-implements upstream diegosouzapw#10903 against the current release/v3.8.50 tip.

ensureLoopbackServerReady cached __loopbackReadyPromise unconditionally, so a
single failed readiness probe at boot cached a REJECTED promise forever. Every
subsequent model-sync self-fetch then got that same rejection back (line 188
short-circuit), fell through to the in-process fallback permanently, and no probe
ever ran again — the process never recovered even after the loopback listener
came up moments later. Each waiting caller also logged the failure independently
(~270 log lines in 26h, in bursts).

Fix (src/app/api/providers/[id]/sync-models/route.ts):
- Attach a .catch() to the readiness attempt that nulls __loopbackReadyPromise on
  rejection (only if it is still the current attempt), so the NEXT caller
  re-probes and the process self-heals. A resolved probe stays cached, so a
  healthy process still runs exactly one probe sequence (the boot-race dedup from
  P#2221 is preserved).
- Throttle the readiness-failure warning to at most once per 60s across callers
  (warnReadinessFailureThrottled), so the now-possible re-probes don't amplify
  log volume. __resetLoopbackReadinessForTests resets the throttle too.

Adds two regression tests to tests/unit/api/sync-models-readiness.test.ts: a
failed probe is NOT memoized (a later call re-probes and resolves), and a
successful probe IS memoized (one probe for a healthy process). Verified:
sync-models-readiness 10/10 (8 existing + 2 new), model-sync-route 18/18,
zai-web-model-sync-route 1/1; typecheck:core and lint clean.

Fixes diegosouzapw#10903. Refs diegosouzapw#2221.
diegosouzapw pushed a commit that referenced this pull request Aug 23, 2026
@arminanton
Integration: security tier + self-hosted operator blockers (rebased o…
1027682 
…nto v3.8.51) (#10952)

Validated on the resolved merge against the current tip (527da65 + the post-#11281 rebaseline): the single conflict was a comment-only collision in providers/[id]/models/route.ts (kept the tip's #10828-ordering note). Focused suites 125/125 across all 13 touched test files (build-sqlite-stub, cc-compatible, copilot-claude-messages, copilot-gemini-route, executor-github, ghe-copilot, github-copilot-discovery-token, github-copilot-model-discovery, noauth-sibling-7620, provider-header-profiles, provider-models-config, request-log-payloads, upstream-error-passthrough), typecheck:core clean, file-size/changelog-integrity OK. Merged --admin over the inherited 2026-08-23 base-red cluster (#9985) — the reds are proven tip failures (CLI catalog cluster + @testing-library allowlist, being drained by #11280), not from this diff. Note: the rebase means several items the body listed (relay x-relay-path SSRF, /v1/search blocked-providers, #10736 rotation fence, #10903, #10865, #10899, #10916) already landed upstream and are NOT in this delta — the delta is: better-sqlite3 build guard + build heap/worker caps + telemetry-off (#10060 re-derived), credential-echo passthrough refusal + OCR/moderation redaction + call-log key redaction, Copilot CLI 1.0.81-6 wire identity + Claude→/v1/messages name-matched routing + discovery token fix, CC model_not_found 400, compat overrides for no-auth aliases (#7620-pinned). The Copilot wire-identity change is the one to watch in production. Thank you @arminanton — and the ported-author credits in the commit history (@rqzbeh, yidecode, the #10899/#10916 authors) are preserved. Your config-posture finding (REQUIRE_API_KEY default vs 0.0.0.0) is noted for a maintainer decision, as you scoped it.
@maxmad64bis
maxmad64bis deleted the fix/loopback-gate-memory-success branch September 24, 2026 21:13
muhamadgalihsaputra pushed a commit to niyatna/NiyatnaRoute that referenced this pull request Sep 27, 2026
@maxmad64bis
fix(providers): stop the loopback readiness gate from memorizing fail…
2253ff8 
…ure (diegosouzapw#10903)

Validado no worktree combinado: typecheck:core, changelog-integrity, complexity, cognitive-complexity, file-size, lint todos verdes. Fix real bem documentado (loopback readiness gate memorizava falha permanentemente + log-spam por caller). CI vermelho é o base-red já rastreado em diegosouzapw#9985. Obrigado!
muhamadgalihsaputra pushed a commit to niyatna/NiyatnaRoute that referenced this pull request Sep 27, 2026
@arminanton
Integration: security tier + self-hosted operator blockers (rebased o…
6af4bc9 
…nto v3.8.51) (diegosouzapw#10952)

Validated on the resolved merge against the current tip (585c2b7 + the post-diegosouzapw#11281 rebaseline): the single conflict was a comment-only collision in providers/[id]/models/route.ts (kept the tip's diegosouzapw#10828-ordering note). Focused suites 125/125 across all 13 touched test files (build-sqlite-stub, cc-compatible, copilot-claude-messages, copilot-gemini-route, executor-github, ghe-copilot, github-copilot-discovery-token, github-copilot-model-discovery, noauth-sibling-7620, provider-header-profiles, provider-models-config, request-log-payloads, upstream-error-passthrough), typecheck:core clean, file-size/changelog-integrity OK. Merged --admin over the inherited 2026-08-23 base-red cluster (diegosouzapw#9985) — the reds are proven tip failures (CLI catalog cluster + @testing-library allowlist, being drained by diegosouzapw#11280), not from this diff. Note: the rebase means several items the body listed (relay x-relay-path SSRF, /v1/search blocked-providers, diegosouzapw#10736 rotation fence, diegosouzapw#10903, diegosouzapw#10865, diegosouzapw#10899, diegosouzapw#10916) already landed upstream and are NOT in this delta — the delta is: better-sqlite3 build guard + build heap/worker caps + telemetry-off (diegosouzapw#10060 re-derived), credential-echo passthrough refusal + OCR/moderation redaction + call-log key redaction, Copilot CLI 1.0.81-6 wire identity + Claude→/v1/messages name-matched routing + discovery token fix, CC model_not_found 400, compat overrides for no-auth aliases (diegosouzapw#7620-pinned). The Copilot wire-identity change is the one to watch in production. Thank you @arminanton — and the ported-author credits in the commit history (@rqzbeh, yidecode, the diegosouzapw#10899/diegosouzapw#10916 authors) are preserved. Your config-posture finding (REQUIRE_API_KEY default vs 0.0.0.0) is noted for a maintainer decision, as you scoped it.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@diegosouzapw diegosouzapw Awaiting requested review from diegosouzapw diegosouzapw is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@maxmad64bis @diegosouzapw