Repository navigation
Conversation
478f98d to
a425403
Compare
cheaper-inference#11551) `src/app/api/v1/models/route.ts` passes a third argument to `getUnifiedModelsResponse`: { scheduleBackgroundRefresh: (task) => after(task) } but the function has accepted two parameters since cheaper-inference#9199 removed the injection point, so the object was silently discarded. `catalogCache` never imported `after` at all and kept scheduling the stale-while-revalidate rebuild with `setTimeout(…, 0)`. `setTimeout(…, 0)` only yields a turn of the event loop — the rebuild still runs while the response is in flight, so a catalog build that blocks for hundreds of milliseconds delays the stale body the client was promised. That is the flush-before-rebuild guarantee cheaper-inference#8728 was written for, and it was not in effect. - `catalogCache` gains an exported `BackgroundRefreshScheduler` and a `defaultBackgroundRefreshScheduler` built on `after()`, falling back to a macrotask when called outside a Next request scope (unit tests, scripts) where `after()` throws and flush ordering is meaningless anyway. - `resolveCachedCatalogResponse` takes `scheduleBackgroundRefresh` and `getStaleWhileRevalidateMs` on its existing options object. Neither participates in the cache key. - `getUnifiedModelsResponse` accepts the third argument the route has been passing all along and forwards it. No behaviour change for callers that inject nothing beyond where the rebuild runs. Closes cheaper-inference#11551
|
CI on this PR is red, and I do not think any of it is mine. Recording the check rather than The same six gates fail — Locally, at the base commit this branch is rebased onto ( Happy to rebase again whenever the base is green — it has moved 45 commits since I opened this, |
Summary
src/app/api/v1/models/route.tspasses a third argument togetUnifiedModelsResponse:getUnifiedModelsResponsehas accepted two parameters since #9199 removed the injectionpoint, so that object has been silently discarded.
catalogCache.tsnever importedafteratall and kept scheduling the stale-while-revalidate rebuild with
setTimeout(…, 0).setTimeout(…, 0)only yields one turn of the event loop — the rebuild still runs while theresponse is in flight. A catalog build that blocks for hundreds of milliseconds therefore delays
the stale body the client was promised, which is exactly the flush-before-rebuild guarantee
#8728 exists to provide. It was not in effect.
catalogCachegains an exportedBackgroundRefreshSchedulertype and adefaultBackgroundRefreshSchedulerbuilt onafter(). It falls back to a macrotask whenafter()throws — that happens outside a Next request scope (unit tests, scripts importingthe module directly), where there is no response to flush and the ordering is meaningless.
resolveCachedCatalogResponseacceptsscheduleBackgroundRefreshandgetStaleWhileRevalidateMson its existing options object. Neither participates in the cachekey —
buildCatalogCacheKeystill reads onlyhideAutoCombos/hideNoThinkVariants.getUnifiedModelsResponseaccepts the third argument the route has been passing all along andforwards it.
Related Issues
fix(build): repair broken production build, red lint gate and SWR crash #10198 (
37d70785c7)The issue's diagnosis was already exact — route, call site, line numbers and both prior PRs. I
verified each claim against
release/v3.8.51rather than re-deriving it, and answered the onequestion it left open (below).
Validation
npm run lint— clean on all three files (eslintexit 0)origin/release/v3.8.51at
6eb7bf0ba9(2026-08-26); focused checks rerun after the rebase, same resultstests in
tests/integration/v1-models-swr-response-flush-8728.test.tswere deliberatelyleft red by the issue as the spec; this PR turns them green rather than adding new ones.
They are unmodified.
the /v1/models route wires Next after() as its response-flush-safe scheduler—not okonbase,
okon this branch.an external client receives the stale body before synchronous refresh finishes blocking—I could not run this one here, and I would rather say so than imply I did. It listens on an
AF_UNIX socket at a filesystem path, which Windows refuses with
EACCES; the test skips only onEPERM, so it errors instead of skipping. That is environmental — it fails identically on anunmodified tree.
So I verified its assertions with an identical harness over TCP — same builder, same
BLOCK_MS = 300synchronous rebuild, same injected scheduler deferring tooutgoing.once("finish"),same five assertions:
scheduledCount(must be 1)"old"scheduledCount === 0on base is the whole bug in one number: the scheduler the route passes isnever invoked.
Regression sweep —
tests/unit/*catalog*.test.ts+*models*.test.ts, 1180 tests:1174 pass / 5 fail, identical on base and on this branch. The 5 are pre-existing and
unrelated (two Windows-only
EBUSYteardown unlinks ofstorage.sqlite, plus threelong-standing assertion failures in combo/cursor-agent/GLM-alias tests). In particular all seven
real cases of
model-catalog-cache-swr-8728.test.tspass — including "an expired successfulentry is served immediately while a refresh runs behind it", which exercises the new default
scheduler's fallback path.
Tests Added Or Updated
None added. The contract tests already existed and were red on purpose; this PR makes the
production code satisfy them without touching them.
Coverage Notes
src/app/api/v1/models/catalogCache.tsandcatalog.tsare the production files touched. Thescheduler path is covered by
model-catalog-cache-swr-8728.test.ts(default/fallback scheduler)and
v1-models-swr-response-flush-8728.test.ts(injected scheduler + flush ordering).Reviewer Notes
The open question in the issue — "is
route.tsin scope oftsconfig.typecheck-core.json?" —is answered: no, and that is why the excess argument survived. That config lists 27 explicit
files in
files[]with"include": [], and none is undersrc/app/api/.tscdoes catch thecall: adding the three files to a scratch config extending
tsconfig.typecheck-core.jsonreportson base — and that error is gone on this branch. Comparing the full error sets from that
scratch config: 96 errors on base, 95 here, the only difference being that
TS2554. This PRintroduces no new type errors. (The other 95 are pre-existing noise from running
strictNullChecksover files that were never in that config's scope — not something this PR should churn.)
Widening the typecheck scope to cover
src/app/api/v1/models/would have caught this at thecommit that introduced it, and would catch the next one. I have deliberately not done it
here: it surfaces ~95 unrelated errors in these three files alone, so it is its own piece of
work and its own review, not a rider on a bug fix.
Two smaller judgement calls worth a look:
after()fallback is atry/catch, not a feature check. Next exposes no supported wayto ask "am I in a request scope?", and
after()throwing is its documented behaviour outsideone. The alternative — requiring every non-route caller to inject a scheduler — would turn a
missing argument into a crash in scripts and tests that legitimately have no response to flush.
getStaleWhileRevalidateMsis a function, not a number, so a caller can widen the windowfor one call without this module memoising a value meant to hold once. Only the flush test uses
it today (
() => Number.POSITIVE_INFINITY), to keep the test independent of the 30 s constant.