Repository navigation
fix(ci): clear the orphan base-reds on release/v3.8.51 — the #12867 pipeline extraction, a legacy-DB boot abort and the catalog readers - #13349
Conversation
…vider pipeline #12867 lifted the non-2xx branch out of chatCore.ts into providerExecutionPipeline.ts::toOutcome, but reimplemented it instead of delegating, dropping three behaviors the non-streaming failure path relies on: 1. A non-JSON upstream body fell into the inline JSON.parse catch and surfaced as the (empty) statusText — "upstream error" — discarding the text the client needs. Now parseUpstreamError() supplies the message, as chatCore did. 2. The body-derived retry-after ("Please retry after 20s") was never parsed: retryAfterMs was hard-coded null into applyStatusRestatement/createErrorResult. 3. recordRateLimitBody was plumbed through PipelineStateHooks and wired at both chatCore call sites, but never called — so updateFromResponseBody(), which drains the runtime reservoir on a body-derived 429, silently stopped running for every request routed through this pipeline. restatement rules also now match against the real upstream payload rather than the request body that transformedBody carried whenever the parse failed. Regression guard: tests/unit/chat-rate-limit-body-lock.test.ts (2 base-red failures on the release/v3.8.51 tip) — both green; provider-execution-pipeline 13/13; typecheck:core clean.
#12832 declared `idx_cl_request_provider ON call_logs(request_type, provider)` inside SCHEMA_SQL. That block runs before ensureCallLogsColumns() heals a legacy call_logs table, so on any lineage predating the request_type column the CREATE INDEX aborted the whole schema exec with "no such column: request_type" and the server never finished opening the database. Move the index next to the other request_type/combo indexes in ensureCallLogsColumns(), which runs after the ALTER TABLE healing (and is also called on the in-memory path), so both fresh and upgraded databases get it. Proven by tests/unit/db-core-init.test.ts, "legacy call_logs schemas are upgraded before combo target indexes are created" — failing on the release tip, green now.
… pools
prepareVirtualAutoComboInputs() reads getCustomModels(providerId) straight into
`for (const m of customModels) if (m.id ...)`. That blob is operator-writable and
is returned as raw parsed JSON, so a null / non-object row threw "Cannot read
properties of null (reading 'id')" and EVERY auto/* combo failed to materialize
("[catalog] Could not materialize built-in auto model auto/<id>"), silently
degrading the whole zero-setup routing surface to its minimal catalog entries.
Filter the rows to objects first, the same way the /v1/models custom-model pass
already does. Regression test in combo-auto-pool-visible-only.test.ts fails with
the original TypeError before the guard.
#12934 unioned customModels into getAllActiveSyncedModels() alongside the dispatch-time readers it was actually fixing (#12597: getActiveSyncedCatalog, reconcileProvidersWithActiveSyncedCatalog, getActiveProvidersWithSyncedModel). getAllActiveSyncedModels() is not a dispatch reader. Its three consumers read it as "what the provider's live sync reported": /v1/models feeds its synced-emission loop (and has a separate custom-model pass right after, which owns the specialty-registry dedupe, hidePaid and the vision overrides), /api/models uses it to decide whether an exclusive-listing provider suppresses a static row, and getSyncedAutoAliases derives tier aliases from it. Blurring custom rows into "synced" made a custom embedding/rerank model on jina-ai come out as the alias row `jina/<id>` (parentless primary) with the registry's canonical `jina-ai/<id>` demoted to its child — the inverse of the identity every other specialty model of that provider carries, and it also dropped the registry's `dimensions`. Drop the union there only; the dispatch trio keeps it, so #12597's tests and the 400-on-picker-model fix are untouched. Locked by a new case in custom-models-live-catalog-12597.test.ts and by both jina cases in models-catalog-route.test.ts ("does not duplicate imported/custom Jina specialty models"), which encode the two identities side by side.
… modules #12746 (executeTarget → gates/attempt/loop) and #12811 (round-robin) moved the combo dispatcher out of open-sse/services/combo.ts. The #5976 invariant moved with it intact — executeTargetAttempt.ts:1169 and roundRobinCombo.ts:1049 both still exclude a 500/429 on a per-model-quota provider before recording a provider cooldown — but the guard only read combo.ts and went red on an empty file. Scan the whole combo surface instead of one filename and require the skip in EVERY module that calls recordProviderCooldown(), with the pattern matched on whitespace-normalized source so a prettier rewrap cannot silently disarm it. The invariant asserted is unchanged; the guard is now stronger than the single substring check it replaces.
…traction Four drifts, all from PRs merged into the tip on 2026-09-07: - #12867 extracted chatCore.ts's streaming execution loop into chatCore/providerExecutionPipeline.ts. Its two getProviderCredentials() sites now go through the injected `connection.getProviderCredentials` handle, which the bare-identifier AST scan never saw — the sites would have left the inventory unnoticed. Count property-access calls too and inventory the new file. - #12867 also re-expressed the codex 429 managed-lease fence: the inline `provider === "codex" && !managedLease` became `allowAccountRotation: !managedLease && …` in chatCore.ts, gated in the pipeline as `canRotateAccount`. Assert both halves of that seam instead of the vanished inline form. - #12746 moved combo.ts's getProviderConnectionById into combo/executeTargetGates.ts (class B, unchanged). - #12805 added the Grok Build reset-credit path: src/lib/usage/grokResetCredits.ts (class B — same isConnectionUnavailableToAuxiliaryActivity fence as its codex sibling, so it also joins auxiliaryIsolationSources) and src/app/api/usage/codex-reset-credit/route.ts (class C — resolves the connection's provider to pick a library, never to serve a request). Also pin CATALOG_BUILD_TIMEOUT_MS in models-catalog-route.test.ts: #12627's 8s cold-build bound is sized for a warm production process, and a tsx test runner building the full catalog from a fresh SQLite file crosses it (10-13s observed), returning a `catalog_build_timeout` error body with no `data` array. The bound's own behavior stays covered by 12627-catalog-inflight-timeout.test.ts.
…-basereds-orphans
…-basereds-orphans
…eline parseUpstreamError() already returns errorCode/errorType; the pipeline was dropping both on the floor. Gates that key on the pair — notably isAntigravityMissingProjectError — never fired, so a config-class 422 turned into a generic account cooldown instead of a fail-closed answer. Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
…racts Regression guards for the two defects above: the pipeline must surface the upstream error code/type, and a Kiro error frame now terminates the stream after forwarding response.failed (the #12506 contract the sibling stream-passthrough suite already encodes). Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
…mage errors Three more regressions from the #12867 pipeline extraction, all red on the base tip: - the non-streaming leg stopped refreshing credentials after a 401 and stopped persisting failure state, so a Copilot token was never retried and per-model quota locks lost their helper references; - handleImageGeneration (codex) crashed reading fields off a body that #12506 now sanitizes before it reaches the classifier; - the pipeline discarded the raw upstream body, which provider-error classification needs (quota vs rate-limit vs ban) — surfaced as rawMessage/upstreamBody, both internal to the classifier. Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
…l-arg fix #13009 taught the CLI registry parser to read positionals declared with .addArgument() but did not re-run the generator, so check:agent-skills-sync has been red on the base tip ever since. Regenerated with the official generator (--apply --only=cli-tunnel); the only delta is 'tunnel create' gaining its [type] positional. Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
|
Two new base-reds appeared on the tip while this branch was being built, both reproduced on the clean tip
Also confirmed on the clean tip, so still not this branch's defect: |
…e-v3.8.51-basereds-orphans
Four base-reds that appeared on the tip today, none owned by any open PR: - providers-constants-split / provider-node-reserved-prefix: GreenPT (#13024) and EURouter (#13025) move the totals 238 -> 240 and 408 -> 410. - provider-translate-path-golden: same two providers, regenerated with UPDATE_GOLDEN=1; the snapshot delta is purely additive. - glm-sse-transform-arity: #12925 declared the buffer-size slot GLM had been filling since #12179. The invariant is unchanged — the value must land in a real parameter, never a dropped positional — so the guard now checks it from the helper side and requires the named constant at the call site instead of a magic literal. - injection-guard-scan-bound-3932: #13104 split the 16 KB budget between head and tail, so 'outside the window' is now the middle of an oversized body, not its end. The 16 KB ceiling these cases protect is untouched. Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
|
Another four base-reds surfaced on the tip from today's merges, none of them owned by an open PR, so they are fixed here in
The last two are worth a second look because they invert what the old assertion said, so I kept each invariant explicitly rather than relaxing it:
Still not fixed here and still red: |
…d sweep - stryker.conf.json: register three covering tests that drifted out of tap.testFiles on the base tip (combo-predicates-epoch-cooldown, opencode-400-model-unavailable, chat-correlation-id-exhaustion). Reproduced on the clean tip, so this is inherited drift, not this branch's. - file-size: models-catalog-route.test.ts was already at its frozen ceiling, so the one-line CATALOG_BUILD_TIMEOUT_MS pin cannot be absorbed. Comment trimmed to a single trailing note and testFrozen bumped 1652 -> 1653 with the rationale recorded in the baseline. Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
|
Status of the quality gates on this branch, after Cleared here:
Needs your call —
I tried three ways out and rejected all three, so you are not re-treading them:
So the options are yours: accept the +2/+1 and re-ratchet at release, or land this and track the decomposition under #3501. Reverting the fix instead would put |
…e-v3.8.51-basereds-orphans # Conflicts: # config/quality/file-size-baseline.json
…ough added #12653 made /v1/images/edits fall through to the next combo target, and each per-target attempt resolves credentials of its own — one more classified site in that route (5 -> 6). Drift from the base tip, not from this branch. Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
# Conflicts: # config/quality/file-size-baseline.json # open-sse/handlers/chatCore.ts # open-sse/handlers/chatCore/providerExecutionPipeline.ts # src/lib/db/models/activeSyncedCatalog.ts # tests/unit/hard-session-lease-bypass-inventory.test.ts # tests/unit/kiro-tool-call-validation.test.ts
… to what the tip still lacks Reconciled against release/v3.8.51. The non-streaming credential refresh and body-derived rate-limit lock this PR restored landed first via #13069, and the codex image-generation stringify via #12945, so those halves are dropped in favour of the tip's implementation. What remains is what the tip did not have: the upstream error code/type (lost in the #12867 extraction, which let a config-class Antigravity 422 degrade into an account cooldown), the legacy call_logs boot abort, the malformed custom-models row guard, and the realigned test guards.
…inference#12867 pipeline extraction, a legacy-DB boot abort and the catalog readers (cheaper-inference#13349) Merged after a real reconciliation — this PR and cheaper-inference#13069 fixed the same cheaper-inference#12867 regression (the non-streaming leg losing failure classification and credential refresh) with different strategies, and cheaper-inference#13069 landed first. Rather than stacking two implementations, the tip's was kept and this PR was trimmed to what the tip still lacked. **Dropped, already on the tip:** - non-streaming credential refresh and failure-state persistence → cheaper-inference#13069 (`applyProviderFailureClassification`); the `persistProviderFailureState` hook this PR added would have been dead code - body-derived rate-limit lock and non-JSON body message → already in the tip's pipeline (`chat-rate-limit-body-lock` passes there) - codex image-generation stringify of a sanitized error body → cheaper-inference#12945 (`stringifyImageErrorForLog`, which would otherwise be declared twice — TS2393) - the two test realignments (`hard-session-lease` inventory wording, kiro stream reader) → cheaper-inference#12945 **Kept, missing on the tip:** - **upstream error `code`/`type` in the pipeline error outcome** — ported onto the tip's implementation. Running this PR's own test against the tip returned `errorCode: undefined` instead of `missing_project_id`, so a config-class Antigravity 422 still degraded into an account cooldown. - legacy `call_logs` boot abort (index created after column healing) - malformed operator custom-models row no longer kills every `auto/*` pool (`virtualFactory.ts` 1219 → 1230, annotated) - realigned guards **Evidence on the reconciled tree** - 104 of 106 focused assertions. The 2 red (`models-catalog-route`, `provider-node-reserved-prefix`) fail identically on the pure tip. Against the tip, this PR turns 7 previously red cases green. - ESLint, `typecheck:core`, `check:open-sse-typecheck` (0 errors), `check:changelog-integrity`: all clean - changelog fragment rewritten to claim only what this PR still delivers Unblocking this also surfaced a repo-wide red: the eight llm.txt mirrors added by cheaper-inference#13660 kept the pre-cheaper-inference#13216/cheaper-inference#13248 counts, which failed the pre-commit `docs-sync` gate for everyone. Fixed separately in cheaper-inference#13674. `skills/cli-tunnel/SKILL.md` needed no change — cheaper-inference#13216 already landed the identical edit.⚠️ base-red inherited: cheaper-inference#12732
Clears the orphan half of the base-red on
release/v3.8.51(#12732) — the failures that no other open PR was covering. Eleven test files that are red on the clean tipaf49d497are green here.No new ESLint warnings(owned by #13215),Fast Quality Gates→pack-policy(owned by #12929) andopen-sse-typecheck(owned by #13216), plus the redaction family owned by #12945 / #13036 / #13227. None of those are this branch's defect and none are touched here.The dominant root cause is #12867, not #12506
Four of the five production fixes trace to the same commit: #12867 (
d6f31501) extracted the provider execution leg out ofchatCore.tsintoopen-sse/handlers/chatCore/providerExecutionPipeline.tsand re-implemented the non-2xx branch with an inlineJSON.parseinstead of delegating toparseUpstreamError(). Four behaviors went missing, silently, for every request routed through the new pipeline:parseUpstreamError()message extraction429surfaced as the emptystatusText—upstream error— instead of the upstream wordingretryAfterMsPlease retry after 20snever drained the runtime reservoir, so the limiter was never lockedrecordRateLimitBody(plumbed throughPipelineStateHooks, wired at two call sites, never called)updateFromResponseBody()stopped running entirely on this patherror.code/error.typeisAntigravityMissingProjectError— could never fire, so a config-class422degraded into a generic 5s account cooldownThe same extraction also moved the post-response block behind
if (stream), which dropped the401/403credential refresh and the failure-state persistence for the non-streaming leg: a Copilot token was never retried after a401, and per-model quota locks lost their helper references. Both legs now drive one shared refresh.Everything else
src/lib/db/core.ts— a real boot bug. fix(db): call_logs provider stats read true on empty and legacy data #12832 declaredCREATE INDEX idx_cl_request_provider ON call_logs(request_type, provider)insideSCHEMA_SQL, which runs beforeensureCallLogsColumns()heals a legacy table. On any install predatingrequest_typethe wholedb.exec(SCHEMA_SQL)aborts withno such columnand the server never opens the database. The index moved next to the otherrequest_typeindexes, after the healing.autoCombo/virtualFactory.tsiteratedgetCustomModels()raw; onenullrow in the operator-writable blob killed everyauto/*pool with aTypeError.activeSyncedCatalog.ts— fix(catalog): union picker customModels into the dispatch-time live catalog (#12597) #12934 unionedcustomModelsintogetAllActiveSyncedModels()alongside the dispatch readers it actually needed. That reader is not a dispatch reader, and/v1/modelshas its own custom-model pass right after, so a custom embedding came out as a parentlessjina/<id>primary with the registry'sjina-ai/<id>demoted to its child. The dispatch trio keeps the union; only this reader loses it.imageGeneration.ts— codex image errors crashed reading fields off a body that fix(security): harden public error boundaries #12506's sanitization had already rewritten.Guards were re-pointed, never relaxed
Three source-scanning guards broke because the code they read moved, not because the invariant broke. Each was rewritten to follow the code and each came out stricter:
combo-provider-cooldown-siblingnow requires the per-model-quota skip in every module that callsrecordProviderCooldown(, not just incombo.ts(the dispatcher moved tocombo/executeTargetAttempt.tsandcombo/roundRobinCombo.tsin refactor(combo): split executeTarget into gates, attempt, and loop #12746/refactor(combo): move handleRoundRobinCombo into roundRobinCombo.ts #12811).hard-session-lease-bypass-inventorynow counts property-access calls too, which is how fix(chat): continue after a server-owned tool on Chat Completions #12867's extracted file reachesgetProviderCredentials— the bare-identifier scan would have let those two sites leave the inventory unnoticed. Three assertions replace the one dead regex for the codex managed-lease fence.chatcore-hierarchical-admissionfollows the composite-slot acquisition into the split admission loops.kiro-tool-call-validationwas the one genuinely stale test: #12506 made an upstream error frame terminate the stream after forwardingresponse.failed, a contract its siblingstream-passthrough-error-redactionalready encodes for nine cases. The harness was switched to a reader; all four content assertions are intact and one was added.Validation
af49d497antigravity-missing-project-chatchat-rate-limit-body-lockchatcore-hierarchical-admissionchatcore-translation-pathscombo-provider-cooldown-siblingdb-core-inithard-session-lease-bypass-inventoryimage-generation-handlerkiro-tool-call-validationmodels-catalog-routeprovider-execution-pipelinenon-streaming-provider-legtypecheck:core,typecheck:noimplicit:coreandcheck:cyclesare clean. A 2664-test battery over everychatcore-*,chat-*, pipeline,db-*and catalog file was run on this branch and on the clean tip for comparison: eight failures are fixed here, and the deltas that only appear on this branch all pass in isolation — they are theCATALOG_BUILD_TIMEOUT_MScold-build flake this PR documents (an 8s bound against a 10-13s tsx cold build under load), not regressions. Repo-widenpm run lintstill reports the three inheritedno-explicit-anyerrors owned by #13215 and nothing from these files.Two production fixes ship with new TDD regression tests that fail on the base with the original error.
Merge-order notes
src/lib/db/core.tsalso appears in fix(db): prune health-check repair backups #13341 and fix(db): close failed initialization connections #13342.tests/unit/models-catalog-route.test.tsalso appears in test: realign two stale assertions with product behavior (#13313) #13315.Refs #12732