fix(ci): clear five uncovered base-reds on release/v3.8.51 — auto-combo catalog event-loop pin, unweighted quota-share, resilience key set, pack-gate stamp, synced-catalog env doc - #13678
Merged
Conversation
…the first The combo resolver turns an unset step weight into 0 (comboStructure.ts), and #10881 made normalizeWeight treat 0 as disabled plus return definition order when the total weight is 0. A quota-share combo without explicit weights therefore had no DRR quanta and dispatched every request to its first target — the combo-matrix/quota-share integration suite saw openai six times out of six. An all-zero set is now an unweighted combo and shares evenly; an explicit 0 still disables a target when its siblings are weighted. Refs #12732
#12046 derives vision/modalities for the built-in auto/* combos by resolving catalog metadata for every target of every combo. The ~40 auto combos share one candidate pool and the loop neither memoized nor yielded, so the #9147 fixture (60 connections, 720 synced models) went from a ~4s build with a 167ms longest event-loop gap to ~11s and a 860-1070ms gap on an idle box — past the 800ms contract and past #12628's 8s cold-build bound, which is why the test came back 500 catalog_build_timeout on every release-green run. Metadata depends only on the target's provider/model/connection scope within a build, so memoize it per build and yield between misses. Same fixture: 2.3-3.1s build, 56-72ms longest gap. The 9147 test is unchanged. Refs #12732
…y key set #12043 added credentialHealthCheck.intervalMinutes to DEFAULT_RESILIENCE_SETTINGS and to the /api/resilience GET projection. It is operator configuration (the background sweep cadence), not runtime breaker state, but the exact key-set assertion was never updated, so resilience-http-e2e failed on the release tip. The providerBreakers/runtime absence checks stay as they were. Refs #12732
check:pack-artifact assembles dist/ through build:cli, which never writes dist/BUILD_SHA (only build:release does). #12959 pointed the provenance ref at HEAD, but the #10427 guard still stops at 'dist/BUILD_SHA is missing' before it ever reaches the ancestry check — reproduced on tip + #13635 + #13436, the first tree whose Turbopack build compiles. ci.yml sequences build -> stamp -> validate; the validator now does the same in both entry points, keeping PACK_GATE_ENV for the validate step. The guard is unchanged: an unstamped dist/ or one built from another commit still fails. On that tree the stamped gate passes: 'BUILD_SHA 5cb3ae5d9 is on the release line'. Refs #12732
This was referenced Sep 14, 2026
#13248 (#12849) added the override for when a connection's synced model list stops being authoritative, but neither .env.example nor ENVIRONMENT.md lists it, so the env/docs contract gate reports it as code-only. The other five vars that gate reports are already added by #13635 and #13361; this touches a different region of .env.example so it does not collide with either. Refs #12732
diegosouzapw
changed the title
Owner
Author
|
CI triage for
The files this PR touches all pass in CI: |
This was referenced Sep 15, 2026
diegosouzapw
added a commit
that referenced
this pull request
Sep 16, 2026
…(stryker, CLI i18n, paid-target fixture, call-log traceId, Jina prefix, callLogStats import, gitleaks) (#13747) * fix(ci): clear the release/v3.8.51 base-reds left by the 09-15 batch — stryker coverage, CLI ready_timeout key, paid-target fixture, call-log traceId, Jina custom prefix Every PR into release/v3.8.51 pushed after #13635/#13678 still failed Fast Quality Gates and all four Unit fast-path shards on the same 16 tests. Each one reproduces on the pure tip; none is a product defect: - mutation-test-coverage: noauth-model-lockout and local-token-budget-429-skips-cooldown (#13606) were missing from stryker.conf.json tap.testFiles. - cli-i18n-catalog: --ready-timeout calls t("serve.ready_timeout") with no catalog entry; added to en, zh-CN and zh-TW (the parity-checked locales). - paid-model-target(-routes)-6540: #13407 removed Together's one-time credit from the free catalog, so "together/..." classifies as unknown and the save-time guard correctly lets it through. Fixture is now gemini/gemini-3.1-pro-preview, plus a precondition test on the fixtures. - attempt-logging-early-keepalive-merge / video-bridge-log-redaction: #13546 keys the call-log row on traceId; baseCtx now defaults traceId to pendingRequestId (same pattern as chatcore-attempt-logging). The keepalive test also moves to the 30s wall-clock poll deadline video-bridge uses. - models-catalog-route: custom Jina rows keep the jina-ai/ prefix; #13403 changed the custom assertion to jina/ (only synced rows use the alias). Refs #12732 * fix(ci): clear the four reds the first r4 CI run surfaced — callLogStats duplicate import, Uzbek gitleaks false positive, redaction probe traceId, file-size - src/lib/db/callLogStats.ts: the #13641 merge left ERROR_TYPE_CONTRACT imported twice (TS2300), failing API Route Typecheck and check:dashboard-typecheck on every PR. - .gitleaks.toml: the Uzbek catalog from #13727 translates outputTokenDesc as "Yakunlash/javob tokenlari"; generic-api-key reads it as a token value. - dashboard-request-failed-redaction-probe: reads the persisted row by traceId (#13546); with pendingRequestId it asserts null. - models-catalog-route: drop the explanatory comment, which pushed the frozen file over its size cap; the rationale lives in the changelog fragment. Refs #12732 * fix(ci): re-freeze the two test files #13748/#13749 grew past their file-size caps PR-mode check:file-size relaxes source files against the base but not testFrozen, so image-generation-handler.test.ts (2133->2235, #13748) and batch_api.test.ts (1345->1348, #13749) failed Fast Quality Gates on every PR, this one included. Caps set to the merged LOC, with the justification entry. Refs #12732 * fix(ci): register free-badge-provider-gate (#13645) in stryker tap.testFiles #13645 landed a covering test for src/sse/services/auth.ts without the stryker entry, so the strict mutation-test-coverage gate went red again. Refs #12732 * fix(ci): clear two more base-reds the #13440/#13439 merges added - stryker.conf.json: register daily-reset-tz-threading (#13440), which covers accountFallback.ts and rrState.ts. - .gitleaks.toml: allowlist the PROTECTED_PRIORITY_INFRA_502_ENABLED flag id (#13439); generic-api-key reads its key: as a token (secrets ratchet 0 -> 1). Refs #12732 * docs(changelog): tidy the stryker base-red fragment wording Refs #12732
muhamadgalihsaputra
pushed a commit
to niyatna/NiyatnaRoute
that referenced
this pull request
Sep 27, 2026
…bo catalog event-loop pin, unweighted quota-share, resilience key set, pack-gate stamp, synced-catalog env doc (diegosouzapw#13678) * fix(combo): rotate unweighted quota-share targets instead of pinning the first The combo resolver turns an unset step weight into 0 (comboStructure.ts), and diegosouzapw#10881 made normalizeWeight treat 0 as disabled plus return definition order when the total weight is 0. A quota-share combo without explicit weights therefore had no DRR quanta and dispatched every request to its first target — the combo-matrix/quota-share integration suite saw openai six times out of six. An all-zero set is now an unweighted combo and shares evenly; an explicit 0 still disables a target when its siblings are weighted. Refs diegosouzapw#12732 * fix(models): resolve auto-combo target metadata once per catalog build diegosouzapw#12046 derives vision/modalities for the built-in auto/* combos by resolving catalog metadata for every target of every combo. The ~40 auto combos share one candidate pool and the loop neither memoized nor yielded, so the diegosouzapw#9147 fixture (60 connections, 720 synced models) went from a ~4s build with a 167ms longest event-loop gap to ~11s and a 860-1070ms gap on an idle box — past the 800ms contract and past diegosouzapw#12628's 8s cold-build bound, which is why the test came back 500 catalog_build_timeout on every release-green run. Metadata depends only on the target's provider/model/connection scope within a build, so memoize it per build and yield between misses. Same fixture: 2.3-3.1s build, 56-72ms longest gap. The 9147 test is unchanged. Refs diegosouzapw#12732 * test(resilience): list credentialHealthCheck in the configuration-only key set diegosouzapw#12043 added credentialHealthCheck.intervalMinutes to DEFAULT_RESILIENCE_SETTINGS and to the /api/resilience GET projection. It is operator configuration (the background sweep cadence), not runtime breaker state, but the exact key-set assertion was never updated, so resilience-http-e2e failed on the release tip. The providerBreakers/runtime absence checks stay as they were. Refs diegosouzapw#12732 * fix(ci): stamp BUILD_SHA before the release-green pack gate validates check:pack-artifact assembles dist/ through build:cli, which never writes dist/BUILD_SHA (only build:release does). diegosouzapw#12959 pointed the provenance ref at HEAD, but the diegosouzapw#10427 guard still stops at 'dist/BUILD_SHA is missing' before it ever reaches the ancestry check — reproduced on tip + diegosouzapw#13635 + diegosouzapw#13436, the first tree whose Turbopack build compiles. ci.yml sequences build -> stamp -> validate; the validator now does the same in both entry points, keeping PACK_GATE_ENV for the validate step. The guard is unchanged: an unstamped dist/ or one built from another commit still fails. On that tree the stamped gate passes: 'BUILD_SHA 5cb3ae5d9 is on the release line'. Refs diegosouzapw#12732 * docs(env): document OMNIROUTE_SYNCED_CATALOG_STALE_AFTER_MS diegosouzapw#13248 (diegosouzapw#12849) added the override for when a connection's synced model list stops being authoritative, but neither .env.example nor ENVIRONMENT.md lists it, so the env/docs contract gate reports it as code-only. The other five vars that gate reports are already added by diegosouzapw#13635 and diegosouzapw#13361; this touches a different region of .env.example so it does not collide with either. Refs diegosouzapw#12732
muhamadgalihsaputra
pushed a commit
to niyatna/NiyatnaRoute
that referenced
this pull request
Sep 27, 2026
…(stryker, CLI i18n, paid-target fixture, call-log traceId, Jina prefix, callLogStats import, gitleaks) (diegosouzapw#13747) * fix(ci): clear the release/v3.8.51 base-reds left by the 09-15 batch — stryker coverage, CLI ready_timeout key, paid-target fixture, call-log traceId, Jina custom prefix Every PR into release/v3.8.51 pushed after diegosouzapw#13635/diegosouzapw#13678 still failed Fast Quality Gates and all four Unit fast-path shards on the same 16 tests. Each one reproduces on the pure tip; none is a product defect: - mutation-test-coverage: noauth-model-lockout and local-token-budget-429-skips-cooldown (diegosouzapw#13606) were missing from stryker.conf.json tap.testFiles. - cli-i18n-catalog: --ready-timeout calls t("serve.ready_timeout") with no catalog entry; added to en, zh-CN and zh-TW (the parity-checked locales). - paid-model-target(-routes)-6540: diegosouzapw#13407 removed Together's one-time credit from the free catalog, so "together/..." classifies as unknown and the save-time guard correctly lets it through. Fixture is now gemini/gemini-3.1-pro-preview, plus a precondition test on the fixtures. - attempt-logging-early-keepalive-merge / video-bridge-log-redaction: diegosouzapw#13546 keys the call-log row on traceId; baseCtx now defaults traceId to pendingRequestId (same pattern as chatcore-attempt-logging). The keepalive test also moves to the 30s wall-clock poll deadline video-bridge uses. - models-catalog-route: custom Jina rows keep the jina-ai/ prefix; diegosouzapw#13403 changed the custom assertion to jina/ (only synced rows use the alias). Refs diegosouzapw#12732 * fix(ci): clear the four reds the first r4 CI run surfaced — callLogStats duplicate import, Uzbek gitleaks false positive, redaction probe traceId, file-size - src/lib/db/callLogStats.ts: the diegosouzapw#13641 merge left ERROR_TYPE_CONTRACT imported twice (TS2300), failing API Route Typecheck and check:dashboard-typecheck on every PR. - .gitleaks.toml: the Uzbek catalog from diegosouzapw#13727 translates outputTokenDesc as "Yakunlash/javob tokenlari"; generic-api-key reads it as a token value. - dashboard-request-failed-redaction-probe: reads the persisted row by traceId (diegosouzapw#13546); with pendingRequestId it asserts null. - models-catalog-route: drop the explanatory comment, which pushed the frozen file over its size cap; the rationale lives in the changelog fragment. Refs diegosouzapw#12732 * fix(ci): re-freeze the two test files diegosouzapw#13748/diegosouzapw#13749 grew past their file-size caps PR-mode check:file-size relaxes source files against the base but not testFrozen, so image-generation-handler.test.ts (2133->2235, diegosouzapw#13748) and batch_api.test.ts (1345->1348, diegosouzapw#13749) failed Fast Quality Gates on every PR, this one included. Caps set to the merged LOC, with the justification entry. Refs diegosouzapw#12732 * fix(ci): register free-badge-provider-gate (diegosouzapw#13645) in stryker tap.testFiles diegosouzapw#13645 landed a covering test for src/sse/services/auth.ts without the stryker entry, so the strict mutation-test-coverage gate went red again. Refs diegosouzapw#12732 * fix(ci): clear two more base-reds the diegosouzapw#13440/diegosouzapw#13439 merges added - stryker.conf.json: register daily-reset-tz-threading (diegosouzapw#13440), which covers accountFallback.ts and rrState.ts. - .gitleaks.toml: allowlist the PROTECTED_PRIORITY_INFRA_502_ENABLED flag id (diegosouzapw#13439); generic-api-key reads its key: as a token (secrets ratchet 0 -> 1). Refs diegosouzapw#12732 * docs(changelog): tidy the stryker base-red fragment wording Refs diegosouzapw#12732
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Clears five release-green failures on
release/v3.8.51that no open PR covers. I found them by running the full slow gate set (unit, vitest, integration, pack-artifact) on an idle 32-core box against the tip plus #13635. Each red was then re-checked on the untouched tip, so none of them comes from an open PR.Two of the five are product bugs, not stale tests:
GET /v1/modelspinned the event loop for about a second, and overran the 8s cold-build bound (fix(catalog): derive auto-combo modalities from effective target pool #12046). The built-inauto/*combos resolve catalog metadata for every target of every combo. The ~40 auto combos draw on the same candidate pool, and the loop neither memoized nor yielded. On the fix(providers): Overload occurs when there are too many connections/models. #9147 fixture (60 connections, 720 synced models) the build went from ~4s with a 167ms longest gap (parent of fix(catalog): derive auto-combo modalities from effective target pool #12046) to ~11s with an 860–1070ms gap on an idle box. That breaks the 800ms contract and trips fix(api): bound hung GET /v1/models catalog rebuilds #12628's 8s bound, which is why9147-catalog-eventloop-yieldcame back500 catalog_build_timeouton every release-green run. A CPU profile put ~95% of each long stretch in that.map, with half of it ingetProviderPrefixes. The fix memoizes each distinct target once per build and yields between misses. Same fixture: a 2.3–3.1s build with a 56–72ms longest gap. The 9147 test is unchanged. The 400→800ms bump in fix(ci): clear the base-reds the afternoon merge batch left on release/v3.8.51 (round 5: provider count 352, TS2554/TS2677) #12144, landed a few hours after fix(catalog): derive auto-combo modalities from effective target pool #12046, had been hiding this.quota-sharecombo without explicit weights sent every request to its first target (feat(routing): adaptive feedback loop v2 — operational/semantic quality, confidence, TTFT/ITL, end-to-end test #10881). The combo resolver turns an unset step weight into 0 (comboStructure.ts). feat(routing): adaptive feedback loop v2 — operational/semantic quality, confidence, TTFT/ITL, end-to-end test #10881 madenormalizeWeighttreat 0 as disabled and return definition order when the total weight is 0, so an unweighted combo had no DRR quanta.combo-matrix/quota-sharesaw openai 6/6. An all-zero set now shares evenly, and an explicit 0 still disables a target next to weighted siblings. Production-mintedqtSd/combos always carryweight: 100, so they were not affected. Any hand-built quota-share combo was.The other three (the env doc is listed under Change):
resilience-http-e2e: the configuration-only key-set assertion predates feat(resilience): operator-configurable global credential health check interval #12043, which addedcredentialHealthCheck(the sweep interval) to the/api/resilienceprojection. It is configuration, not runtime breaker state. TheproviderBreakers/runtimeabsence checks are untouched.check:pack-artifactbuilds throughbuild:cli, which never writesdist/BUILD_SHA. With fix(ci): clear the base-reds on release/v3.8.51 — doc counts, catalog test flake, pack provenance + pack policy #12959'sOMNIROUTE_RELEASE_REF=HEADthe guard still stops at "dist/BUILD_SHA is missing", reproduced on tip + fix(ci): clear the release/v3.8.51 base-reds on the PR fast path #13635 + fix(build): stop the client bundle from reaching server-only modules, and make the guard find them #13436, the first tree whose Turbopack build compiles. The validator now builds, then stamps, then validates, the same orderci.ymluses. The guard itself is unchanged. On that tree the stamped gate passes withBUILD_SHA 5cb3ae5d9 is on the release line.Change
src/app/api/v1/models/catalogHelpers.ts:memoizeTargetMetadata()keys on provider/model/connection scope and yields after each miss.catalog.tswires it into the auto loop; net +2 lines, within the file-size freeze.open-sse/services/combo/quotaShareStrategy.ts:applyDrrtreats an all-zero weight set as unweighted.scripts/quality/validate-release-green.mjs:runPackArtifactGate()runs build:cli, then write-build-sha, then check:pack-artifact withPACK_GATE_ENV. Both entry points use it.tests/integration/resilience-http-e2e.test.ts: addscredentialHealthCheckto the expected key set..env.example+docs/reference/ENVIRONMENT.md: documentOMNIROUTE_SYNCED_CATALOG_STALE_AFTER_MS(fix(nvidia): NVIDIA provider unusable: static nvidiaHostedModels.snapshot.json allowlist is stale (12 of 14 ids gone upstream) #12849), which nothing else covers. This one surfaced on this PR's own Docs Gates run. The same gate also reports five more vars (OMNIROUTE_PRESSURE_SELF_RESTART*,OMNIROUTE_SQLJS_WASM_PATH,OMNIROUTE_VACUUM_MIN_DELETED_ROWS,OMNIROUTE_WAL_*), already added by fix(ci): clear the release/v3.8.51 base-reds on the PR fast path #13635 and fix(ci): clear two release-green HARD failures on v3.8.51 #13361. I left those alone to avoid a third collision, so Docs Gates stays red here until one of them lands.Validation
quota-share-strategy: the new "unweighted steps still alternate" case fails on the tip (ek-unweighted-1×4), passes after.validate-release-green: the new pack-gate sequencing case fails against the tip script, passes after (32/32).9147-catalog-eventloop-yield, unmodified: fails on the tip, passes after, 3/3 runs on the idle box.combo-matrix/quota-share2/2 andresilience-http-e2e5/5 (plus 3 pre-existing skips).11947-auto-combo-modalities,combo-openrouter-modalities-12613,catalog-helpers-extraction,models-catalog-combo-metadata,model-token-limit-catalog,12058-models-catalog-canonical-self-aliased,12627-catalog-inflight-timeout, plus the newcatalog-target-metadata-memo-9147: 48/48 on the loaded devbox.quota-share-strategy,quota-combo-balancing,combo-routing-engine,v388-quota-share-usage-guard,combo-cooldown-retry,serial/combo-quota-share-cooldown-wait-timing: 203/203.typecheck:coreclean,check:open-sse-typecheck0 errors,check:complexity-ratchetsOK (2821/3218, 1272/1437),check:changelog-integrityOK, ESLint 0 errors on changed files.check:file-sizeshows only the pre-existingopen-sse/utils/stream.tsdrift.895dda383(after fix(ci): clear the orphan base-reds on release/v3.8.51 — the #12867 pipeline extraction, a legacy-DB boot abort and the catalog readers #13349, fix(ci): clear the base-reds on release/v3.8.51 — doc counts, catalog test flake, pack provenance + pack policy #12959 and docs(i18n): sync provider and migration counts in the eight new llm.txt mirrors #13674). Merges cleanly with fix(build): stop the client bundle from reaching server-only modules, and make the guard find them #13436, fix(build): decouple client combo control center and provider registry from server-only imports (#13474) #13565, fix(tests): retire claude-3-5-sonnet-20241022 from the combo integration suites #13056, test: realign two stale assertions with product behavior (#13313) #13315, fix(models): return 503 with Retry-After when a cold catalog build exceeds its time bound #13438 and feat(api): add per-key allowAutoCombos to gate the built-in auto/* combos #13670.Reviewer notes: what is still red after this, and who owns it
ComboControlCenterClient.tsximportopen-sse/services/model.ts, whose dynamic@/lib/db/*imports drag playwright/sharp into the client bundle. fix(build): stop the client bundle from reaching server-only modules, and make the guard find them #13436 and fix(build): decouple client combo control center and provider registry from server-only imports (#13474) #13565 both fix it and collide with each other onsrc/lib/combos/controlCenter.ts; fix(build): decouple client combo control center and provider registry from server-only imports (#13474) #13565 also collides with fix(ci): clear the release/v3.8.51 base-reds on the PR fast path #13635 onsrc/lib/oauth/constants/oauth.ts. Measured: tip + fix(ci): clear the release/v3.8.51 base-reds on the PR fast path #13635 + fix(build): stop the client bundle from reaching server-only modules, and make the guard find them #13436 compiles and passes pack policy. Tip + fix(build): decouple client combo control center and provider registry from server-only imports (#13474) #13565 alone compiles but fails pack policy until fix(ci): clear the release/v3.8.51 base-reds on the PR fast path #13635'sopencode-plugin-v2allowance lands.claude-3-5-sonnet-20241022fixture: fix(tests): retire claude-3-5-sonnet-20241022 from the combo integration suites #13056.chatcore-compression-integration×2 (pt-BR output style): test: realign two stale assertions with product behavior (#13313) #13315 is open, but see the comment there. The English output comes fromresolveOutputStyleLanguageauto-detection returningenon zero hints, which overrides the combo's explicitpt-BRpack. The fixture change in test: realign two stale assertions with product behavior (#13313) #13315 routes around that path instead of covering it, so this needs a product decision.qdrant-routesembedding-models ×3: a contract conflict. feat: list Qdrant embeddings from configured providers #11249/[bug] Qdrant embedding-model list excludes local (no-API-key) providers like Ollama #11949 list only configured providers; fix(memory): generic embedding/rerank provider listing + runtime fallback for all configured providers #11390 merges the whole curated registry plus an unconditional default. The owner has to pick one.