Repository navigation
fix(guardrails): keep an explicit bare model id when its vision capability is unknown (#14003) - #14844
Conversation
204a488 to
b290dad
Compare
…ility is unknown (cheaper-inference#14003) A model id with no provider prefix resolves its vision capability from MODEL_SPECS and the shared vision-model list only, because the provider registry and models.dev sync lookups inside getResolvedModelCapabilities both require a parsed provider. When neither yields a verdict the capability is `null` (unknown) rather than `false` (proven text-only), and VisionBridgeGuardrail.preCall treated the two identically: it only short-circuits to pass-through on `supportsVision === true`. So any bare wire id missing a spec entry became reroute-eligible. With `kimi-for-coding` the credential guard could not save it either, because hasUsableCredentialsForModel splits on "/" and treats the model name as a provider, finds no connection rows, and reports a hard `false` instead of the `null` that would have failed open. The request was then whole-request rerouted to `command-code/moonshotai/Kimi-K2.6` with no error surfaced: the wrong model answered, billing followed the wrong connection, and call_logs recorded the substitute as intended. Unknown capability on a bare id now keeps the requested model and describes the images, mirroring the existing "keep the credentialed model" path. A successful whole-request reroute is also logged at `warn` with the original and substitute model ids, so a substitution can no longer happen silently. Provider-qualified ids are deliberately unchanged (their registry and synced rows are authoritative, and reclassifying their unknown-vs-text-only treatment is a separate policy decision), as are combos, forced modes, the `auto/` prefix, and every proven-vision pass-through. Refs cheaper-inference#14003
b290dad to
d040735
Compare
|
Re-verification pass (2026-09-26) — no code change, head is still Base reconciliation (current numbers). This PR's regression tests pass in CI. In Complete failing-test inventory on this head (so it is clear nothing here is vision-bridge related):
Zero guardrail or vision-bridge failures across all four shards. Baseline confirmation for the two red jobs.
Passing on this head: Still happy to rebase onto |
143c8a1
into
cheaper-inference:release/v3.8.51
… 4) (#15109) Release-captain base-red fix (v3.8.51 release PR #11442, unit shards 3-4): one production defect (proxyLogger pulled into every settings→proxies import and queried the DB at import time; helper extracted to src/lib/proxyLogHost.ts) and seven contract propagations from #14732, #15044, #15067, #13548, #14117×#14844, #12810. 74/74 across the seven files, 176/176 proxy-log neighbours, typecheck clean.
Summary
VisionBridgeGuardrail.preCall()treated an unknown vision capability the same as a proven text-only one, so any bare wire id missing aMODEL_SPECSentry silently had the whole request rerouted to a different provider's model.warnwith the original and substitute model ids, so a substitution can no longer happen silently.Root cause.
getResolvedModelCapabilities()returnssupportsVision: boolean | null, wherenullmeans "no verdict" (resolveVisionCapability()falls through toreturn nullwhen the static spec, registry, synced row and vision-model list all miss).preCall()only short-circuits to pass-through onsupportsVision === true, sonullandfalsewere indistinguishable and every unknown bare id became reroute-eligible.For a bare id this is a reliable signal, not an accident of the data:
resolveCapabilityInput()yieldsprovider: null, and bothgetRegistryModel()andgetSyncedCapabilityForResolved()return early without a provider. A bare id can therefore only get a verdict fromMODEL_SPECSorisVisionModelId(), sonullmeans OmniRoute has no static knowledge of that wire id at all.Why the credential guard did not save it.
hasUsableCredentialsForModel("kimi-for-coding")doesmodel.split("/")[0], so for a bare id the model name is treated as a provider id, matches noprovider_connectionsrows, and returns a hardfalseinstead of thenullthat would have failed open.preCall()only keeps the requested model when that check returnstrue, so the request was rerouted — exactly the reportedkimi-for-coding->command-code/moonshotai/Kimi-K2.6substitution, with no error surfaced, billing on the wrong connection, andcall_logsrecording the substitute as intended.Why it bites generally. Providers rename and upgrade wire models faster than the static spec table tracks, so any registry/autoSync-sourced model that starts accepting images without a spec entry gets silently substituted. This PR removes that failure mode for bare ids; the data fix for the current K2.8 rename is tracked separately in #14117 (different files, no overlap).
The gate is deliberately narrow — it requires all of: not
auto/, not a forced model, noprovider/prefix in the model string,supportsVision === null, and anot-combodecision. Notes on each:capabilities.provider.capabilities.providerisnullfor many provider-qualified ids whose model is absent from the registry (e.g.deepseek/deepseek-chat), so keying off it misclassifies them as bare.Unchanged: provider-qualified ids, combos and their
no-visionfallback, forceddescribe/reroutemodes, theauto/prefix,visionBridgeRerouteTextOnly, and every proven-vision pass-through.Related Issues
kimi-k2.8-previewspec + registry entry; disjoint files, both are needed)Validation
This is a remote-only contribution (authored through the Git Data API, no local checkout), so the focused loop was not executed locally and no local results are claimed. The commands below are the exact focused loop; the full unit shards, Vitest, the coverage ratchet and the production build run in CI (#8329).
npm run lint— not run locally; delegated to CIrelease/v3.8.51tip05a5c15. The base has since advanced 21 commits (192 files); none of them touchsrc/lib/guardrails/visionBridge.ts, this PR's test file, or its changelog fragment (verified via the compare API), and the PR isMERGEABLEwith no conflict. No rebase was pushed because it would only re-run the full matrix for an identical tree; happy to rebase if preferred.Pre-existing red baseline on this base branch (please read)
release/v3.8.51is already red in the unit shards, independently of this PR. Verified against three recently merged PRs on the same base (#14758, #14796, #14525, #14188) — all fourUnit Tests fast-pathshards reportfailureon their own heads. Recurring pre-existing failures in shard 4/4:tests/unit/chatcore-upstream-timeouts.test.tstests/unit/proxyfetch-upstream-status-capture.test.tstests/unit/i18n-glossary-consistency-check.test.ts,tests/unit/settings-i18n-keys.test.ts,tests/unit/pack-artifact-policy.test.tsFast Quality Gatesfails on exactly the same two gates on baseline PR #14796 (mutation-test-coverage check:dashboard-typecheck) and on the other baselines formutation-test-coverage, so neither is attributable to this change.typecheck:coreandts7-diagnostics-ratchetpass, and the ratchet reports 0 added diagnostics.What CI actually verified on this PR (head
d040735), no local claims:API Route Typecheck,Change Classification,Docs Gates,No new ESLint warnings,Vitest (fast-path),semgrep,semgrep-cloud-platform/scan, andMerge integrity (changelog + generated skills)all pass. Across the four unit shards, no vision-bridge or guardrail test fails, and this PR's new test file passes all 5 cases. The remaining shard failures are the pre-existing ones listed above plususage-pending-sweep,combo-responses-sse-failure-fallback,source-scanner-guards,flush-empty-retry,ghe-copilot, andbuild/mcp-bundle-startup— none of which this PR touches.Two earlier CI runs on this PR did surface real regressions from a first version of the gate, both now fixed:
vision-bridge-auto-reroute.test.ts(2 cases) — the first version keyed "bare" offcapabilities.provider, which isnullfordeepseek/deepseek-chat, so a provider-qualified id was misclassified as bare. Its describe result also populated the module-level describe cache, making the following case observe a cache hit. Fixed by keying off the model string.visionBridge-combo-reroute.test.ts("zero-vision combo reroutes the whole request") — the first version suppressed the reroute for a bare combo name. Fixed by restricting the gate tonot-combo.Tests Added Or Updated
tests/unit/guardrails/vision-bridge-bare-id-unknown-vision-14003.test.ts(new, 5 cases):kimi-for-codingwith unknown capability and a credential check that reportsfalse(the bare-id blind spot) while the reroute target is healthy: the model must not be swapped, noreroutedmeta is emitted, the image is described, and the skip is logged naming the original model. Fails without this fix.deepseek/deepseek-chat(provider-qualified, also unknown) still reroutes and now logsWhole-request vision reroute. Locks the string-prefix boundary and guards against over-broadening.kimi-coding/kimi-for-coding) still reroutes.kimi-k3) still short-circuits before any new logic runs.auto/model. Asserted as "the new gate does not suppress the auto path" rather than via the resulting target model, because the vision router's selection/negative caches are module-level and make the concrete substitute order-dependent; the end-to-end auto reroute is already covered bytests/unit/guardrails/vision-bridge-auto-reroute.test.ts.Each case uses a distinct image URL so the module-level describe cache cannot satisfy a later case without calling the bridge.
Coverage Notes
src/lib/guardrails/visionBridge.tsis production code undersrc/, covered by the test file above. The change adds one derived boolean and onewarnlog; the new branch is exercised by case 1, and case 2 proves it is narrow.Reviewer Notes
hasUsableCredentialsForModelreturning a hardfalsefor an id with no/). It is a real latent issue affecting other callers, so it is left for a separate PR.kimi-k2.8-previewspec, which makeskimi-for-codingresolve to a known vision model and take the existing early pass-through. This PR removes the silent-substitution failure mode for the next bare id that appears without a spec entry. The two touch disjoint files (modelSpecs.ts+ registry vsvisionBridge.ts) and can merge in either order.modelSpecs.ts/visionModels.ts/the Kimi registry — data-side only, no guardrail file), fix(providers): advertise the Kimi Code CLI version 2.1.1 #14842 (Kimi Code CLI version,runtime.tsonly). No guardrail-behaviour claim is open on this issue.