feat(model-capabilities): drive model capabilities and labels from one manifest - #5597
Merged
Conversation
Flatten the family-rule matcher from six match kinds (exact, prefix, gpt5-token, gpt5-base, segment, segment-prefix) to two (exact, prefix), dropping the per-rule integer match_priority. Materialize all 30 curated Databricks v2 models as complete six-axis exact records so their outcomes are structurally immune to matcher changes (verified byte-identical, all axes, against the prior interpreter as oracle). Add a boundary-aware longest-prefix resolver contract: prefix tokens bind only on -, ., or end-of-string. The reshape preserves every curated outcome and drops only uncurated / adversarial inputs to the provider fallbacks. scripts/model-capabilities-compat.md enumerates all eight delta mechanism classes with grammar, prior result, and reshaped destination; the normative corpus re-points its eight intentional-delta vectors to the new behavior and adds collision, uncurated-token, negative, and boundary vectors (101 vectors total). Data-only: interpreters and consumers land in later phases. Co-authored-by: Will Pfleger <pfleger.will@gmail.com> Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
…pat doc Adds the two highest-value corpus vectors the reshape review flagged as missing: a databricks_v2 gpt-5-<multi-digit> case (class B2, base prefix binds while the openai-responses route is preserved) and a guard for the sole deliberately-divergent exact record (databricks-gpt-5-2). Removes the standalone compat table so the delta contract lives in the PR description, restoring the two-file boundary; corpus notes now point at the PR. Co-authored-by: Will Pfleger <pfleger.will@gmail.com> Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
Reads scripts/model-capabilities.json at compile time (include_str!) through strict serde (deny_unknown_fields + real enums), caches it in a OnceLock, and resolves a model's six-axis capability profile: provider canonicalization inside the resolver, provider-qualified exact lookup, boundary-aware longest family-prefix match with lexicographic tie-break, then provider fallback. validate_manifest() owns the semantic invariants strict parsing can't express. The shared normative corpus is the cross-language contract: a #[test] runs all 103 executable vectors, plus migrated relational/invariant tests and new whitespace/tie-break/complete-result cases. ThinkingEffort gains a Deserialize impl so it can be reused as the manifest effort enum rather than redefined. Co-authored-by: Will Pfleger <pfleger.will@gmail.com> Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
The hardcoded DATABRICKS_V2_KNOWN_MODELS const duplicated data that now lives in scripts/model-capabilities.json. Route the authenticated-empty v2 catalog through the manifest accessor so the JSON is the single runtime source; the const had no consumers outside buzz-agent. Co-authored-by: Will Pfleger <pfleger.will@gmail.com> Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
…d-tables Splice the six legacy effort hand-tables (openai_efforts_for_model, normalize_effort_for_openai_route, clamp_adaptive_effort, anthropic_efforts_for_model, gpt5_token/base_matches) out of config.rs and llm.rs; anthropic_thinking_config and normalize_effort_for_* now resolve capabilities from the manifest interpreter, threading a provider argument. Migrate the test surface to the shared normative corpus: 42 pure-classification hand-table tests deleted (subsumed by the 103-vector corpus + resolver relational tests), 4 databricks_v2 route-normalization regression tests added. Retire the effortTable fixture apparatus (JSON + .test.mjs + the valid_effort_values_for_provider_model guard) — the corpus running in both the Rust and TypeScript harnesses is the sole cross-language drift guard. Co-authored-by: Will Pfleger <pfleger.will@gmail.com> Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
…ifest The desktop TS side now consumes scripts/model-capabilities.json directly (JSON import + zod) through the same @model-capabilities-manifest alias at all four build touchpoints, mirroring @features-manifest — no codegen, no hand-tables. modelCapabilities.ts is a 5-step resolver that mirrors the Rust interpreter (prototype-safe provider fallback, internal canonicalization, registryLabel). getProviderEffortConfig and the label surface (resolveModelLabel, DATABRICKS_MODEL_NAMES derived at runtime from the manifest's databricks_v2 exact_records) are thin projections over it. A node corpus harness replays scripts/normative-corpus.json (103 vectors + registryLabel axis) as the cross-language drift guard against the Rust side. CI path filters gain both scripts/*.json under rust + desktop, and buzz-agent --lib is wired into test-unit / run-tests.sh so the Rust half of the drift guard actually executes in CI. Co-authored-by: Will Pfleger <pfleger.will@gmail.com> Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
…egacy classifier
Production Databricks v2 dispatch classified the wire route with a
segment-matching hand-table (`databricks_v2_route_for_model`), while the
manifest carried an authoritative `databricks_v2_wire_route`. The two
disagreed: the corpus pins `databricks_v2 + goose-opus-5 -> mlflow-chat`
(class F), but the `opus`-segment classifier routed it to anthropic-messages.
The manifest route field therefore had zero production consumers.
The dispatch seam now resolves the route from
`model_capabilities::resolve("databricks_v2", model).databricks_v2_wire_route`,
mapping the manifest enum onto the three concrete wire shapes this path serves.
The legacy `DATABRICKS_V2_OPENAI_CODE_NAMES` / `DATABRICKS_V2_CLAUDE_NAMES`
constants, `model_name_segments`, and `databricks_v2_route_for_model` are gone;
route classification now lives solely in the manifest. The manifest's non-wire
`NotApplicable`/`RouteUnknown` variants are structurally unreachable for a
concrete DBv2 model at dispatch (documented at the seam) and fold to the
mlflow-chat concrete-unknown fallback rather than panicking.
Two seam-level tests replace the deleted hand-table test: one asserts the
manifest-ratified routes through the real dispatch functions (goose-opus-5 ->
mlflow-chat), the other is a mutation-bypass guard proving the seam is a pure
projection of the manifest — reintroducing any segment shortcut fails it.
Also aligns TS Zod manifest schemas from `.loose()` to `.strict()`, mirroring
Rust's `deny_unknown_fields` so a typo'd manifest key fails on desktop instead
of passing silently; documented keys (`_comment`, `_provenance`, `_sources`, …)
are modeled explicitly and covered by accept-verbatim + reject-unknown tests.
Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
Ready-for-review prep for the model-capabilities manifest PR: - Merge origin/main (15 commits) into the branch. The one changed-in-both file, UserProfilePopover.tsx, auto-merged cleanly: main's #5574 extracted interaction handlers into useProfileInteractionActions (disjoint region), leaving the managedAgent.model label render path untouched, so the resolveModelLabel rewire survives with identical semantics. - Re-point scripts/normative-corpus.json:562 _note from the deleted DATABRICKS_V2_OPENAI_CODE_NAMES constant to the manifest mechanism that now owns the behavior (boundary-aligned family_tokens + databricks_v2 concrete-unknown fallback). Doc-only; every expect block byte-identical. * origin/main: fix(desktop): preserve agent mention separator after send (#5623) fix(link-previews): proxy sent preview media (#5627) feat(deletion): add durable whole-community deletion (#4425) fix(desktop): preserve live channel timelines (#5662) Refine channel settings and profile panels (#5574) fix(deps): bump webbrowser to 1.2.4 for RUSTSEC-2026-0257 (#5659) feat(acp): deliver channel description in prompt [Context] (#4552) fix(desktop): launch Databricks OAuth from passive model discovery (#5607) chore(release): release Buzz Desktop version 0.5.10 (#5613) fix(desktop): remove 0.5.9+ perf regressions, speed up get_channels (#5599) perf(desktop): coalesce read state localStorage persistence (#5591) fix(relay): stop panicking the ingest worker on reactions to project events (#5294) fix(desktop): bound initial timeline retention (#5603) fix(relay): log event kind on the HTTP bridge /events line (#5291) feat(tracing): add PostgreSQL tracing spans (#3678) Co-authored-by: Will Pfleger <pfleger.will@gmail.com> Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
…ions The normative corpus was a second hand-maintained encoding of capability behavior: every manifest edit required hand-editing 103 expected outputs or CI went red. Convert it to a Rust-generated golden snapshot so the manifest is the only place answers live. - Add an inputs-only question table (INPUTS) plus a generator that runs the production `resolve()` over every query and serializes CapabilityResult directly — no expected outputs authored anywhere. - `just regen-model-corpus` rewrites the file; `corpus_matches_generated_snapshot` fails CI with a run-regen message if the committed bytes drift. The TS harness still replays the same JSON, so TS==Rust stays machine-checked. - Drop the goose provenance `_sources` entry and rewrite the known-models comment to own the 2-ID list as our curated empty-catalog fallback, removing the stale "opt-in drift check script" claim (no such script). First regen is semantically identical to the prior corpus (0 answer deltas); the committed golden file is byte-identical. Co-authored-by: Will Pfleger <pfleger.will@gmail.com> Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
…ties-manifest * origin/main: Harden shared agent instruction review (#4220) chore(release): release Buzz Desktop version 0.5.11 (#5714) feat(acp): report standard adapter usage (#4950) fix(mobile): settle hydrated threads on latest reply (#4702) perf(desktop): persist channel snapshot hash (#5684) fix(agent): raise output limit and allow 3 recoveries (#5475) fix(desktop): defer foreground resume work (#5696) perf(desktop): coalesce thread-activity localStorage writes (#5693) Batch observer-store publications per relay envelope (#5680) feat(buzz-acp): idle re-sleep for woken lazy pools (#5682) Signed-off-by: Duncan <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz> Co-authored-by: Will Pfleger <pfleger.will@gmail.com> Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
The generator's INPUTS question table encoded expected capability answers (effort sets, routes, thinking modes, fallbacks, precedence) in its id/group/note string literals — a second hand-maintained encoding of behavior that had already drifted stale against ratified deltas. Rewrite every id/group/note to describe only the query and why the input is worth probing; drop answer-only notes to None. Rename the two stale labels (dbv2-goose-opus-5-is-anthropic, gpt5-base guard note). The (provider, raw_model_id) pairs that drive resolve() are unchanged, so the regenerated corpus is semantically identical: 0 expect deltas across all 103 vectors. Co-authored-by: Will Pfleger <pfleger.will@gmail.com> Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
Four INPUTS labels still stated answer relationships (route sourced from a family rule, efforts differing from a family rule, route preserved, snapshot diverging from the base stem) rather than only posing a query. A later manifest change could falsify those relations while regen faithfully preserved the stale prose. Reword them as relational questions and rename two ids accordingly; query pairs, generator, and drift gate untouched, so the regenerated corpus keeps 0 expect deltas across all 103 vectors. Co-authored-by: Will Pfleger <pfleger.will@gmail.com> Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
wpfleger96
marked this pull request as ready for review
August 13, 2026 20:14
… the source buzz-agent's Databricks discovery emits the raw endpoint id as the model name on every path (the API has no display-name field), and the desktop label resolver returns any nonblank discovered name before the registry tier. So the picker showed `databricks-gpt-5-5` instead of `GPT-5.5` on the real discovery contract — the manifest's curated label never ran. Curate `ModelEntry.name` at all four construction seams (v2 discovery, v1 parse, auth-empty fallback, configured fallback) via a read-only `databricks_registry_label` accessor over the manifest's databricks_v2 exact records; `id` stays the raw wire/config value. As an echo defense, `resolveModelLabel` now treats a discovered name equal to the trimmed id as absent so the registry tier runs — covering older buzz-agent binaries and any harness that echoes ids. A genuinely distinct name (including the "(default catalog)" suffixed label) still wins tier 1. Co-authored-by: Will Pfleger <pfleger.will@gmail.com> Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
…ties-manifest * origin/main: (35 commits) chore(release): release Buzz Desktop version 0.5.12 (#5903) fix(mobile): unwrap batched observer telemetry (#5805) perf(desktop): update active turns incrementally (#5897) fix(link-previews): send while previews finish in background (#5697) fix(desktop): cut steady-state relay traffic from polls and read-state echo (#5879) fix(desktop): support channel message path links (#5889) feat(mobile-messages): render compact Buzz permalink chips (#5639) test(desktop): await channel E2E bridge readiness (#5886) fix(link-preview): refetch a link when it re-enters the composer (#5510) feat(desktop-messages): render compact Buzz permalink chips (#5638) Fix video comment effect wrapping (#5748) Teach agents to inherit Buzz product intent (#5875) feat(desktop): one relative date ladder across chat and the Inbox (#3769) fix(desktop): amortize observer journal eviction with a low-water mark (#5808) Unify agent profile content (#5788) Standardize settings section layout (#5855) fix(desktop): share one timer across same-interval useNow consumers (#5861) Clarify immediate spoken huddle replies (#5863) Scope desktop presence subscriptions to active demand (#5830) Polish mobile profiles, DMs, and sheets (#5401) ... Signed-off-by: Duncan <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz>
tlongwell-block
approved these changes
Aug 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Centralizes model capability knowledge — thinking mode, supported effort levels, wire routes, and human-readable labels — into a single manifest,
scripts/model-capabilities.json. Rust and TypeScript each get a small interpreter that reads the same manifest, replacing hand-maintained tables scattered across both languages that had already drifted apart. A capability change is now a data edit, not parallel edits to two code paths. Supersedes the codegen approach explored in #3603.A cross-language contract keeps the two interpreters honest:
scripts/normative-corpus.jsonis a golden snapshot generated from the Rust resolver (103 vectors covering all six capability axes) and replayed natively in TS. CI fails if either language disagrees with the corpus or the corpus drifts from the resolver. Regenerate withjust regen-model-corpus.Behavior changes
openai-compatproviders no longer offersmax. The request path always clampedmaxtoxhighon the wire, so the UI stops offering a value that was silently rewritten. UI-only, wire-identical.goose-opus-5) now route to the MLflow chat wire instead of Anthropic Messages — they lose Anthropic prompt caching but still succeed on a valid OpenAI-compatible wire. Curateddatabricks-claude-*records and any name starting withclaudeare unchanged. A handful of other uncurated/adversarial name shapes similarly fall back to MLflow chat instead of pattern-matched routes; every curated model resolves identically to before, all axes.name({id, name: id}) on every path.ModelEntry.nameis now curated at all four construction seams inbuzz-agent— v2 discovery, v1 parse, the auth-empty default catalog, and the configured-model fallback — via a read-onlydatabricks_registry_labellookup over the manifest'sdatabricks_v2exact records;idstays the raw wire/config value. A known id renders its curated label (databricks-gpt-5-5→GPT-5.5), an unknown id passes through unchanged, and the default-catalog row readsGPT-5.5 (default catalog). As a defense against olderbuzz-agentbinaries and any harness that echoes ids,resolveModelLabeltreats a discovered name equal to the trimmed id as absent and falls through to the registry tier; a genuinely distinct name (including the suffixed default-catalog label) still wins.Cleanup
Deletes the duplicated capability tables and their tests: the
config.rsgpt5 matchers, effort tables, and clamp logic; the legacy segment-based Databricks v2 route classifier inllm.rs; and the TS hand tables pluseffortTable.fixture.json. All are replaced by manifest lookups through the shared resolver — no line of capability data exists in two places.