Skip to content

feat(model-capabilities): drive model capabilities and labels from one manifest - #5597

Merged
wpfleger96 merged 14 commits into
mainfrom
duncan/model-capabilities-manifest
Aug 17, 2026
Merged

feat(model-capabilities): drive model capabilities and labels from one manifest#5597
wpfleger96 merged 14 commits into
mainfrom
duncan/model-capabilities-manifest

Conversation

@wpfleger96

@wpfleger96 wpfleger96 commented Aug 11, 2026

Copy link
Copy Markdown
Member

Summary

Centralizes model capability knowledge — thinking mode, supported effort levels, wire routes, and human-readable labels — into a single manifest, scripts/model-capabilities.json. Rust and TypeScript each get a small interpreter that reads the same manifest, replacing hand-maintained tables scattered across both languages that had already drifted apart. A capability change is now a data edit, not parallel edits to two code paths. Supersedes the codegen approach explored in #3603.

A cross-language contract keeps the two interpreters honest: scripts/normative-corpus.json is a golden snapshot generated from the Rust resolver (103 vectors covering all six capability axes) and replayed natively in TS. CI fails if either language disagrees with the corpus or the corpus drifts from the resolver. Regenerate with just regen-model-corpus.

Behavior changes

  • Effort dropdown for openai-compat providers no longer offers max. The request path always clamped max to xhigh on the wire, so the UI stops offering a value that was silently rewritten. UI-only, wire-identical.
  • Databricks v2 routing (wire-visible): uncurated endpoint names carrying a bare Claude code-name segment (e.g. goose-opus-5) now route to the MLflow chat wire instead of Anthropic Messages — they lose Anthropic prompt caching but still succeed on a valid OpenAI-compatible wire. Curated databricks-claude-* records and any name starting with claude are unchanged. A handful of other uncurated/adversarial name shapes similarly fall back to MLflow chat instead of pattern-matched routes; every curated model resolves identically to before, all axes.
  • Curated model labels on the real discovery path. The Databricks API returns no display name, so discovery emits the raw endpoint id as the model name ({id, name: id}) on every path. ModelEntry.name is now curated at all four construction seams in buzz-agent — v2 discovery, v1 parse, the auth-empty default catalog, and the configured-model fallback — via a read-only databricks_registry_label lookup over the manifest's databricks_v2 exact records; id stays the raw wire/config value. A known id renders its curated label (databricks-gpt-5-5GPT-5.5), an unknown id passes through unchanged, and the default-catalog row reads GPT-5.5 (default catalog). As a defense against older buzz-agent binaries and any harness that echoes ids, resolveModelLabel treats a discovered name equal to the trimmed id as absent and falls through to the registry tier; a genuinely distinct name (including the suffixed default-catalog label) still wins.

Cleanup

Deletes the duplicated capability tables and their tests: the config.rs gpt5 matchers, effort tables, and clamp logic; the legacy segment-based Databricks v2 route classifier in llm.rs; and the TS hand tables plus effortTable.fixture.json. All are replaced by manifest lookups through the shared resolver — no line of capability data exists in two places.

Duncan and others added 5 commits August 11, 2026 16:00
Flatten the family-rule matcher from six match kinds (exact, prefix,
gpt5-token, gpt5-base, segment, segment-prefix) to two (exact, prefix),
dropping the per-rule integer match_priority. Materialize all 30 curated
Databricks v2 models as complete six-axis exact records so their outcomes
are structurally immune to matcher changes (verified byte-identical, all
axes, against the prior interpreter as oracle). Add a boundary-aware
longest-prefix resolver contract: prefix tokens bind only on -, ., or
end-of-string.

The reshape preserves every curated outcome and drops only uncurated /
adversarial inputs to the provider fallbacks. scripts/model-capabilities-compat.md
enumerates all eight delta mechanism classes with grammar, prior result,
and reshaped destination; the normative corpus re-points its eight
intentional-delta vectors to the new behavior and adds collision,
uncurated-token, negative, and boundary vectors (101 vectors total).

Data-only: interpreters and consumers land in later phases.
Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
…pat doc

Adds the two highest-value corpus vectors the reshape review flagged as
missing: a databricks_v2 gpt-5-<multi-digit> case (class B2, base prefix
binds while the openai-responses route is preserved) and a guard for the
sole deliberately-divergent exact record (databricks-gpt-5-2). Removes the
standalone compat table so the delta contract lives in the PR description,
restoring the two-file boundary; corpus notes now point at the PR.

Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
Reads scripts/model-capabilities.json at compile time (include_str!) through
strict serde (deny_unknown_fields + real enums), caches it in a OnceLock, and
resolves a model's six-axis capability profile: provider canonicalization
inside the resolver, provider-qualified exact lookup, boundary-aware longest
family-prefix match with lexicographic tie-break, then provider fallback.
validate_manifest() owns the semantic invariants strict parsing can't express.

The shared normative corpus is the cross-language contract: a #[test] runs all
103 executable vectors, plus migrated relational/invariant tests and new
whitespace/tie-break/complete-result cases. ThinkingEffort gains a Deserialize
impl so it can be reused as the manifest effort enum rather than redefined.

Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
The hardcoded DATABRICKS_V2_KNOWN_MODELS const duplicated data that now
lives in scripts/model-capabilities.json. Route the authenticated-empty
v2 catalog through the manifest accessor so the JSON is the single
runtime source; the const had no consumers outside buzz-agent.

Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
…d-tables

Splice the six legacy effort hand-tables (openai_efforts_for_model,
normalize_effort_for_openai_route, clamp_adaptive_effort,
anthropic_efforts_for_model, gpt5_token/base_matches) out of config.rs and
llm.rs; anthropic_thinking_config and normalize_effort_for_* now resolve
capabilities from the manifest interpreter, threading a provider argument.

Migrate the test surface to the shared normative corpus: 42 pure-classification
hand-table tests deleted (subsumed by the 103-vector corpus + resolver
relational tests), 4 databricks_v2 route-normalization regression tests added.
Retire the effortTable fixture apparatus (JSON + .test.mjs + the
valid_effort_values_for_provider_model guard) — the corpus running in both the
Rust and TypeScript harnesses is the sole cross-language drift guard.

Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
…ifest

The desktop TS side now consumes scripts/model-capabilities.json directly
(JSON import + zod) through the same @model-capabilities-manifest alias at all
four build touchpoints, mirroring @features-manifest — no codegen, no
hand-tables. modelCapabilities.ts is a 5-step resolver that mirrors the Rust
interpreter (prototype-safe provider fallback, internal canonicalization,
registryLabel). getProviderEffortConfig and the label surface
(resolveModelLabel, DATABRICKS_MODEL_NAMES derived at runtime from the
manifest's databricks_v2 exact_records) are thin projections over it.

A node corpus harness replays scripts/normative-corpus.json (103 vectors +
registryLabel axis) as the cross-language drift guard against the Rust side.
CI path filters gain both scripts/*.json under rust + desktop, and
buzz-agent --lib is wired into test-unit / run-tests.sh so the Rust half of
the drift guard actually executes in CI.

Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
@wpfleger96 wpfleger96 changed the title feat(model-capabilities): reshape match kinds to exact+prefix (Phase 1, data only) feat(model-capabilities): drive model capabilities and labels from one manifest Aug 12, 2026
Duncan and others added 6 commits August 12, 2026 12:46
…egacy classifier

Production Databricks v2 dispatch classified the wire route with a
segment-matching hand-table (`databricks_v2_route_for_model`), while the
manifest carried an authoritative `databricks_v2_wire_route`. The two
disagreed: the corpus pins `databricks_v2 + goose-opus-5 -> mlflow-chat`
(class F), but the `opus`-segment classifier routed it to anthropic-messages.
The manifest route field therefore had zero production consumers.

The dispatch seam now resolves the route from
`model_capabilities::resolve("databricks_v2", model).databricks_v2_wire_route`,
mapping the manifest enum onto the three concrete wire shapes this path serves.
The legacy `DATABRICKS_V2_OPENAI_CODE_NAMES` / `DATABRICKS_V2_CLAUDE_NAMES`
constants, `model_name_segments`, and `databricks_v2_route_for_model` are gone;
route classification now lives solely in the manifest. The manifest's non-wire
`NotApplicable`/`RouteUnknown` variants are structurally unreachable for a
concrete DBv2 model at dispatch (documented at the seam) and fold to the
mlflow-chat concrete-unknown fallback rather than panicking.

Two seam-level tests replace the deleted hand-table test: one asserts the
manifest-ratified routes through the real dispatch functions (goose-opus-5 ->
mlflow-chat), the other is a mutation-bypass guard proving the seam is a pure
projection of the manifest — reintroducing any segment shortcut fails it.

Also aligns TS Zod manifest schemas from `.loose()` to `.strict()`, mirroring
Rust's `deny_unknown_fields` so a typo'd manifest key fails on desktop instead
of passing silently; documented keys (`_comment`, `_provenance`, `_sources`, …)
are modeled explicitly and covered by accept-verbatim + reject-unknown tests.

Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
Ready-for-review prep for the model-capabilities manifest PR:
- Merge origin/main (15 commits) into the branch. The one changed-in-both
  file, UserProfilePopover.tsx, auto-merged cleanly: main's #5574 extracted
  interaction handlers into useProfileInteractionActions (disjoint region),
  leaving the managedAgent.model label render path untouched, so the
  resolveModelLabel rewire survives with identical semantics.
- Re-point scripts/normative-corpus.json:562 _note from the deleted
  DATABRICKS_V2_OPENAI_CODE_NAMES constant to the manifest mechanism that
  now owns the behavior (boundary-aligned family_tokens + databricks_v2
  concrete-unknown fallback). Doc-only; every expect block byte-identical.

* origin/main:
  fix(desktop): preserve agent mention separator after send (#5623)
  fix(link-previews): proxy sent preview media (#5627)
  feat(deletion): add durable whole-community deletion (#4425)
  fix(desktop): preserve live channel timelines (#5662)
  Refine channel settings and profile panels (#5574)
  fix(deps): bump webbrowser to 1.2.4 for RUSTSEC-2026-0257 (#5659)
  feat(acp): deliver channel description in prompt [Context] (#4552)
  fix(desktop): launch Databricks OAuth from passive model discovery (#5607)
  chore(release): release Buzz Desktop version 0.5.10 (#5613)
  fix(desktop): remove 0.5.9+ perf regressions, speed up get_channels (#5599)
  perf(desktop): coalesce read state localStorage persistence (#5591)
  fix(relay): stop panicking the ingest worker on reactions to project events (#5294)
  fix(desktop): bound initial timeline retention (#5603)
  fix(relay): log event kind on the HTTP bridge /events line (#5291)
  feat(tracing): add PostgreSQL tracing spans (#3678)

Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
…ions

The normative corpus was a second hand-maintained encoding of capability
behavior: every manifest edit required hand-editing 103 expected outputs or
CI went red. Convert it to a Rust-generated golden snapshot so the manifest
is the only place answers live.

- Add an inputs-only question table (INPUTS) plus a generator that runs the
  production `resolve()` over every query and serializes CapabilityResult
  directly — no expected outputs authored anywhere.
- `just regen-model-corpus` rewrites the file; `corpus_matches_generated_snapshot`
  fails CI with a run-regen message if the committed bytes drift. The TS
  harness still replays the same JSON, so TS==Rust stays machine-checked.
- Drop the goose provenance `_sources` entry and rewrite the known-models
  comment to own the 2-ID list as our curated empty-catalog fallback,
  removing the stale "opt-in drift check script" claim (no such script).

First regen is semantically identical to the prior corpus (0 answer deltas);
the committed golden file is byte-identical.

Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
…ties-manifest

* origin/main:
  Harden shared agent instruction review (#4220)
  chore(release): release Buzz Desktop version 0.5.11 (#5714)
  feat(acp): report standard adapter usage (#4950)
  fix(mobile): settle hydrated threads on latest reply (#4702)
  perf(desktop): persist channel snapshot hash (#5684)
  fix(agent): raise output limit and allow 3 recoveries (#5475)
  fix(desktop): defer foreground resume work (#5696)
  perf(desktop): coalesce thread-activity localStorage writes (#5693)
  Batch observer-store publications per relay envelope (#5680)
  feat(buzz-acp): idle re-sleep for woken lazy pools (#5682)

Signed-off-by: Duncan <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz>
Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
The generator's INPUTS question table encoded expected capability
answers (effort sets, routes, thinking modes, fallbacks, precedence) in
its id/group/note string literals — a second hand-maintained encoding of
behavior that had already drifted stale against ratified deltas.

Rewrite every id/group/note to describe only the query and why the input
is worth probing; drop answer-only notes to None. Rename the two stale
labels (dbv2-goose-opus-5-is-anthropic, gpt5-base guard note). The
(provider, raw_model_id) pairs that drive resolve() are unchanged, so
the regenerated corpus is semantically identical: 0 expect deltas across
all 103 vectors.

Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
Four INPUTS labels still stated answer relationships (route sourced
from a family rule, efforts differing from a family rule, route
preserved, snapshot diverging from the base stem) rather than only
posing a query. A later manifest change could falsify those relations
while regen faithfully preserved the stale prose. Reword them as
relational questions and rename two ids accordingly; query pairs,
generator, and drift gate untouched, so the regenerated corpus keeps 0
expect deltas across all 103 vectors.

Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
@wpfleger96
wpfleger96 marked this pull request as ready for review August 13, 2026 20:14
@wpfleger96
wpfleger96 requested a review from a team as a code owner August 13, 2026 20:14
Duncan and others added 2 commits August 14, 2026 16:22
… the source

buzz-agent's Databricks discovery emits the raw endpoint id as the model
name on every path (the API has no display-name field), and the desktop
label resolver returns any nonblank discovered name before the registry
tier. So the picker showed `databricks-gpt-5-5` instead of `GPT-5.5` on
the real discovery contract — the manifest's curated label never ran.

Curate `ModelEntry.name` at all four construction seams (v2 discovery,
v1 parse, auth-empty fallback, configured fallback) via a read-only
`databricks_registry_label` accessor over the manifest's databricks_v2
exact records; `id` stays the raw wire/config value. As an echo defense,
`resolveModelLabel` now treats a discovered name equal to the trimmed id
as absent so the registry tier runs — covering older buzz-agent binaries
and any harness that echoes ids. A genuinely distinct name (including the
"(default catalog)" suffixed label) still wins tier 1.

Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
…ties-manifest

* origin/main: (35 commits)
  chore(release): release Buzz Desktop version 0.5.12 (#5903)
  fix(mobile): unwrap batched observer telemetry (#5805)
  perf(desktop): update active turns incrementally (#5897)
  fix(link-previews): send while previews finish in background (#5697)
  fix(desktop): cut steady-state relay traffic from polls and read-state echo (#5879)
  fix(desktop): support channel message path links (#5889)
  feat(mobile-messages): render compact Buzz permalink chips (#5639)
  test(desktop): await channel E2E bridge readiness (#5886)
  fix(link-preview): refetch a link when it re-enters the composer (#5510)
  feat(desktop-messages): render compact Buzz permalink chips (#5638)
  Fix video comment effect wrapping (#5748)
  Teach agents to inherit Buzz product intent (#5875)
  feat(desktop): one relative date ladder across chat and the Inbox (#3769)
  fix(desktop): amortize observer journal eviction with a low-water mark (#5808)
  Unify agent profile content (#5788)
  Standardize settings section layout (#5855)
  fix(desktop): share one timer across same-interval useNow consumers (#5861)
  Clarify immediate spoken huddle replies (#5863)
  Scope desktop presence subscriptions to active demand (#5830)
  Polish mobile profiles, DMs, and sheets (#5401)
  ...

Signed-off-by: Duncan <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz>
@wpfleger96
wpfleger96 merged commit 1b7e5ac into main Aug 17, 2026
31 checks passed
@wpfleger96
wpfleger96 deleted the duncan/model-capabilities-manifest branch August 17, 2026 15:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants