feat: wire the llm-router into the harness and retire the legacy routing surfaces - #250
Merged
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Important Review skippedToo many files! This PR contains 198 files, which is 48 over the limit of 150. To get a review, narrow the scope: ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (198)
You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
ytallo
changed the title
Contributor
skill-check — worker0 verified, 17 skipped (no docs/).
Four for four. Nicely done. |
andersonleal
approved these changes
Jun 12, 2026
ytallo
force-pushed
the
feat/llm-router-integration
branch
from
June 12, 2026 13:28
93aef68 to
8f2d9d7
Compare
ytallo
force-pushed
the
feat/llm-router-rs
branch
from
June 12, 2026 13:28
def74d2 to
2731bbc
Compare
ytallo
marked this pull request as ready for review
June 12, 2026 18:10
sergiofilhowz
approved these changes
Jun 12, 2026
…n_key Absent options serialized as JSON null, which provider-side schemas reject where a string or array is expected; every default turn would fail validation. The payload builder now omits absent keys and always carries resolution_key = request_id so providers can dedupe per-turn credential resolution across retry attempts.
Consumers that need the provider before streaming (prompt selection, provisioning metadata) call router::route and pin the result as the explicit provider on router::chat, so preview and execution can never diverge. Same decide(), same inputs, same typed error codes as the chat pipeline.
…, catalog getters provider-resolve.ts becomes the llm-router provider-protocol client: token-persisting registerWithRouter (iii-state scope llm-provider-registration, capped-backoff retry), router::ready re-declare subscription, and token-gated resolveProviderViaRouter. The provider stream schema tolerates null options, accepts the router's resolved max_output_tokens, and takes string|number resolution keys. Catalog reads (getCatalogModel, fetchModelLimit) and the discovery reconcile move to router::models::get/reconcile.
One-time idempotent boot migration copies the harness entry's providers block into the router-owned llm-router entry and seeds routing parity with the old local decide() (anthropic default; gpt-/o<digit>- to openai, kimi-/moonshot-v1- to kimi). The harness entry is re-registered permissions-only; paste-a-key reactivity is the router's configuration trigger now.
All five providers self-declare via router::provider::register (re-declaring on router::ready), resolve credentials via the token-gated resolve, reconcile discovery into the router catalog, and honor the router's resolved max_output_tokens as the clamp override. The Phase-1 provider::<id>::complete functions are gone — router::complete drives the stream function.
Provisioning previews the decision once via router::route, pins the routed
provider on the run request (prompt family becomes a pure provider switch),
and resolves model metadata against the router catalog. The streaming step
calls router::chat with a deterministic request_id
(${session_id}:${started_at_ms}); run::abort fires a best-effort
router::abort with the same id so the upstream actually stops. The
close-without-terminal synthetic stays as defense-in-depth. Compaction
summarises via router::complete. Outer trigger budgets are 320s — they must
exceed the router's 300s stream budget.
The picker lists via router::models::list and router::provider::list, the provider-credentials deep links open the llm-router configuration entry, and the ui::models::changed fanout rebinds from the models state-scope trigger to the router::models::changed pubsub topic.
Deleted: the local provider-router decide() library, the provider registry (harness::provider::register/resolve/list) and its refresh-on-config bridge, the models-catalog module (models::list/get/supports/reconcile + scope 'models'), per-provider Phase-1 complete functions, and the compaction stream collector. The catalog Model type moves to types/model.ts, wire- aligned with the router (provider-side extras are optional — the router catalog doesn't persist them). The harness entry is permissions-only; the harness worker now depends on llm-router instead of models-catalog.
Root permission rules deny the router's spend/credential/catalog-write surface to agents and allow the read surface (models list/get/supports, provider list); the dead harness::provider and Phase-1 router::stream_assistant lines are gone, and the legacy models::* allows are retargeted. The harness worker manifest depends on llm-router; the router README documents router::route, the env-fallback process boundary, and token-loss recovery.
Provider, catalog, orchestrator, and compaction suites mock router::provider::resolve / router::models::get / router::models::reconcile / router::complete (envelope payloads, token gating, no channel plumbing for the summariser). New coverage: provider stream schema null tolerance and router-built payloads, registration-token persistence/retry/terminal rejection, ready-topic re-declare binding, and config-migration idempotency and seeding. Suites for the deleted registry, refresh-on-config, models-catalog, and local decide() are gone with their subjects.
LLM providers are standalone llm-router plugin workers from here on (provider-anthropic is the first; the rest follow as their own workers) — they self-register with the router over the provider protocol, so nothing provider-specific belongs in the harness process. Removes the five provider modules with their tests, worker definitions, and the TS provider-protocol helpers (registration tokens, discovery/models.dev enrichment, output-token clamping — the router owns the budget now). The catalog read helper survives as runtime/catalog.ts and AgentFunctionSchema moves to types/function.ts. Until a provider worker is installed, chats fail loudly with router/no_provider_for_model.
The downstream plumbing (run request → router::chat → provider thinking config) was complete but unreachable: the run::start payload schema had no thinking_level field, so Zod stripped it for every caller — harness::trigger included. The schema now takes the level enum; invalid levels fail at the boundary instead of surfacing as provider warnings. Verified live end to end: a run::start with thinking_level=high on claude-sonnet-4-6 lands an assistant message with a thinking block in the session tree.
A thinking select (off…xhigh) joins the composer next to the model picker; the chosen level rides the run::start payload as thinking_level. Thought panes now auto-open while their content streams — reasoning is visible in real time — and collapse back to the duration summary when the block ends.
buildSummaryMessage stamps Date.now() and buildContextView calls it again internally, so an exact-timestamp deep-equal flakes whenever the two calls land across a millisecond boundary (as on the CI runner). The summary expectations now accept any numeric timestamp.
ytallo
force-pushed
the
feat/llm-router-integration
branch
from
June 12, 2026 18:57
e09d581 to
545deb9
Compare
This was referenced Jul 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stacked on #241 (the
llm-routerworker). This PR cuts the harness and console over to the router and removes everything provider-shaped from the harness process: the router becomes the single front door for every LLM call (routing, provider registry, credential resolution, model catalog, streaming relay with typed errors/retries/timeouts, abort), and LLM providers are standalone router plugin workers from here on — they self-register over the provider protocol (#247 is the first; the rest follow as their own workers). Until a provider worker is installed, chats fail loudly withrouter/no_provider_for_model.Router fixes (first two commits)
provider::<id>::streampayload omits absent options instead of serializing them asnull(provider-side schemas rejectnullwhere a string or array is expected), and always carriesresolution_key = request_idso providers can dedupe per-turn credential resolution across retry attempts.router::routefunction: a read-only routing preview ({model, provider?} → {provider, candidates}) over the samedecide()and error codes asrouter::chat. Consumers that need the provider before streaming pin the preview as the explicitprovideron the chat call, so preview and execution can never diverge.Harness cutover
router::chatwith a deterministicrequest_id(${session_id}:${started_at_ms});run::abortfires a best-effortrouter::abortwith the same id, so the upstream actually stops generating. The close-without-terminal synthetic stays as defense-in-depth behind the router's terminal-frame guarantee. Outer trigger budgets are 320s, above the router's 300s stream budget. On the session-manager base (feat: harness with session manager #251), the router channel is the token source and session-manager is the sink: coalesced delta batches land assession::update_messagesnapshots on the turn's assistant entry (thesession::message_updatedfeed is the live token surface), with a strict final update before completion — thinking blocks ride the same snapshots, so live thought panes work unchanged.router::routecall per turn serves prompt-family selection and model-metadata resolution; the routed provider is persisted on the run request.run::startnow acceptsthinking_level('off'…'xhigh') — the field was previously stripped at the payload boundary, making the (complete) downstream plumbing unreachable. Verified live: a turn withthinking_level: highon sonnet lands athinkingblock in the transcript; an unsupported level degrades with a provider warning. The console gains a thinking-level picker next to the model selector, and thought panes auto-open while streaming so reasoning is visible in real time (collapsing to the duration summary when done).harnessconfiguration entry to the router-ownedllm-routerentry via an idempotent boot-time migration that also seeds routing parity with the old localdecide()(anthropic default;gpt-/o<digit>-→ openai,kimi-/moonshot-v1-→ kimi). Theharnessentry is permissions-only now.router::models::list/router::provider::list, provider-credential deep links open thellm-routerentry, and theui::models::changedfanout rebinds to therouter::models::changedpubsub topic.Removed
provider-anthropic/openai/kimi/lmstudio/llamacpp) with their worker definitions and test suites.runtime/catalog.tskeeps the catalog read).turn-orchestrator/provider-router.ts(localdecide()), theharness::provider::register/resolve/listregistry and its refresh-on-config bridge, themodels-catalogmodule (models::list/get/supports/reconcile+ state scopemodels), and the compaction stream collector.Modeltype moves totypes/model.ts(wire-aligned with the router) andAgentFunctionSchematotypes/function.ts. Agent permission rules deny the router's spend/credential/catalog-write surface and allow the read surface.Behavior notes
provider-anthropicis added.ANTHROPIC_API_KEY, …) resolves in the llm-router process — launch the router with those variables or paste keys into thellm-routerentry (documented in the router README, with the token-loss recovery procedure).registrystate key cleared once (providers re-bind onrouter::ready).Test plan
cargo test(54) andcargo test --test integration(12, engine-backed: relay, cancellation, abort, retry, token gate, paste-a-key, route/chat parity)pnpm vitest runinharness/— 895 tests on the rebased (post-feat: harness with session manager #251 session-manager) basepnpm vitest run+tsc --noEmitinconsole/web— 667 testsrun::start, router restart withrouter::readyre-declare, harness restartthinking_levelon the wire → live-streaming thought panes (content sampled growing mid-stream), combined with an agent function call in the same turnsession::message_updated; the stack needs the session-manager worker running)/compactvia UI, paste-a-key flow